Onward: OpenUru Repository Reorganization

[Hoikas]

I keep meaning to respond to this thread with something useful. Unfortunately, that planned useful response hasn't had the time to be created.

Question on the ignore whitespace stuff... does it ignore added/removed line breaks?

No. The diff still works line-by-line, it only ignores whitespace inside the lines. I don’t know of any differ that does not work line-by-line.

There don’t seem to be many such changes however, if any at all, from a cursory glance.

We had plans to run astyle over the code to normalize indentation, bracket styles, etc. Fortunately for you guys, that hasn't been done yet . The only formatting change that we've made in H-uru/Plasma is the tabs -> 4 spaces change. H-uru/moulscripts has had its newlines converted to unix style \n.

[rarified]

Here's a suggestion... if a good argument can be made for reformatting the source code.

What if we make a clone repository from the CWE-ou source (with changes as they come in from whatever source) where during the pull process from CWE-ou to the clone, reformatting as you prefer is done on-the-fly. That way you can have a beautified repo for browsing/documentation, even changes if they can be repackaged as patches to the original source, that always is in sync with the CWE-ou repo.

If the reformatting is done mechanically, I can install the hooks to invoke the reformatter when updating the subrepo.

_R

[rarified]

I've begun to look through the H'uru GitHub repository for changes related to Unicode support.

It appears, at least from the commit comments that the work for wide characters is spread out along many different changesets.

It would be helpful if someone familiar with the Unicode changes could respond with a list of the changeset identifiers (git) related to the full Unicode fixes, and then I could assemble a patch back to our CWE-ou repository from those. I'll try to proceed without it, but a list would certainly help speed things along.

Thanks
_R

[rarified]

If the unicode changes being requested are those limited to text messages in the KI, are changesets

MOULSCRIPT[Python/xKI.py]: 80186c1 "Fix some more unicode errors" by Adam Johnson
MOULSCRIPT[Python/xKI.py]: ca9ec2d "Support cp1252 in xKI chat messages" by Adam Johnson

all that is required?

Thanks

[edit: added second changeset after review]

[Hoikas]

The unicode support was added in two stages. One stage introduced basic unicode functionality, and the other stage improved support for international keyboard layouts. I will attempt to list the relevant commits. We also had to add python package support to plPythonPack--this means that you will need to add the Python 2.3 encoding package to your MOULSCRIPTS repository as we are using Python 2.7

H-uru/Plasma:
af81194913c4e531f1c630914d0405bb866e2d80
0eef21f2bf1cc90254910338eeeb3f151ad52190
6836fe423c284e5edb03a1fb71dde7108134118e
95e9047a3f9fe16b503d025419de8e9b1f30a18c
169c0c502a53c3cd440c8343535d866de2bb80f6
0509bacf0fe378bfac7a9dd899b215f397652341
45b82c1103b745074d49a77e914312716cad43e9

H-uru/moul-scripts
ca9ec2d85b90a22ddb67013585d25e79ddd7d6d0
8e3ff4d7de92fd7485d6cb9a20bf125f0dce9f02
80186c1220c89392ef864b67129d16a667b6f907

[rarified]

Thanks, Adam!

[Christian Walther]

Following up on my earlier porting explanations, a warning, in case anyone is doing such things and finds it useful: I just discovered two obstacles in Mercurial/Git interoperability.

1. Git revision 1a3ca94 can transform into two different Mercurial revisions: d2e31bf73a2c when Mercurial 1.8.4 is used, and 7bfb960b4d04 when Mercurial 1.9 or 2.0 is used (with the same version of Hg-Git). This is a bad thing, because it means that any revisions based on it (basically the entire H-uru/Plasma history) change their identity as well. This means that anyone who wants to move anything that includes this revision from Git to Mercurial and wants to be compatible with what has already been done and published by me (using Mercurial 1.8 at the time) must use Mercurial 1.8 too. Fortunately, there appears to be a way around that: Instead of converting the troublesome revision from Git to Mercurial yourself, pull the already converted one from Bitbucket. Pulling more revisions based on it from Git to Mercurial using Mercurial 1.9/2.0 then appears to correctly recognize it as an ancestor and results in matching revision IDs with the 1.8 case.
   
   I have been unable to figure out what change exactly causes the difference, because Mercurial’s history is highly nonlinear and many of the revisions in development branches don’t work with Hg-Git. I also don’t know which of the two outcomes is more correct, if any.
2. Mercurial revision f7cecd95cba5 does not round-trip-convert correctly. It converts as hg f7cecd95cba5 → git 08fa54e → hg 050a1be2ccf2 → git 08fa54e. I am still investigating why. Until it’s clear which of the conversions is faulty, avoid pushing this revision into Git and publishing any work based on it.

[Christian Walther]

Here’s what I found out regarding point 2: Neither of the conversions is faulty, it is revision f7cecd95cba5 itself that is corrupted, and 050a1be2ccf2 is what it should be. Git cannot represent the corruption, so converting to Git and back uncorrupts it.

The easiest way to prove that is to replicate the revision, with no involvement of Hg-Git or Git, by pulling its ancestors into a new repository, doing the merge, applying any differences from the automated merge result that are there in the original revision for reasons unknown to me, and committing with the same commit metadata:

Code: Select all

hg init testrepo
cd testrepo
hg pull -r 2e5a5 https://bitbucket.org/OpenUru_org/cwe-ou
hg pull -r 8246b https://bitbucket.org/OpenUru_org/cwe-ou
hg up -r 8246b
hg merge -r 2e5a5
chmod a-x MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/config.guess MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/config.sub MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/configure MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/depcomp MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/install-sh MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/ltmain.sh MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/missing 
rm MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/testimg.bmp MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/testimg.jpg MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/testimgp.jpg MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/testorig.jpg MOULOpenSourceClientPlugin/StaticSDKs/XPlatform/jpeg-8c-rgba/testprog.jpg
hg commit -m 'Merge "Open source re-licensing" into OpenUru base' --date "Thu Oct 20 20:06:21 2011 -0600" --user CyanWorlds
hg log -v -r tip

This ends up with revision ID 050a1be2ccf2 here as well.

The corruption is of the same kind as the one that Hg-Git itself could create before I fixed it: For certain files that ended up with equal contents as in the first parent revision in the merge (but had different histories in the two parents), the file histories were not properly merged, but the file revision from the first parent was taken and the one from the second parent left disconnected.

Compare e.g. from a repository containing only f7cecd95cba5 and its ancestors:

Code: Select all

$ hg debugindex .hg/store/data/_m_o_u_l_open_source_client_plugin/_plasma20/_sources/_plasma/_feature_lib/pf_surface/pl_layer_bink.cpp.i 
   rev    offset  length   base linkrev nodeid       p1           p2
     0         0    2264      0       0 80fd3cb83d5f 000000000000 000000000000
     1      2264     191      0       5 402929fa720b 80fd3cb83d5f 000000000000
     2      2455     642      0      10 077de9bb1e2d 80fd3cb83d5f 000000000000

From a repository containing only 050a1be2ccf2 and its ancestors:

Code: Select all

$ hg debugindex .hg/store/data/_m_o_u_l_open_source_client_plugin/_plasma20/_sources/_plasma/_feature_lib/pf_surface/pl_layer_bink.cpp.i 
   rev    offset  length   base linkrev nodeid       p1           p2
     0         0    2264      0       0 80fd3cb83d5f 000000000000 000000000000
     1      2264     191      0       5 402929fa720b 80fd3cb83d5f 000000000000
     2      2455     642      0      10 077de9bb1e2d 80fd3cb83d5f 000000000000
     3      3097       0      0      12 fb42538d91ad 077de9bb1e2d 402929fa720b

(Revision 5 is dcfd1e399912, 10 is a4a583e629c6, and 12 is the merge (050a1be2ccf2) here.)

In this case the corruption is probably harmless, as nothing important (in particular no file renames) happened on the cut off second leg of the merge and treating the file as if it had actually not been merged is good enough. Trying to eradicate the corrupted revision is probably not worth the effort, as several people (at least Skoader and me) already have work based on it.

(Note that my choice of words “corruption” is based on my limited understanding of how Mercurial works. Someone who is more familiar with Mercurial may arrive at the conclusion that it is not “corruption” but completely valid and that there are legitimate reasons for Mercurial to produce the revision I call “corrupted”. I do not know how Cyan Worlds created that revision, knowing that would help clear this up. In that case it would simply be a feature of Mercurial that Git cannot represent (because it does not track file histories, only file contents) and that Hg-Git would have to separately encode in the Git commit message (as it does for other things unsupported by Git such as file renames and per-revision branch names) to ensure round-trip fidelity, which it currently obviously does not do.)

—

What do we take from this? Apparently Mercurial/Git interoperability using Hg-Git is not quite as seamless and effortless as I had hoped. Relying too much on its repeatability and round-trip fidelity without checking can be dangerous. However, it can still work and be useful when done carefully.

A useful rule of thumb could be:
Any revision should only be converted from the VCS it was originally committed in to the other once, and never converted back. If you want to work with a certain revision in a certain VCS, check first if it is already available in that VCS (either because it was originally committed in it or because it has already been converted elsewhere by you or someone else) and if so pull it from there. Only convert it yourself if you can’t find it. If you deviate from these rules (which is sometimes easier), double-check that you end up with the same revision IDs as if you hadn’t.

[rarified]

Christian,

Have you tried the "convert" extension in Mercurial? I believe it is more recent than hg-git, can convert from Hg to or from Git (or a number of other SCMs), and better still, can do some mapping on various fields (such as names) between SCM repositories. I've had good luck with it when pulling from H-uru (fixing A'moaca's name, for example), and it can do some major structural fixes such as reparenting commits, etc. You might take a look.

_R

[Christian Walther]

I think I did when you last mentioned it. I wasn’t convinced it was useful for our purpose, but maybe I can have another look. Can it create the same revision IDs as Hg-Git? Otherwise it’s useless, since we already committed to the Hg-Git way. I’m not sure the additional features you cite are a good thing in our situation – more configuration parameters just makes it harder to get them right so that everyone gets matching revision IDs. The fewer parameters the better.