Incompatibility of OpenUru.org and H-uru CWE repositories

[rarified]

Apologies if those seven are indeed identical, as I mentioned I had not had the time to locate the divergent change set. I'm away from my computers until tonight and I'll examine your results more thoroughly then.

_R

[Christian Walther]

(But not before I fix a'moaca'’s name in r8.)

Alright, r8 is fixed in https://bitbucket.org/cwalther/cwe-hurubased now.

Any independent verification that what I have done there achieves the goal would of course be welcome, i.e.

• it is identical to what is currently at https://foundry.openuru.org/hg/CWE except for a removed .hgignore file in all file trees, the unfortunate removal of an apostrophe from the author name in r2-5, and, consequently, the revision identifiers – no code formatting changes, directory reorganizations, CMake introductions or anything introduced
• unlike the current state of https://foundry.openuru.org/hg/CWE, it is compatible with http://github.com/H-uru/Plasma.git in that pulling from both over any combination of intermediate steps (with the bridge between Mercurial and Git being Hg-Git), in either Mercurial or Git, will result in a repository with a single common root for all branches.

[branan]

Apologies if those seven are indeed identical, as I mentioned I had not had the time to locate the divergent change set. I'm away from my computers until tonight and I'll examine your results more thoroughly then.

_R

You could have just asked

Code: Select all

commit f8bc99d0c300116c9d2ee883d8256d3a28b9b07c
Author: Branan Purvine-Riley <branan@gmail.com>
Date:   Mon Apr 11 16:27:55 2011 -0700

    Fix line endings and tabs

commit 9ea5838f66d7ae13ac21395b5176a535421d89ce
Author: Branan Purvine-Riley <branan@gmail.com>
Date:   Mon Apr 11 15:14:03 2011 -0700

    Remove old visual studio project files

commit 23329340181fef0c049262066162cb8514430006
Author: Branan Purvine-Riley <branan@gmail.com>
Date:   Mon Apr 11 15:12:51 2011 -0700

    Move Plasma20 to be the new repository root

Those are the major restructuring commits, all back-to-back. Before them is a mix of compiler fixes for modern VS versions, and cmake porting. Some of that might be applicable to you, if you're interested in the compiler fixes.

I *think* the patch to fix the awful numlock behavior is before these changes. I consider that one a "bugfix", personally.

[JWPlatt]

I *think* the patch to fix the awful numlock behavior is before these changes. I consider that one a "bugfix", personally.

I'd actually pay for that fix. But this is open source, so I'll just say thanks, and request maybe you select that one as a process test here that rarified discussed.

[rarified]

Hi Christian,

Between Branan pointing out the changesets and my own exploring this evening, I agree that those first 7 or so changesets are ones we could combine.

I then tried to take things a step further. I cloned a local repo from the current OpenUru repository, then attempted to update it from a "repaired" repository. Unfortunately, as you might expect, the clone would not accept any of the changesets from the repaired repository because there was no locatable common ancestor. Since Mercurial uses a hash of each changeset as the real identifier for that changeset, all those early changesets will have different IDs if the .hgignore file is removed.

Obviously, the consequence is all clones from the OpenUru repository will have to be re-obtained if we were to rebuild the repository (and people with work-in-progress would have to extract their deltas and apply them to new clones). Again, I think you already knew this, hence your question about the number of known clones.

So let me backtrack and ask if you understand the method that hg-git uses to map Git changeset identifiers to Mercurial? The identifier themselves are different (Git uses a longer hash) so they're not directly compared. Is hg-git building it's own mercurial-style ID and then looking for it in the Mercurial repository? Or is there another mechanism being used?

I'm not eager to try Branan's magic dual root since it violates the Mercurial data structure assertions. But I'm wondering if an h-uru branch where each changeset is not a standalone node with a single parent, but rather a merge of the corresponding CWE node on the default branch, plus the previous h-uru branch node, would then provide a match for hg-git to anchor against. I'm going to try that tomorrow.

_R

[branan]

So let me backtrack and ask if you understand the method that hg-git uses to map Git changeset identifiers to Mercurial? The identifier themselves are different (Git uses a longer hash) so they're not directly compared. Is hg-git building it's own mercurial-style ID and then looking for it in the Mercurial repository? Or is there another mechanism being used?

I can answer this one. hg-git converts the commit data back and forth between git and hg. It doesn't directly handle commit hashes at all - it simply guarantees that a round-trip conversion will have the same data, and thus the same hash. There's a caveat - any commit that uses git-only or hg-only features will break not only itself but also all of its children, since child hashes depend on the parent hash.

I'm not eager to try Branan's magic dual root since it violates the Mercurial data structure assertions. But I'm wondering if an h-uru branch where each changeset is not a standalone node with a single parent, but rather a merge of the corresponding CWE node on the default branch, plus the previous h-uru branch node, would then provide a match for hg-git to anchor against. I'm going to try that tomorrow.

From mercurial's documentation (emphasis mine)

Each changeset has zero, one or two parent changesets. It has two parent changesets, if the commit was a merge. It has no parent, if the changeset is a root in the repository. There may be multiple roots in a repository (normally, there is only one), each representing the start of a branch.

Now, other tools layered on top of mercurial may expect only one root, but that's a failure of those tools - mercurial is perfectly happy with multiple roots.

Rewinding a bit, though - my reason for suggesting the merged repo in the first place was simply so that hg would still allow pulls between the updated repo and people who had cloned the previous one. If you're thinking of breaking that compatibility anyway, I'd like to focus on finding solutions to the "getting the code to Cyan" issue as it pertains to the current GoW repo structure, if that's still your main concern with using it. I'm still confident issues in that regard can be resolved with proper use of the tools available in hg and a bit of automation.

[Christian Walther]

Again, I think you already knew this, hence your question about the number of known clones.

Correct. And apparently I haven’t been clear enough about explaining it… I guess I was expecting any interested parties to be familiar enough with decentralized version control to know it. Sorry. I hope we’re on the same page now.

But I'm wondering if an h-uru branch where each changeset is not a standalone node with a single parent, but rather a merge of the corresponding CWE node on the default branch, plus the previous h-uru branch node, would then provide a match for hg-git to anchor against.

There is no such thing as a “h-uru branch where each changeset is […] a merge of the corresponding CWE node on the default branch, plus the previous h-uru branch node”. If you take changesets from H-uru and start adding new parents to them, they become different changesets and are no longer ancestors of what’s in the H-uru repo. You would then have to rebase the entire H-uru repo onto them, which is what I did and what was declined at https://github.com/H-uru/Plasma/issues/26. My goal now is to get changesets into the OU repo that are ancestors of H-uru. If there’s an elegant way of achieving this without disturbing existing OU clones, all the better, but I don’t think there is. (I’m not convinced yet that the “monster merge” is useful, I haven’t done the research about that yet because I’m still hoping that it will become unnecessary.)

From mercurial's documentation (emphasis mine)

Each changeset has zero, one or two parent changesets. It has two parent changesets, if the commit was a merge. It has no parent, if the changeset is a root in the repository. There may be multiple roots in a repository (normally, there is only one), each representing the start of a branch.

Now, other tools layered on top of mercurial may expect only one root, but that's a failure of those tools - mercurial is perfectly happy with multiple roots.

You’re right, apparently I was wrong when I suggested otherwise above. It was only my GUI that was confused about the multi-root repo, not Mercurial itself.

Rewinding a bit, though - my reason for suggesting the merged repo in the first place was simply so that hg would still allow pulls between the updated repo and people who had cloned the previous one. If you're thinking of breaking that compatibility anyway, …

Well, I am indeed thinking of that, and I thought I had gotten Rarified to think of it as well, but apparently I was just confused about him being confused about what I wanted…

[Christian Walther]

Some intermediate results, before this thread gets completely forgotten:

I have examined Branan’s merge. After making another commit on top of it that deletes the MOULOpenSourceClientPlugin folder, trying to merge a branch based on the OU root indeed triggers Git’s rename detection. The bad news is, it fails, on most files. Apparently the whitespace changes are too much for it, and as far as I can see there is no way to tweak it to get over that. So I get conflicts like

Code: Select all

CONFLICT (delete/modify): MOULOpenSourceClientPlugin/Plasma20/Sources/Plasma/PubUtilLib/plInputCore/plInputDevice.cpp deleted in HEAD and modified in cursors-hg.

and have to use techniques like this script (from the Git FAQ) to manually resolve them. (There are still content conflicts after that, but that is inevitable between such diverged branches.)

However, this is not a problem of the merge of the two roots. These rename detection failures will also occur with my original proposal of dropping one root, or any other conceivable way of porting changes across the “great H-uru reorganization” (file moves and whitespace conversion) using Git. So the idea occurred to me, how about trying it on the Mercurial side, which doesn’t detect renames, but explicitly tracks them?

Aside from that, it seems a bit confusing to me to merge the two repositories first, in their current states, which introduces duplicates of most files, and then delete these duplicates in a subsequent commit. Wouldn’t it be more elegant to do the moves first, on the OU side, and then merge, once the files are in the same places on both sides? Specifically, replay the moves as well as anything else that has happened on the H-uru side onto the OU side (basically what I have already done in the initial GitHub report, but squashed into a single commit to avoid cluttering up the history with duplicate commits), so that the two file trees are identical, and then merge them, in a commit that is a no-op in terms of content changes with respect to either parent.

Both of these things are achieved by the following commands (directly executable as a shell script):

Code: Select all

hg clone https://Foundry.OpenUru.org/hg/CWE CWE-hg
cd CWE-hg
hg up -r 5416f15d37e4
hg branch ou-huru-bridge
hg mv MOULOpenSourceClientPlugin/Plasma20/Docs MOULOpenSourceClientPlugin/Plasma20/Sources .
cd Sources/Plasma/FeatureLib/pfConsole/
mkdir ../pfConsoleCore
hg mv pfConsoleCmd.cpp pfConsoleCmd.h pfConsoleContext.cpp pfConsoleContext.h pfConsoleEngine.cpp pfConsoleEngine.h ../pfConsoleCore
cd ../../../..
cd Sources/Plasma/PubUtilLib/plAudio/
hg mv plOGGCodec.cpp plOGGCodec.h plWavFile.cpp plWavFile.h ../plAudioCore
cd ../../../..
hg ci -m "Replaying the file moves of the H-uru master branch."
cd ..

git clone https://github.com/H-uru/Plasma.git CWE-git
cd CWE-hg
hg bookmark -f -r ou-huru-bridge ou-huru-bridge
hg push -r ou-huru-bridge ../CWE-git
cd ../CWE-git
git checkout ou-huru-bridge
git branch temp ou-huru-bridge
git reset --hard master
git reset --soft temp
git branch -D temp
git commit -m "All other changes of the H-uru master branch, squashed into a single commit to avoid cluttering up the revision graph. Check the other leg of the merge for the actual history."
git checkout master
git merge ou-huru-bridge
cd ..

cd CWE-hg
hg pull -r master ../CWE-git
cd ..

So I tried to merge my cursors branch across using Mercurial on this. Alas, only to find that it fails as well:

Code: Select all

remote changed MOULOpenSourceClientPlugin/Plasma20/Sources/Plasma/PubUtilLib/plPipeline/plDXPipeline.cpp which local deleted

I don’t understand why it does this. It should notice the moved files and take them into account in the merge, since it has an explicit record of them. I think this is a bug in Mercurial and I am currently examining it to see if I can find and fix it. That will take a bit more work, so that’s why what I have so far is an “intermediate result”.

If I can fix this problem, then I think that this solution may even be superior to my original proposal of dropping the OU root entirely: It would not force users developing on existing OU clones to rebase, and it would simplify porting of changes across the “great H-uru reorganization”. Of course, it depends on the willingness of the H-uru team to accept a second root in their revision graph and base their further development on the merge.

[rarified]

Thanks for continuing on this. The topic has not been forgotten, I'm just under some tight deadlines in "real work" that are keeping me occupied at the moment. I also have continued thinking about different approaches, have read the code for the mercurial Hg-git plugin, and have tried a couple of approaches in the background. I'll look closer at your approach in a day or two.

_R

[Christian Walther]

No worries, I figured you were busy.