Onward: OpenUru Repository Reorganization

CyanWorlds.com Engine Project Management
Christian Walther
Member
Posts: 317
Joined: Sat Dec 13, 2008 10:54 am

Re: Onward: OpenUru Repository Reorganization

Post by Christian Walther »

Since I’ve surfaced on the MOUL forum, I guess I gotta surface here too to reassure people that I’m not ignoring them…
Paradox wrote:I will double check with CWalther…
Request noted. I’ll respond as soon as I have time, which probably won’t be before the weekend. (It’s going to be longer than the MOUL forum post I could barely justify today – and hey, the results were worth it! :))
User avatar
D'Lanor
Member
Posts: 142
Joined: Tue Dec 23, 2008 11:23 pm

Re: Onward: OpenUru Repository Reorganization

Post by D'Lanor »

Hoikas wrote:
EDIT: And pushed. I have no idea what your fix looked like, D'Lanor, but it probably wasn't much different from what I came up with. It wasn't overly difficult.
True. It is not that much different but since I have no MOULa environment to test in I had to keep it simple and chose to stay away from xKI. So I just worked with the original string, grabbing from it the same chunks you used (this was something I could easily test in a plain Python console). However that made it look somewhat hacky and inconsistent as a whole.
Christian Walther
Member
Posts: 317
Joined: Sat Dec 13, 2008 10:54 am

Re: Onward: OpenUru Repository Reorganization

Post by Christian Walther »

Here we go! :)
Paradox wrote:I believe there's still one remaining issue with pulling changes back and forth between the repositories. We can pull changes from bitbucket into github with no problems, but I don't think a path exists to pull changes from github back to bitbucket. I will double check with CWalther if this is the case, but my understanding is that you would also need to apply the same changes we made so that both repositories are compatible.
It’s important to distinguish between the questions of version control system (Git vs. Mercurial) and repository hoster (GitHub vs. Bitbucket vs. OU Foundry) on the one hand, and of code bases (CWE-ou default, H-uru/Plasma master) on the other hand. These two questions are completely independent.

Thanks to Hg-Git, the first question is no problem at all. Revisions can easily and with round-trip fidelity be pushed/pulled back and forth between Git and Mercurial at will, and by extension between GitHub and Bitbucket. Anyone can use whatever tool they prefer or is better suited to the job for any step of their work.

So, “pulling changes from github back to bitbucket”, which is an instance of question 1, is easy. However, I think that’s not what you actually mean. It will just give you the same stuff on both sides. I assume when you say “pull changes from github to bitbucket”, you actually mean “port changes from the H-uru codebase to the CWE-ou codebase”. That’s an instance of question 2, and it can happen in either Git or Mercurial and the result can be published on either GitHub or Bitbucket at your preference. This is where things get interesting.

In principle, the short answer to your question is “no”. No changes are required to the OpenUru.org repositories to publish a change ported from the H-uru codebase in them (other than pushing that revision, obviously). There is an asymmetry between the OU side and the H-uru side, as will hopefully become clear later on.

However, you are correct that porting a change from H-uru to CWE-ou will most likely be more difficult or more work than the other way around. This is just because there has been much more development on the H-uru side than on the CWE-ou side, so porting from CWE-ou to H-uru mostly consists of forward-porting, while moving from H-uru to CWE-ou is mostly backporting. Forward-porting is comparatively easy, it’s basically a merge. Backporting requires cherry-picking or rebasing and is more likely to run into conflicts because there are many opportunities for the change-to-be-ported to depend on changes that came before it in the source development line, but are not present in the target development line.

These things are hard to explain in words, so I made some pictures.

Image
Figure A shows an overview of the current situation. Most of this is present in https://github.com/H-uru/Plasma and https://bitbucket.org/cwalther/cwe, parts of it in https://bitbucket.org/OpenUru_org/cwe-ou and http://foundry.openuru.org/hg/CWE-ou. (Edit: see next post for a graphical representation.)

It shows the revision graph with the heads of the two main development lines, which are the default branch of CWE-ou and the master branch of H-uru/Plasma. It illustrates that there are two distinct origins, that are however almost identical in terms of file trees (one empty file missing in one). It also illustrates how The Merge, discussed in topic Incompatibility of OpenUru.org and H-uru CWE repositories, establishes a common ancestor to the CWE-ou default and H-uru master branches. Note that this common ancestor lies on the CWE-ou development line. This is the asymmetry that I alluded to above. The common ancestor is already part of the OU repositories, that’s why no change is required there. It is the “MOULa build 1.902” revision.

Let’s first consider the situation that someone has done some development based on that common ancestor: left side of figure B. This was the case with my original cursors pull request, which was my motivation to undertake all this.
Image
This change can easily be ported to either of the CWE-ou default and H-uru master branches: it is simply merged with them, as the right side of figure B shows. Note that in the case of the merge to H-uru master, there is a large number of revisions on the right leg of the merge from the common ancestor, creating ample opportunity for merge conflicts. In particular, the two orange ones labeled as “Preparation for The Merge” in figure A are part of that leg, containing the file moves and whitespace conversions that make up one of the big differences between today’s CWE-ou default and H-uru master codebases. It is advisable to do this merge in Mercurial, because the “preparation” revisions include explicit file move information for Mercurial that helps it get over the file moves without tree conflicts, while Git with its after-the-fact file move guessing in my experience is confused by the whitespace changes and fails. There will usually still be content conflicts due to the whitespace changes, but these can be resolved more easily than tree conflicts.

Next, let’s try porting some development based on the current CWE-ou default to H-uru master: figure C.
Image
In this case, merging doesn’t work, because that way you would include everything starting from the common ancestor (in this case, one additional revision, the CWE-ou default tip (hg 2e5a53a / git a2f42c3)). Instead, you need to cherry-pick or rebase the desired revisions across (I think the term “cherry-picking” is commonly used for single revisions only, while “rebasing” is the general term). You’re going back by one revision and forward by many here, which has some chance of working without too many conflicts (apart from whitespace), assuming that your VCS safely takes you across the bumps in the “preparation” revisions as above (using Mercurial at least for that part may be helpful).

Finally, porting changes from H-uru master to CWE-ou default: figure D.
Image
The procedure is exactly the same here, only that this time you’re going back by many revisions and forward by one, which is more likely to run into conflicts.

I have never attempted either of the last two scenarios, so their treatment is purely hypothetical.
Last edited by Christian Walther on Sat Sep 17, 2011 3:35 pm, edited 1 time in total.
Christian Walther
Member
Posts: 317
Joined: Sat Dec 13, 2008 10:54 am

Re: Onward: OpenUru Repository Reorganization

Post by Christian Walther »

I keep thinking of new things to diagram… :)

This one shows which parts of the total revision graph are currently available in which public repositories:

Image
User avatar
JWPlatt
Member
Posts: 1137
Joined: Sun Dec 07, 2008 7:32 pm
Location: Everywhere, all at once

Re: Onward: OpenUru Repository Reorganization

Post by JWPlatt »

Wow, that was a bit of work Christian. Impressive, as always. Whitespace keeps coming up, as does structure and cherry picking. I've been meaning to write about what these things mean to getting code here, but, sadly, I haven't gotten around to it.

In a post on the previous page of this thread I mentioned rarified has offered "to clone the h'uru repo onto Foundry and set up an analysis on Hudson (Jenkins) to find the true extent of actual changes." Paradox responded with the technical difficulties about "pulling changes back and forth" between the CWE-ou repo and the GoW fork, but we have not heard back about whether the GoW is interested in rarifed's original idea.

The conversation moved on, but I'd like to get it back using whitespace as an example. Rarified's analysis would cull out the real meat whereas any current diff would be lost in the whitespace noise. Now - besides rarified's offer, Phoenix also did some work which he wrote about in a post immediately following mine. I have to admit I haven't taken a close look at it to understand what it means, but I'm just guessing that his "GoW White Space Cleanup" restores the original whitespace, leaving only the changed code to produce an intelligible diff. I'm sorry I haven't taken the time to look it over, but that's my guess - or wishful thinking. If my guess is correct, it could go a long way in helping to get GoW code into CWE-ou.

Please take a look at these ideas because, speaking just for myself and not the team, I don't see a better way to get code into CWE-ou and on to MOULa than to fix at least the whitespace problem, let alone some others like structure, so that the code can be more easily reviewed, tested and accepted. As it stands, I see no GoW merge solution without cherry picking and whitespace restoration in any case, and repo compatibility grows more moot as time passes. I would really like to have seen whitespace changes restricted to lines of code actually modified, if at all, among other things. Now things are more difficult. But maybe I'm just lacking in time and imagination, so other ideas to make this easy are welcome.

As a start, we can use the request to get unicode suport into MOULa and concentrate just on that code.
Perfect speed is being there.
User avatar
JWPlatt
Member
Posts: 1137
Joined: Sun Dec 07, 2008 7:32 pm
Location: Everywhere, all at once

Re: Onward: OpenUru Repository Reorganization

Post by JWPlatt »

Oh, you know, I logged on only to write this post, below, but got distracted by Christian's shiney graphics:

Bitbucket now supports changeset commenting for reviews:
http://blog.bitbucket.org/2011/08/22/no ... -mentions/

It would be nice to have inline code comments too, but changeset comments are a good start.

That said, our Bitbucket mirror is for people who enjoy the ease-of-use that comes with social repositories. We still have an enterprise class code management system on Foundry that does anything you can't find on Bitbucket. ;)
Perfect speed is being there.
Christian Walther
Member
Posts: 317
Joined: Sat Dec 13, 2008 10:54 am

Re: Onward: OpenUru Repository Reorganization

Post by Christian Walther »

Getting the differences, ignoring whitespace changes and taking into account moved files, is as simple as

Code: Select all

hg diff --git --ignore-all-space -r 2e5a53a -r d8c1551
However, I’m not sure if that’s useful. The diff is huge (27 MB, although removed files like MsDevProjects and Cypython-2.3.3 account for the majority of that), and having all the unrelated changes lumped together like that does not look very informative to me. Seeing a changed line of code tells you very little in absence of the commit message that explains why it was changed and what is achieved by changing it, unless you are very familiar with the code.
User avatar
JWPlatt
Member
Posts: 1137
Joined: Sun Dec 07, 2008 7:32 pm
Location: Everywhere, all at once

Re: Onward: OpenUru Repository Reorganization

Post by JWPlatt »

lol. Interesting. In all this time, no one has mentioned a whitespace diff option, so I never thought about it. Why is it always you to the rescue, Christian? Some people, like me, don't like CLIs, even if we know them, and think of them as relics of a more primitive time, so I brought up TortoiseHg to see if there is a corresponding menu option for, you know, normal people who would like to get involved. Sure enough, Diffview has the whitespace option. When I get a chance I'll see if I can play around with the h'uru repo. Could help. Someone will have to check Fisheye/Crucible on Foundry for similar features. Does the Bitbucket UI have it? Does Github?
Perfect speed is being there.
User avatar
rarified
Member
Posts: 1061
Joined: Tue Dec 16, 2008 10:48 pm
Location: Colorado, US

Re: Onward: OpenUru Repository Reorganization

Post by rarified »

Christian, I'm way behind on tracking stuff so first let me also say thanks for your essay and analysis. As usual it's excellent. Where I'd like to go from here is produce a couple of examples of code fragments as they exist on either repository, and what happens when we merge them. Line by line listing.

Question on the ignore whitespace stuff... does it ignore added/removed line breaks? I was going to use the standard 'diff -b' comparisions which apply fuzzy comparisions on each line, but if a newline was added to change

if (foo) { bar; }

to

if (foo) {
bar;
}

then it wouldn't compare identically with 'diff'. Yet the newlines should be considered 'whitespace' as far as the language is concerned.

_R
One of the OpenUru toolsmiths... a bookbinder.
Christian Walther
Member
Posts: 317
Joined: Sat Dec 13, 2008 10:54 am

Re: Onward: OpenUru Repository Reorganization

Post by Christian Walther »

rarified wrote:Question on the ignore whitespace stuff... does it ignore added/removed line breaks?
No. The diff still works line-by-line, it only ignores whitespace inside the lines. I don’t know of any differ that does not work line-by-line.

There don’t seem to be many such changes however, if any at all, from a cursory glance.
Post Reply

Return to “Management”