Incompatibility of OpenUru.org and H-uru CWE repositories

CyanWorlds.com Engine Project Management
Christian Walther
Member
Posts: 317
Joined: Sat Dec 13, 2008 10:54 am

Re: Incompatibility of OpenUru.org and H-uru CWE repositorie

Post by Christian Walther »

rarified wrote:Let's close the discussion tonight so if it is desired I can have the weekend to do it.
I think that’s a bit short notice to give all potentially affected folks a chance to veto, but if you’re willing to take the responsibility to go ahead without that, that’s fine with me.
branan wrote:We might be able to replicate the coolest merge ever and create a new repository that has both GoW and OU as parents. It'll look ugly as sin, and have duplicates of the commits that were modified during the conversion... but it will create a repo that will work for what we need.
If I understand correctly, that would result in a repository with two origins (roots, whatever you call them – revisions with no parent). Note that while Git gracefully deals with that, Mercurial appears to have some trouble with it, from what I’ve seen. It refuses to pull in an additional origin in the first place unless you --force it, and when you do, it tries to attach it to the previous origin rather than letting it stand alone, but still treats file additions as additions, so merges still don’t work even though there seemingly is a common ancestor.

I think having a common origin, as my proposal in the first post would achieve, would make things easier for us, given that we’re not only trying to fix up disparate bits of history, but also to work across two version control systems.

But the “Linus magic” option would be a good fallback plan if that doesn’t go through – I wasn’t aware of that possibility, thanks for bringing it up.
branan wrote:Hg-git should allow two-way work, and it's what we used for the initial conversion
Ah, that’s good to know (sorry for not doing my research). It means that most likely what mangled a'moaca'’s name was a bug in Hg-Git and we should be able to fix it before it wreaks more havoc on future commits. I’ll have a look – I’m not familiar with the internals of either Mercurial or Git, but maybe I’ll find it.

Edit: OK, it’s apparently not Hg-Git’s fault: I repeated my experiment and noticed that Hg-Git got the name into Git just fine, it was the subsequent ‘git rebase’ that chopped the apostrophe. Or try this, no Hg or Hg-Git involved at all:

Code: Select all

tsessebe:~ cwalther$ git init test
Initialized empty Git repository in /Users/cwalther/test/.git/
tsessebe:~ cwalther$ cd test
tsessebe:~/test cwalther$ echo "Hello World" > hello.txt
tsessebe:~/test cwalther$ git add hello.txt
tsessebe:~/test cwalther$ git commit -m "hello" --author "a'moaca' <none@none>"
[master (root-commit) 43a6777] hello
 Author: a'moaca <none@none>
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 hello.txt
tsessebe:~/test cwalther$ git log | cat
commit 43a6777b25912c1f9c4bbfbe94753d5fb56d235e
Author: a'moaca <none@none>
Date:   Sat Apr 30 14:15:32 2011 +0200

    hello
tsessebe:~/test cwalther$ git --version
git version 1.7.5
Edit 2: Let me show you this, a'moaca', and leave it to you to duke it out with Linus: :)
(from http://git.kernel.org/pub/scm/git/git.git/ident.c)

Code: Select all

static int crud(unsigned char c)
{
	return  c <= 32  ||
		c == '.' ||
		c == ',' ||
		c == ':' ||
		c == ';' ||
		c == '<' ||
		c == '>' ||
		c == '"' ||
		c == '\\' ||
		c == '\'';
}

/*
 * Copy over a string to the destination, but avoid special
 * characters ('\n', '<' and '>') and remove crud at the end
 */
static int copy(char *buf, size_t size, int offset, const char *src)
{
	size_t i, len;
	unsigned char c;

	/* Remove crud from the beginning.. */
	while ((c = *src) != 0) {
		if (!crud(c))
			break;
		src++;
	}

	/* Remove crud from the end.. */
	len = strlen(src);
	while (len > 0) {
		c = src[len-1];
		if (!crud(c))
			break;
		--len;
	}
…
(It looks like he wouldn’t be averse to a fix though: “We should probably do better quote removal from names anyway”.)
User avatar
branan
Member
Posts: 84
Joined: Wed Apr 06, 2011 11:35 pm

Re: Incompatibility of OpenUru.org and H-uru CWE repositorie

Post by branan »

I was able to get the repos to play nice together. I've got the merged repo at bitbucket.

This is only the first step - the old MOULOpenSourceClientPlugin folder has to be removed, as do the batch files. I just wanted to post the "merged" repo so everyone could see it.

Git will (hopefully) handle the changed paths gracefully. It should see that the old MOULOpenSourceClientPlugin/ files actually reside under Sources/ now, and adjust patches appropriately during merges. Hopefully our line ending conversion didn't break that. I haven't tested that level of awesome (yet) - I just tested basic "commits can be shared" logic with a test file in each of hg and git being pushed to each other and merged. I have no idea how well hg handles that - it has an actually concept of "moved" files, so might not recognize the pseudo-move that results from removing the MOULOpenSourceClientPlugin directory. Moar Science is needed

Here's the repository graph in a sane form:
Spoiler
Image
User avatar
rarified
Member
Posts: 1061
Joined: Tue Dec 16, 2008 10:48 pm
Location: Colorado, US

Re: Incompatibility of OpenUru.org and H-uru CWE repositorie

Post by rarified »

Wow, a lot of work went into that. Thanks.

For the remainder of the discussion, I'd like to mention two precepts that we want to adhere to in the OpenUru.org (OU) repository:
  • Retaining original directory structures until absolutely necessary to change, and then in consultation with Cyan, and
  • A widely accepted principle regarding source from outside sources: never make formatting changes to original code blocks, no matter how strong my personal preference is for something better (in my judgment).
We do have some issues with this incarnation, which I'll elaborate on a little below, but we do have a plan.

Contrary to the first principle there has been a lot of rearrangement of the sources and directories. Even if it is an organization that appears more rational, what will Cyan have to do to compare their source to this tree?

With regards to the second principle, the quantity of whitespace changes in the merged tree will now take more time to compare to the original sources, and if we hope to get updated and fixed source utilized by MOULa we have to be sensitive to the effort it places on Cyan to qualify the code, and perhaps integrate it into their own source repositories. (If we had a formatting standard the code was supposed to adhere to, then "fixing" nonconformant code would be something we could probably agree with, but in the absence of having been provided with such a standard, we really should preserve the original formatting of code otherwise not needing functional changes.)

I'm still looking at a couple of topologies for tracking common points, but it now may be difficult to fully merge at those points with the SCMs we have, which are agnostic to the content of the files and cannot determine which changes to formatting is innocuous.

Please accept that we don't arbitrarily make up these rules. They're the result of long experience with shared source projects, before "Open Source" was even a watchword. I've been on both the giving and receiving end and have seen how ignoring these principles are received (answer: not well).

That said, we want to find a solution that best preserves the work you have done, anything Cyan or we have done, integrates it all into OU support, and maintains the development path for everyone to MOULa without putting additional load upon you, us, Cyan (i.e., Mark) or those who wish to work most easily in any branch. We are trying to consider everyone now and into the future, not just our own mutual interests today. This is what we want to do:
  • Pull your current branch as-is into a "huru" branch on Foundry to mirror your development work. The mirror would be kept read-only to outside users so the single point for committing changes to "huru" will be your own Github repository.
  • Pull future changes from your Github repo into the "huru" branch automatically through our automated Jenkins facility.
  • Encourage development on both branches - there is value in having the "huru" branch local for comparison and extraction of changes. Contributors that make their initial commits to one repository won't have to do cross-repository comparisons to propagate their work to the other. The changeset will be present within the Mercurial repo for merging to whichever branch desired.
  • Point JIRA to your Github repo so that issues can be filed in JIRA against your branch.
  • Mirror the entire CWE repo on our Bitbucket account to make things more convenient for Mercurial users.
  • Complete all these changes within one week.
  • We would ask that the H-uru team determine if any of their changes qualify as a "fix" that would be applicable to the currently offered game, and if so, they prepare one or more of those changes as a delta to apply to the "dev" branch, so we can confirm that our process for review and incorporation will work to everyone's satisfaction.
There will probably be some details to work on to make sure everything goes smoothly. We'd like to note that if 50 more development groups emerge and they all want a branch just like yours, we're not going to do them. Those groups can work locally just as you are, then push changes here to our "dev" branch here, or directly to your remote branch. Changes can then be reconciled from there into the "dev" branch for review into the "default" branch and on to be considered for MOULa. This, for your use for a long as you maintain your fork/branch. The Guild of Writers (GoW) has been involved in development since the beginning and we want to recognize that. We understand your wish to move beyond constraints that are needed for backward compatibility, and explore new platforms and approaches, and we eagerly look forward to the time when the ideas you now have the freedom to explore can make it into the communal game. It's not quite there yet, but we want to do this for you and to settle community division and confusion as they wonder how they will get their code and content onto MOULa. Please discuss, but it is our hope you will accept this offer to work together on the project so we can get on with it.

Finally, please understand we believe the release of Open Source was not meant to be a call for competition, unless some groups explicitly take that up as one of their goals. OU and the GoW have different strengths and resources. For our part, we are here to support development from others, from contributors with a wide spectrum of interests, skills, and their own resources. We are trying to use our collective experience to make the process of incorporating additions and improvements to the game both productive and expedient. Those processes didn't exist when we started, and what processes exist now are just proposals. It will take several iterations of using those processes to refine them into effective ones. Until and unless we have people who are willing to take a risk and try working through what we have, improvements won't happen. You can help influence the outcome by participating.

_R
One of the OpenUru toolsmiths... a bookbinder.
User avatar
branan
Member
Posts: 84
Joined: Wed Apr 06, 2011 11:35 pm

Re: Incompatibility of OpenUru.org and H-uru CWE repositorie

Post by branan »

rarified wrote:Wow, a lot of work went into that. Thanks.

For the remainder of the discussion, I'd like to mention two precepts that we want to adhere to in the OpenUru.org (OU) repository:
  • Retaining original directory structures until absolutely necessary to change, and then in consultation with Cyan, and
  • A widely accepted principle regarding source from outside sources: never make formatting changes to original code blocks, no matter how strong my personal preference is for something better (in my judgment).
We do have some issues with this incarnation, which I'll elaborate on a little below, but we do have a plan.

Contrary to the first principle there has been a lot of rearrangement of the sources and directories. Even if it is an organization that appears more rational, what will Cyan have to do to compare their source to this tree?

With regards to the second principle, the quantity of whitespace changes in the merged tree will now take more time to compare to the original sources, and if we hope to get updated and fixed source utilized by MOULa we have to be sensitive to the effort it places on Cyan to qualify the code, and perhaps integrate it into their own source repositories. (If we had a formatting standard the code was supposed to adhere to, then "fixing" nonconformant code would be something we could probably agree with, but in the absence of having been provided with such a standard, we really should preserve the original formatting of code otherwise not needing functional changes.)
Our only major directory move was getting rid of the MOULOpenSourceClientPlugin top-level - and I somehow doubt Cyan has that on their source tree. In addition, both mercurial and git allow checking out a repo that contains unix line-endings in windows form if you so desire. The only other issue would be the tab conversion, which can probably be scripted for sending patches to Cyan (since the tab->spaces conversion was scripted in the first place) Assuming that can be scripted (and I'm confident it can be), patches from our repository should be mostly applicable to Cyan's internal codebase. If you have other reasons not to try merging the repos (or even just replacing yours with ours, since we have all of your code patches merged), I'd love to hear them. I'll assume you do have other reasons and keep on with the rest of your points.
rarified wrote:I'm still looking at a couple of topologies for tracking common points, but it now may be difficult to fully merge at those points with the SCMs we have, which are agnostic to the content of the files and cannot determine which changes to formatting is innocuous.
Git has some facilities for noticing when a file has been moved based on contents. It's not foolproof, and I haven't tested it in this context. If you've done tests and it doesn't work, I'll defer to you here. If you haven't tested, I think we should test it before we write it off entirely. I'll note that if we do agree on some sort of merged tree, this point is entirely moot except for any currently outstanding patches from the community.
rarified wrote:This is what we want to do:
  • Pull your current branch as-is into a "huru" branch on Foundry to mirror your development work. The mirror would be kept read-only to outside users so the single point for committing changes to "huru" will be your own Github repository.
  • Pull future changes from your Github repo into the "huru" branch automatically through our automated Jenkins facility.
  • Encourage development on both branches - there is value in having the "huru" branch local for comparison and extraction of changes. Contributors that make their initial commits to one repository won't have to do cross-repository comparisons to propagate their work to the other. The changeset will be present within the Mercurial repo for merging to whichever branch desired.
  • Point JIRA to your Github repo so that issues can be filed in JIRA against your branch.
  • Mirror the entire CWE repo on our Bitbucket account to make things more convenient for Mercurial users.
  • Complete all these changes within one week.
I'm going to have to say "thanks but no thanks" here. The GoW and the H'uru team have a history of strong independence, and it's not in our best interests to have our repository appear to be an OU project. Obviously the management of the repos would be independent, but that wouldn't be immediately visible to a casual observer. In addition, I think having our branch on your Jira would be a possible point of confusion for people trying to report bugs. It's better to keep the two trackers separate, especially if the codebases continue to diverge.

Obviously our code is GPL, so you can pull it into a local OU repo if you really want to. But then we'll have to make it clear it's not at all part of our development structure, and that gets messy for everyone.
rarified wrote:
  • We would ask that the H-uru team determine if any of their changes qualify as a "fix" that would be applicable to the currently offered game, and if so, they prepare one or more of those changes as a delta to apply to the "dev" branch, so we can confirm that our process for review and incorporation will work to everyone's satisfaction.
We can certainly make an effort on this - we've got almost a month's work of work at this point, but areas where we can easily extract patchsets we can possibly send them over.
Nye_Sigismund
Member
Posts: 64
Joined: Wed Sep 29, 2010 12:59 pm

Re: Incompatibility of OpenUru.org and H-uru CWE repositorie

Post by Nye_Sigismund »

branan wrote:I'm going to have to say "thanks but no thanks" here. The GoW and the H'uru team have a history of strong independence, and it's not in our best interests to have our repository appear to be an OU project.
So that's that then, I guess? I'm not a member of OU but that looks pretty final to me...

Moving on, the -dev branch seems like a winning idea. The perfect is the enemy of the good - so let's not continue fretting about over the very best practice. Get that set up so it's easy for people to contribute code, and then it'll be up to the people maintaining the OU branch to review and adopt the good stuff. Ideally, make sure that people can branch the code onto a network so everyone can see what's going on. Really ideally that network should be able to send me email when somebody does something. :P

Get that done and we should be good, I hope?
Last edited by Nye_Sigismund on Mon May 02, 2011 3:03 pm, edited 1 time in total.
Huw Dawson
Team Member
Team OSCAR
User avatar
rarified
Member
Posts: 1061
Joined: Tue Dec 16, 2008 10:48 pm
Location: Colorado, US

Re: Incompatibility of OpenUru.org and H-uru CWE repositorie

Post by rarified »

Nye_Sigismund wrote:Really ideally that network should be able to send me email when somebody does something.
Addressing this point alone, log into Fisheye at http://foundry.openuru.org/fisheye, click the 'Source' item in the top task bar. Select your favorite repository. At the upper right, under the task bar click the "Tools" button, and select "Watch". You'll get an email every time a change is checked in, along with the commit log entry. Go back to the Tools button any later time and you will have the option to discontinue watching the repository.

_R

Edit: added note about discontinuing watch
One of the OpenUru toolsmiths... a bookbinder.
Nye_Sigismund
Member
Posts: 64
Joined: Wed Sep 29, 2010 12:59 pm

Re: Incompatibility of OpenUru.org and H-uru CWE repositorie

Post by Nye_Sigismund »

Thanks Rarified, good to know that functionality exists.

I think I should probably step aside for the time being and allow actual programmers to discuss this stuff... :P
Huw Dawson
Team Member
Team OSCAR
Christian Walther
Member
Posts: 317
Joined: Sat Dec 13, 2008 10:54 am

Re: Incompatibility of OpenUru.org and H-uru CWE repositorie

Post by Christian Walther »

It seems like the discussion is expanding beyond my initial point. That’s not a bad thing, but could we decide and act on that first, as it seems to be an independent subset?

To recap, here’s the state of the matter as I gather it:
  • I propose to discard the first 7 revisions of the OpenUru.org repository and to replace them with the corresponding ones from the H-uru repository. (Note that this does not affect the “two precepts” rarified mentions, since the file trees are identical save for the .hgignore file.)
  • The OpenUru.org management is not opposed to this idea.
  • Rarified is willing to carry out the proposed action if a decision is made in favor of it.
  • In doing so, care is to be taken that author names are preserved. This means avoiding the use of Git, which does not fully support some names (those ending in apostrophes in particular), for certain operations.
  • Nobody knows of any potentially affected parties (with the exception below) that would need to be asked for their opinion. It can be assumed that any that may still exist are reading this and would give their objection here if they were opposed to the idea.
  • As an exception to the above, it is unknown whether Cyan would be affected, and they cannot be assumed to read this forum. Given consent, I will therefore contact Mark and ask.
  • The proposal is considered accepted on explicit approval by Cyan and no explicit objection on this forum within a timespan to be determined.
Does everyone agree with that?

If the proposal goes through, the “monster merge” prototyped by Branan becomes unnecessary and a good basis is established.

If the proposal is declined, the benefits and drawbacks of the “monster merge” need to be analyzed in greater detail.

In either case, whether any further integration or collaboration between OpenUru.org and H'uru/GoW is desirable can then be discussed independently.
User avatar
rarified
Member
Posts: 1061
Joined: Tue Dec 16, 2008 10:48 pm
Location: Colorado, US

Re: Incompatibility of OpenUru.org and H-uru CWE repositorie

Post by rarified »

Christian Walther wrote:It seems like the discussion is expanding beyond my initial point. That’s not a bad thing, but could we decide and act on that first, as it seems to be an independent subset?
The repository work is the next thing on the ToDo list notwithstanding the collaboration proposals. I intend to move forward on that soon (having spent the weekend thinking about the current proposal).
To recap, here’s the state of the matter as I gather it:
  • I propose to discard the first 7 revisions of the OpenUru.org repository and to replace them with the corresponding ones from the H-uru repository. (Note that this does not affect the “two precepts” rarified mentions, since the file trees are identical save for the .hgignore file.)
  • The OpenUru.org management is not opposed to this idea.
  • Rarified is willing to carry out the proposed action if a decision is made in favor of it.
  • In doing so, care is to be taken that author names are preserved. This means avoiding the use of Git, which does not fully support some names (those ending in apostrophes in particular), for certain operations.
  • Nobody knows of any potentially affected parties (with the exception below) that would need to be asked for their opinion. It can be assumed that any that may still exist are reading this and would give their objection here if they were opposed to the idea.
  • As an exception to the above, it is unknown whether Cyan would be affected, and they cannot be assumed to read this forum. Given consent, I will therefore contact Mark and ask.
  • The proposal is considered accepted on explicit approval by Cyan and no explicit objection on this forum within a timespan to be determined.
Does everyone agree with that?
After examining the differences between the current H-uru repository and the original branch from Cyan, I discovered the en-masse changes to the formatting of the code. I don't know the revision(s) (yet) where they were introduced, but they are a solid blocker to making those files a basis for the mainline at OpenUru. We must keep the differences from Cyan's code to the minimum necessary to add fixes or features in the default branch if we are going to minimize the work Cyan must do to qualify new code coming back from OpenUru. Even though as has been pointed out some mechanical translation can be done between formats, the results of performing that assume the original source was uniform in it's formatting and the results of the reverse transformation are identical to the original. We can't make that assumption (and equivalence here is in the context of a source management system, not linguistic equivalence).

The bottom line is I don't know yet how many revisions of the H-uru and OU repositories can be considered identical until this divergent change gets introduced. It will take some work, and time is in relatively short supply.

The alternative in our proposal is to simply start a new branch anchored from the root version where we import *all* the changesets from H-uru (and continue to do so). That way if you have an H-uru based repository and want to correlate it to OU, there are common change points, but they would be on the H-uru branch. Because both the OU "dev" branch and the "huru" branch are in the same repository, migrating a changeset from one branch to the other is made simpler by having the SCM tools available in a single repository.

Another reason I don't think we can simply rebuild the OU repository from H-uru is that other changes (primarily structural to accommodate Cmake, but some cosmetic such as the include path) probably aren't of interest to Cyan. (Barring a statement from them on the scope of acceptable changes we have to assume they desire a minimal set.) I also am not aware of any effort to verify backwards compatibility of the functional changes in the H-uru repository with the current 1.902 sources. As should be inferred by this time, a goal of our default branch is to have code that is readily usable in the existing game. Other branches (project-specific) or forks are for more radical changes, and that's fine. Our tools are still available to use against those as well.
If the proposal goes through, the “monster merge” prototyped by Branan becomes unnecessary and a good basis is established.

If the proposal is declined, the benefits and drawbacks of the “monster merge” need to be analyzed in greater detail.
I think we are at or beyond that point now. Unless you can suggest a way to get the SCMs to link files with formatting changes, beyond the parallel branch scheme I'm describing, I don't see an alternative.

_R
One of the OpenUru toolsmiths... a bookbinder.
Christian Walther
Member
Posts: 317
Joined: Sat Dec 13, 2008 10:54 am

Re: Incompatibility of OpenUru.org and H-uru CWE repositorie

Post by Christian Walther »

Rarified, you are misunderstanding me.
rarified wrote:After examining the differences between the current H-uru repository and the original branch from Cyan, I discovered the en-masse changes to the formatting of the code.
What did you compare? I’m talking about the first seven revisions. Did you compare those? They are identical, except for the presence or absence of an empty .hgignore file.
rarified wrote:We must keep the differences from Cyan's code to the minimum necessary to add fixes or features in the default branch if we are going to minimize the work Cyan must do to qualify new code coming back from OpenUru. Even though as has been pointed out some mechanical translation can be done between formats, the results of performing that assume the original source was uniform in it's formatting and the results of the reverse transformation are identical to the original. We can't make that assumption (and equivalence here is in the context of a source management system, not linguistic equivalence).
I agree with all that but it’s beside the point. I am not proposing to pull in the revisions where H-uru has diverged from OpenUru.org. Whether or not to do that is for a future discussion.
rarified wrote:The bottom line is I don't know yet how many revisions of the H-uru and OU repositories can be considered identical until this divergent change gets introduced. It will take some work, and time is in relatively short supply.
It is the first seven revisions. Those that the two repositories should already have in common if a mistake hadn’t happened in the initial conversion from Mercurial to Git.

Note that, as I said in the first post, I have already done the work. All you would need to do is to clone or pull from there. (But not before I fix a'moaca'’s name in r8.) I would encourage you to redo it on your own though so that we can compare our results and be sure we’ve done it right.
Post Reply

Return to “Management”