Onward: OpenUru Repository Reorganization

[rarified]

In A Nutshell - Immediate Technical And Use Concerns

What we did wrong:

• No anonymous access.
• No clone tracking.
• No obvious way to use our repositories.
• Repo structure was too clever and complicated.

What we're doing about it:

• Use of Bitbucket as a familiar and easy social repository hosting site - provides anonymous access, clone tracking and pull requests.
• Bitbucket synchronized mirrors of CWE, MOULSCRIPT and MOSS (pending) - provides flexible use of either Foundry or Bitbucket.
• Discrete compatible forks instead of one repo with multiple branches (CWE, CWE-ou, MOULSCRIPT, MOULSCRIPT-ou) - provides workflow agreeable to the GoW.

Where do we go from here?

I hope that one of the first words that comes to mind is forward. A lot of us have been giving this question thought, as has been evident from the numerous and passionate topic threads both here and in other venues. I hope with this posting we can do just that. We are presenting here a proposed restructuring of the OpenUru code repository to solve interoperability problems primarily raised by Guild of Writers members, but also to address some of the issues raised by other individual contributors.

I'd like to limit the discussion in this thread to just one element of the OpenUru environment, the Cyan Worlds Engine code repository (CWE). We'll try to stay on topic, be as concise as possible, and make decisions as quickly as we can. For an easier overview of our proposal I'll start with a summary of our current status, a proposed new repository organization, and a roadmap for implementation. A later posting will contain more detail on these summary items including background, alternatives, and yet to be implemented functionality. Anyone with concerns are encouraged to voice those concerns, as long as they are accompanied by alternative approaches.

Problems present in the current OpenUru code repository

1. No description on how to submit changes back to the project
   This may require some additional structure to the current repository
2. Anonymous access not currently enabled on the CWE repository
   This is different than the usual accepted convention for open source projects, leaving a perception of exclusivity and needless hurdles for getting involved.
3. It is hard to determine what version of CWE the repository(s) contain and what each should be used for
   This became evident as we tried to keep part of the repository synchronized with changes Cyan had incorporated from the open source community
4. There was no easy way to explore or locate projects based upon CWE
   This functionality is present in several public code hosting sites such as BitBucket or GitHub, and many find it useful. Also sometimes referred to as "clone tracking".
5. A repository workflow that appeared foreign and clumsy to those familiar with the Guild of Writers processes
   From the repository viewpoint, we were planning to use a Mercurial "push" paradigm to propagate changes. The Guild of Writers is more comfortable with the "pull" paradigm supported by the common public hosting sites for merging changes.
6. Early changes to the Guild of Writers CWE fork made it difficult to integrate their modifications back to the OpenUru repository
   Restructuring and other global changes introduced into the GoW repository, while desirable to move forward on their work, produced incompatibilities between the GoW repository and the OpenUru repository that make exchanging modifications difficult.

Christian Walther along with GoW members have investigated several remedies for the last item on the list, and we understand that the GoW repository has been adjusted to make interoperability much easier with the original repository structure. We appreciate their effort and with the implementation of their changes believe this issue to be resolved.

Proposed alterations to the OpenUru repository organization

• Changes to the original OpenUru CWE repository on the OpenUru Foundry
  • Split the CWE repository into two separate repositories: CWE, and CWE-ou, and eliminate the CWE-dev repository.
    Follows the "fork" paradigm rather than "named branching" as proposed by Hoikas
    Addresses issue #3
    We have accepted Hoikas' user-simplicity argument, and are therefore moving to the multi-repository paradigm instead of using named branches. The CWE repository will contain original CWE code, and only track changes that Cyan has accepted back from the community and incorporated into the MOULa servers. The CWE-ou repository will contain the OpenUru collection of CWE, with additions from the community.
    At one time we offered to instantiate a read-only mirror of the H'uru (GoW) repository on the OpenUru Foundry. One reason was to make it available in the Mercurial format (the H'uru repository is managed by Git). At the time, Branan declined the offer, but while we are reorganizing is a good time to verify that still is your preference.
• Addition of BitBucket as an OpenUru CWE repository hosting site
  • All variations of the CWE repositories present on the Foundry will be made available as BitBucket repositories
    Addresses issues #4 and #5
  • BitBucket repositories will have full anonymous read access
    Issue raised by Paradox, this relieves the wait to implement on the Foundry.
    Addresses issue #2
  • The BitBucket repositories and Foundry repositories will be kept in synchronization automatically
    A consequence of using BitBucket
  • Use BitBucket forks for individual projects and use the BitBucket "pull request" paradigm for integration back to CWE-ou
    Feature requested by Hoikas.
    Also
    Addresses issues #1 and #4
    BitBucket (and GitHub) both use a procedure called a pull request for performing an integration of the changes present in a fork into another repository. We will start using this for integrating changes into the communal CWE-ou repository. This will eliminate the need for the CWE-dev staging repository, and reduce the confusion when several changes are pending for integration. We still will provide a way to handle integration for those with repositories directly cloned from the Foundry repositories, but that process may appear more cumbersome to those familiar with BitBucket or GitHub. Using the BitBucket pull request will lower barriers both technical and procedural for integrating changes into CWE-ou.
    We'd like to discuss how best to handle pull requests in our workflow.

Roadmap for implementation of the repository changes
[While some of these tasks are already underway, they will be adjusted if the proposed changes need to be revised]

• Creation of the BitBucket OpenUru CWE and CWE-ou repositories
  The CWE-ou BitBucket repository has already been created and populated. The creation of the CWE repository is underway, some manipulation of the original repository is being performed to make the CWE repository compatible with forks taken from the CWE or CWE-ou repositories present on either the Foundry or BitBucket
• Renaming the original CWE Foundry repository to CWE-ou and creation of the CWE repository on the Foundry.
  Pending. All named branching will be removed from CWE-ou and the Cyan changes replicated in the new CWE Foundry repository, which will then be synchronized with the BitBucket CWE repository. Once that happens, existing clone repositories will need to take care to reference the new CWE-ou repository explicitly on pull and push requests, or change their repositories configuration to refer to the new CWE-ou repository as the clone's parent by default
• Documenting how to continue using an existing clone taken from the original Foundry repositories, with the new OpenUru repository organization
  Those clones that were obtained from the Foundry will continue to be usable with either the new Foundry organization, or to populate BitBucket forks of the CWE-ou repository by changing the default configuration of the parent repository in those clones. We'll include brief instructions in the detail posting, as well as information on the OpenUru website.
• Decommission and remove the Foundry CWE-dev repository
  I don't see any pending changes that have been pushed into CWE-dev, so it can be removed at any time.
• Reconfigure JIRA and Fisheye to refer to the new repositories
• Set up automated synchronization between the Foundry and BitBucket

Please take the time to look over the issues, and let us know if we've missed something related to repository organization. We'll start a new thread soon for discussing the integration process once the repository organization is settled. Remember, right now this is a proposal. Without your feedback, we risk making decisions and taking actions that may affect your use of OpenUru without knowing your needs.

_Rarified

[branan]

I think that looks excellent. No major comments from me (or from any of the other H'uru devs that I've spoken to).

This doesn't address issue 6, but I think that's best handled once the repositories are in a sane state, so I'll resurrect the other thread when that time comes.

Regarding handling pull requests - we just have our core group of devs that have direct access to the repository, and they also handle pull requests as they come up. Something similar would probably also be best for OU, once a strong development community around the OU repos develops. In the interim, I think someone who's been actively working on the codebase would be qualified to be a gatekeeper until it's clear who the "core" developers around the OU repos are.

EDIT: I've just had a thought regarding Jira: What do you think you're going to get out of re-targeting Jira at the Bitbucket repos that Bitbucket itself can't provide? We've had no problem using GitHub's issues for our work, both as a TODO list and for the few bug reports we've gotten so far. I'd suggest that unless there's a really compelling reason to use Jira that you make Bitbucket issues the primary public issue tracker. That way the code and issue handling is in a single clean interface.

[rarified]

I'm heading out the door in a few moments, but wanted to ask a question...

Do you see value in keeping issue/bug tracking data beyond the interval between when it is submitted and when a fix is committed to the code base?

_R

[Christian Walther]

Sounds good to me!

I have tentatively redone my Bitbucket CWE clone as a fork of CWE-ou. (It also contains the contributions I’ve made to the H-uru fork, and therefore necessarily all H-uru history that comes before them, so don’t be surprised at the number of revisions. Obviously not all of that is a proposed addition to CWE-ou.)

EDIT: I've just had a thought regarding Jira: What do you think you're going to get out of re-targeting Jira at the Bitbucket repos that Bitbucket itself can't provide? We've had no problem using GitHub's issues for our work, both as a TODO list and for the few bug reports we've gotten so far. I'd suggest that unless there's a really compelling reason to use Jira that you make Bitbucket issues the primary public issue tracker. That way the code and issue handling is in a single clean interface.

I think the focus was less on the issue tracking than on the code review tools connected to JIRA. Bitbucket currently appears to be somewhat lacking in that department, e.g. you can’t comment on a commit or on a diff hunk as you can on GitHub.

Do you see value in keeping issue/bug tracking data beyond the interval between when it is submitted and when a fix is committed to the code base?

Yes! History is important, that’s why we do version control! Unless all information from the issue tracker entry is incorporated into the commit message of the commit that closes it, the issue tracker entry can be an important part when trying to figure out why a certain line of code is the way it is.

Of course, in our case where we imported a huge code base without history, the only answer that history consultation can provide to that question will often be “because it was initially imported that way”, but that will change over time and we don’t want to make it harder than it needs to be.

[JWPlatt]

We should note our Bitbucket repo urls, and document them everywhere to be easily found, but for now they are:

CWE (MOULa): https://bitbucket.org/OpenUru_org/cwe
CWE-ou (OU fork): https://bitbucket.org/OpenUru_org/cwe-ou
MOULSCRIPT (MOULa): https://bitbucket.org/OpenUru_org/moulscript
MOULSCRIPT-ou (OU fork): https://bitbucket.org/OpenUru_org/moulscript-ou

The MOULSCRIPT repos are not yet populated, but should be soon. I hope to also get to MOSS soon.

EDIT: I've just had a thought regarding Jira: What do you think you're going to get out of re-targeting Jira at the Bitbucket repos that Bitbucket itself can't provide? We've had no problem using GitHub's issues for our work, both as a TODO list and for the few bug reports we've gotten so far. I'd suggest that unless there's a really compelling reason to use Jira that you make Bitbucket issues the primary public issue tracker. That way the code and issue handling is in a single clean interface.

Ultimately, I am very concerned with retaining history. If it turns out we can import issue tracking from Bitbucket to JIRA, no problem. I'd rather not split tracking between two sources (Foundry and Bitbucket), but it seems necessary for now. Bitbucket is not within our control, so I consider everything there to be at risk of loss due to business failure, change of focus, acquisition, etc. It happens and we are wise to have contingencies or protections for the slight possibility. But if it gets people contributing through the easier nature of Bitbucket issue tracking, that should be the priority - let people choose the tools they want. They will anyway. We'd really like to be able to sync Bitbucket issue tracking with Foundry. With Bitbucket being an Atlassian company, the chances of that are raised. And all of that feeds into the higher order services rarified would like to offer through Foundry, which he'll get to writing more about. Atlassian integration is one reason - besides fault tolerance for Foundry, support of Mercurial, and all your reasons for liking sites like github and Bitbucket - I've been eager to use Bitbucket.

Christian mentions that Bitbucket is lacking in review tools. That seems to be true. But there was a recent comment I found from someone in the know, probably a Bitbucket development or support staffer, to expect something along those lines within a few weeks. I hope it's true. But there's still the question of whether we'll ever be able to sync up our tools with Bitbucket's tracking.

[Mac_Fife]

EDIT: I've just had a thought regarding Jira: What do you think you're going to get out of re-targeting Jira at the Bitbucket repos that Bitbucket itself can't provide? We've had no problem using GitHub's issues for our work, both as a TODO list and for the few bug reports we've gotten so far. I'd suggest that unless there's a really compelling reason to use Jira that you make Bitbucket issues the primary public issue tracker. That way the code and issue handling is in a single clean interface.

That's probably worth a bit of consideration. My initial view would be that Jira was around and in use for other OU projects before CWE came along and there's a plausible argument for not stringing out issue tracking across multiple tools/locations. Depends on what is least confusing for users in the long run, and I don't know that the answer is particularly obvious right now.

[JWPlatt]

Very true. Our focus is not just CWE, but Uru projects generally, of which CWE is one. The tools are here to use by those who want them.

[rarified]

It has been a while since this thread was updated. Here's a status update of what has been done so far to align ourselves with the proposed organization:

BitBucket Repositories
A full complement of BitBucket repositories have been created that mirror the Foundry repositories. They now reflect the naming convention we proposed here:

• CWE The source matching Cyan's original release of the Cyan Worlds Engine (i.e. Game Client and Plasma) along with changes Cyan has accepted for MOULa
• CWE-ou The source for the Community release of CWE, incorporating changes that have been contributed to OpenUru
• MOULSCRIPT The source matching Cyan's original release of the Game "scripts" (i.e. executable code embedded in the game model) along with changes Cyan has accepted for MOULa
• MOULSCRIPT-ou Similar to CWE-ou, the Community release of MOULSCRIPT
• MOSS The open source game server provided by A'moaca' and CJKelly1.

All of these repositories can be found under the OpenUru Bitbucket account at https://bitbucket.org/OpenUru_org. They can be accessed anonymously either directly to your local repository, or forked to another BitBucket repository you want to set up. They are synchronized automatically with their corresponding repository on the Foundry (currently only once a day, but work is underway to resynchronize them after every change to either side).

Foundry Repositories

• The foundry repositories have been changed to permit anonymous read-only access for anyone to pull a clone or browse through the web interface. A new version of Mercurial is about to be installed with some improvements in the web source browser.
• The foundry repositories are still using the old naming conventions, and we are ready to rename them to match the new names used by the BitBucket repositories. The mapping and disposition of the foundry repositories will be as follows:
  • CWE is renamed to CWE-ou
    This is the repository containing the Community source for CWE
  • CWE-cyan is renamed to CWE
    This is the repository containing the Cyan release of CWE
  • MOULSCRIPT is replaced
    This repository was a branch-based merge of MOULSCRIPT with the MOULSCRIPT-dev contributions, and will be deprecated
  • MOULSCRIPT-cyan is renamed to MOULSCRIPT
    This repository will contain the CYAN release of the MOULSCRIPT code base
  • MOULSCRIPT-dev will be merged into MOULSCRIPT-ou and deprecated
    Changes that have been pending in MOULSCRIPT-dev will be promoted to MOULSCRIPT-ou, at which time MOULSCRIPT-dev will be deprecated in favor of the "pull request" model for incorporating changes using BitBucket
  • MOSS is unchanged

Note: Impact to existing clones taken from the Foundry repositories (at http://foundry.OpenUru.org/hg/<repo>)
Once the renames occur, existing local cloned repositories that were pulled from the Foundry will have an out of date configuration that refers to the old Foundry repository name. This problem can be addressed in either of two ways:

1. If you never made any changes to code in your copy of the repository, just start over and make a new clone from the correct Foundry or BitBucket repository that you are interested in
2. If you have made changes to code in your local repository and want to continue working in that repository, you will need to change the configuration file for your local repository to refer to the correct Foundry repository parent.
   • If you are using Mercurial directly, you need to manually edit the file ".hg/hgrc" located in the top directory of your repository. In that file you will see a line of text showing [text] followed very closely by a line starting with default = <something>. The <something> should look like https://foundry.OpenUru.org/hg/CWE. You need to change the CWE part of that line to the new name as shown in the list above.
   • For Windows users that use TortoiseHg, you need to do the same thing, but TortoiseHg gives you a way to edit the file within the TortoiseHg Workbench.
     • Open the TortoiseHg Workbench application, and on the left panel, left-mouse-click on the repository that needs to be changed.
     • On the "sync" toolbar, find the icon that corresponds to the Check for incoming changes from the selected URL activity and click it.
     • You will see a panel with the label Remote repository: .... In that panel, there will be a list of repository names under the heading Paths in Repository Settings. Find the line whose alias is "default", and click it.
     • Above that list there will be a set of text fields that can be changed, and they should show parts of the <something> text I mentioned above, including the /hg/<reponame> string.
     • Change <reponame> as documented in the list above (i.e. If you had cloned the CWE repository, you would change that now to CWE-ou.).
     • Click the "save" icon all the way to the right (the floppy disk), and you're set.
   In either case you should verify that you can reference the correct Foundry repository by using the "incoming" operation in Mercurial, if you use Mercurial from a command line, or on the TortoiseHg Workbench, click the icon that is labelled Check for incoming changes from the selected URL. I would not suggest attempting a "pull" from the Foundry until you have confirmed you have correctly changed the name of the foundry repository, and that the inbound set of changes looks reasonable (it should be a short list if any are shown at all).

Updates to the Foundry Atlassian tools
JIRA and Fisheye have been updated, with Fisheye having a major version update. The release notes indicate there were some performance improvements with Fisheye, please try it out and see if it is more usable now.

On the Horizon...
I'm about to launch the first of the Foundry services, which will be application of the Doxygen automatic documentation tool against the various source repositories. Doxygen (http://www.doxygen.org) is a tool to examine source code files for bits of documentation embedded in comments in the code, and together with an analysis of the actual code itself, produce a web document that is an neat way to navigate through the source code, and place notes about the behavior of interfaces and parts of the code that are non-obvious. It appeared from examining the CWE code that at one time parts of the game engine had been documented with Doxygen, but it never was carried forward to the whole CWE code base.

An example run of Doxygen against the original CWE code base can be found here. This was a very unsophisticated application of Doxygen against the entire code base, so classes for different packages are mingled together, and can get confusing. But it provides a sample of what can be done.

I would like to extend an invitation to the fan developers who have spent so much time unraveling the operation of Plasma to share their knowledge with additions to the Doxygen documentation of CWE. It will certainly help a broader audience understand the fundamentals and limitations of Plasma in a way that does not place a continual demand on your time to answer questions.

Please post questions or concerns about the final repository renames, or anything else I've mentioned here. I've not set a time for the final rename, but probably will set a target date after this weekend.

_R

[Christian Walther]

Thanks for your work, rarified!

I have tested the anonymous access – glad you got that working.

[JWPlatt]

What next can we do for the development community? I'm particularly interested in hearing any remaining constructive criticisms from the Guild of Writers, especially if paired with proposed solutions.

I'm aware that licensing is a big concern. We're already determined to work on that, so no need to convince us there.