OpenUru.org

This is my short list of things to get the ball rolling.

1) Get a list of people who should have commit access, and get it to them ASAP

2) Get a documented process for people who aren't on that list to get their code to the people who are. This could be a bitbucket repo that can be forked and have pull requests applied against it, or it could be something local where patches can be pushed. Attaching the results of 'hg export' to a ticket is always an option, though kind of clunky.

3) Decide on a method for feature-branches, for long running projects that eventually should be merged. Things like cross-platform porting efforts. These sorts of branches need to be first-class citizens within the project IMO, as such eventual-merge projects will hopefully be the main development focuses. If bitbucket is chosen for (2) this is mostly redundant.

And most importantly, whatever processes that are decided upon need to be documented. OU has a bit of a reputation for being hard to navigate and hard to figure out. Hopefully with the CWE project that can be sorted out.

Beyond that, I think it's important to make it easy for people to track other forks. Independent development is good, and obviouysly not all changes and features can (or should) be merged back into the OU master. What OU can do here is try to make it as easy as possible for people to see what forks there are, and to pull changes from those forks when they want to. Keeping independent development without causing fragmentation in the developer community is a careful balance. Using a site like bitbucket for most of the forking/integration work makes this easy, as it provides a list of forks.

branan wrote:OU has a bit of a reputation for being hard to navigate and hard to figure out. Hopefully with the CWE project that can be sorted out.

Deservedly so. But to go a little off topic since you brought it up - we will address your more important points (and I think you have more good ones to come) - but that was before we implemented site navigation. Before we developed our masthead and menus, it was difficult. That's why we installed site nav as part of our open source release. If the site is still difficult, please do let us know why, where and - especially important - how you think we can fix it. That goes for anything, not just the little stuff. (Suggestions about navigation are best made in the "Domain Development" forum near the top of the main index.)

We're a little tired right now, so we're saving the important answers for when we've gotten some rest.

JWPlatt wrote:We're a little tired right now, so we're saving the important answers for when we've gotten some rest.

That's fair.

Oh, and I've got a

4) How will we deal with getting code changes back to cyan?

But obviously that one depends a lot on Cyan and probably isn't up to anyone else.

branan wrote:4) How will we deal with getting code changes back to cyan?

But obviously that one depends a lot on Cyan and probably isn't up to anyone else.

Yes, and it will probably take a bit of time for an answer to bubble up. The question has been asked of Cyan and the initial response was along the lines of "That's something we probably want to be able to do down the road, but we haven't thought too hard on it yet."

A gentle prod every now and then to make sure the question isn't forgotten is probably in order .

Branan,

Please pass the word to folks you know that to get ideas across and have them discussed, it's going to make things a lot more streamlined by having that discussion here. I see you have put yourself in for consideration as the GoW rep. We'll need people like you to manage that.

One of my primary concerns about setting up our repo was to get clones tracked and make that process easier. Bitbucket does that, and I'm trying to promote use of Bitbucket for cloning, so a mirror there is definitely not out of the question. We can point our Foundry tools anywhere, and we want them used. But it needs to get discussion here for the team to consider and everyone to have visible input. That's why these forums, including this management forum, are completely open (to non-spammers). We simply can't visit everywhere, or be expected to respond or be held responsible for things we don't hear. Please be assured, and convey, professionalism and positive, constructive contributions such as yours are promoted here and appreciated.

Branan,

Ok, I'd like us to commit to discussiong the issues you've raised. Essentially, it's about cloning, tracking, efficiency, and a commit path back to Cyan. I agree with this focus. I would like you to keep in mind we're going to want the Foundry in there with everything it offers in terms of all the Atlassian project management and Jenkins Continuous Integration tools we have for tracking issues and automating your builds. Also don't be surprised if more people show up with their own plans to sort out. So this extends to anyone who wants a voice.

Let's discuss this as a team and see how to proceed. What I'm going to ask you to do is - rather than us telling you what we can do, I'd like you to lay out in some detail what you want us to do, then we'll discuss whether it works, or how to make it work. You've discussed in general terms what you'd like. We need to see your philosophy in enough detail to implement it. Good?

Personally, I think the Atlassian tools are unintuitive, and something of a barrier to entry for casual developers. I think they have a place in the workflow, but they can't be the focus of it. What I'd propose is something like this:

Setup a bitbucket repo. This will be the official location for all development forks and branches. We'll expect this to be forked on bitbucket, and we'll accept pull requests as they come in. For these pull requests, we'll use a "is this acceptable for the general developer community" criteria. We stick to the bitbucket tools for handling code review at this level. I would suggest that we have at least one guideline for developers forking on bitbucket: They should use "hg branch" to give their work a unique branch name, so its visible in the main repo where we've merged the code from.

At OU, we should have two repos: cwe and cwe-nextgen. cwe-nextgen will be our "next generation" version of the engine. Network protocol and PRP file format changes are acceptable here, as long as MOSS is kept in sync. This should be the version someone creating a new game using CWE should use, and perhaps any shard admins that want to try and convert Cyan's data (We'll need legal advice from Cyan on whether converting the assets to future CWE PRP formats is allowed before we actively encourage that, obviously). cwe will be a repo that's still compatible with Cyan's code and assets: essentially bugfixes only, though some features might be acceptable in there. cwe-nextgen should be considered "shard stable", at any time. That is, by the time a change makes it to that repo, it's been tested at bitbucket and undergone a thorough code review. (maybe name them cwe-moula and cwe instead, I don't really care)

That's where the Atlassian tools come in. We'd use them internally to conduct the final, thorough review before any code is moved from bitbucket to cwe or cwe-nextgen. I'm not sure of the full scope of what tools you have available or what they can do, so I'd like input on what sorts of things we can or should suggest the core maintainers do at this stage.

This keeps the barrier to entry for casual developers low (bitbucket forks and pull requests are dirt simple) while still giving us the power of the atlassian tools to allow us to maintain a stable repository for serious projects using CWE. You can consider bitbucket to be the "unstable" branch, cwe-nextgen to be the "stable", and cwe exists basically because we will always need to support MOUL shards.

My preference to move focus away from the Atlassian tools is one I make from real-world experience: The KDevelop project is one I occasionally commit code to, and they've been moving towards git+reviewboard. I've seen lots of new developers be confused by what to do, where to sign up, and what sorts of access they have to different repositories and tools. I'd like to avoid that problem here if at all possible, and keeping the Atlassian tools available for the core maintainers while providing a simpler path for casual developers seems a good compromise to me.

I think a similar approach can be used for the SDL and Python files as well, as soon as those are extracted from scripts.zip and put into repositories. IMO there should be two repos - Python and SDL are different sorts of data and keeping the logs mingled makes little sense to me. For Python and SDL we should encourage shard owners to fork on bitbucket even if they're not going to be pushing changes back to us, so their contributions are available for other shard admins to use. I'd like to see many such shard-forks that admins can pick from and merge as they desire.

On first reading, that seems like a pretty reasonable set of suggestions.

On the PRPs, we don't really know what the status is going be on changing them, so that needs to be explored.

JW has licenses for (I think) all the Atlassian tools but we don't necessarily have them all installed right now - JIRA for bug tracking, Fisheye (source browsing), Crucible (code review) and the Confluence wiki are all there right now that I know of.

Rarified, with his extensive knowledge on Mercurial, will hopefully be along shortly to comment on the repo position, and JW will no doubt respond more fully on your post once he gets some time later.

Hi Branan, this is a good list, and all are essential. Let me offer my perspective on items here, but perhaps not in the order you listed.

Lets start with this one, which really impacts an architecture of the community resources that may be hard to change once in place:

3) Decide on a method for feature-branches, for long running projects that eventually should be merged. Things like cross-platform porting efforts. These sorts of branches need to be first-class citizens within the project IMO, as such eventual-merge projects will hopefully be the main development focuses. If bitbucket is chosen for (2) this is mostly redundant.

I agree we'll have to expect to have forks or branches with subgoals in mind, and a hierarchy of cloned repositories is the way to go. Using Bitbucket does make seeing the relationship between repositories more visible than an arbitrary repository graph. That happens because all the repos are on the same host; Mercurial itself doesn't really let the source repo used to clone a repository know much about the clone.

I am, however, motivated to keep a local repository representative of each of these subprojects on the Foundry, independent of whether or not it's the one used by most users to clone from. The reason being that it will be much more efficient for automated tools on the Foundry to retrieve artifacts from a local repo than one far away. Many of the tools poll the repository (i.e. are not event driven) so the closer the better. As an aside for your benefit, none of the tools on the foundry are limited to operating on a local repository (at least the current set of tools). They all can, if need be, use remote repositories. So Fisheye/Crucible can be configured to provide review services on a project's staging repo on Bitbucket. As can the automated build or documentation tools (we we can create Doxygen documentation from one of those Bitbucket repositories).

I'd like to hear the specific benefits you believe will come from using the Bitbucket review system. I havn't used it myself, and I do know the Fisheye and Crucible interfaces are not obvious (nor efficient, on a client's browser, as we've found out). I know you listed ease of use/low barrier as one benefit. Others? Something I'm looking for which steers me toward the Fisheye solution, is to keep as complete a history of design and review decisions as is possible. If we split lower level reviews off to Bitbucket, and then final level reviews to Fisheye/Crucible, don't we lose a record of the refinements that happened during the early stages? And isn't that sort of information useful to both newcomers to see how things happen, and long timers to go back and rediscover why something ended up being implemented as it currently is? I don't know if BB's review data can be imported to Fisheye, I'll start looking if you don't know.

Related to this is asking: What if anything can we do to our Atlassian installation and configuration to make it as approachable as possible? On unspoken goal we've had is to try to keep tool installation as vanilla as possible while making it serve it's purpose. It doesn't make sense to spend resources reconfiguring tools time and time again when they need to be updated. But if a tool is intimidating because of the complexity, and we can improve that with some slight customizations, I'm ok with that. Remember, our goal is to support Uru development, not to do system maintenance for the fun of it.

Ok, on to

2) Get a documented process for people who aren't on that list to get their code to the people who are. This could be a bitbucket repo that can be forked and have pull requests applied against it, or it could be something local where patches can be pushed. Attaching the results of 'hg export' to a ticket is always an option, though kind of clunky.

Some of this I think I addressed in the first item, but again, let's see where you're coming from. How has the PlasmaClient community been handling this situation? I think Zrax is hosting it, right? Do you folks do any reviewing before developers push? Is it formalized somewhere? We obviously opened the top level CWE repo for read only access to get the code out as quickly as possible. My initial instinct is to have people pull from any CWE repo (the one at OpenUru, or a direct Bitbucket clone) to start their projects. They can work directly there, or encourage individual contributors to work in their own clone. When it comes time to start consolidating contributions to the "project" a "review" clone from the project repo is either created or synced with the project repo. The contributer is granted (or already has) write access to the review repo, and pushes his change there (coordinated with other changes as needed). The review tool can look at the changeset in the review repo, record review comments and eventually a decision about the changeset. At which time the review repo is pushed back up the path to the top level project repo.

I like the additional level of category for "CWE-current", "CWE-nextgen", etc. That makes it easier to keep track of different build and execution environments expected as CWE (or others) get modernized or augmented.

Now for the hard ones (at least needing a little tact):

1) Get a list of people who should have commit access, and get it to them ASAP
...
And most importantly, whatever processes that are decided upon need to be documented. OU has a bit of a reputation for being hard to navigate and hard to figure out. Hopefully with the CWE project that can be sorted out.

Here is where we need to tread lightly. We've tenatively asked A'moaca' and cjkelly1 to act as the referees to the top level repositories, since they either wrote it (MOSS) or worked on it enough to get it running (CWE). There are no more experienced developers outside Cyan.

But we run the risk in the case of CWE of OpenUru "dictating policy" by putting something in place unilaterally. What method would you use to (at first) solicit a field of reviewers to do reviews (maybe partitioned into topical areas), and (eventually) identify the qualified individuals to fill the final roles as Gatekeepers? Would that method be acceptable to the breadth of contributors (or at least the ones you're familiar with)?

Same issue with defining processes. Shall we just post one, try it out, and refine it? Process is easier to change and refine than moving repositories around, so I'm not so concerned with getting it perfect out of the box. But I would like something that involves reviewing code, meeting some minimal requirements to preserve things like compatibility and documentation standards. That sort of thing which is needed to improve the code base. So I'm less inclined to start in complete trusting mode.
(Personal impressions: I think Cyan is letting us manage the derivatives of CWE as we see beneficial, they'll decide on their own what to pick up and feed back into their internal development. I'm going to be deferential to A'moaca' and cjkelly1 with regards to MOSS. They'll need to be consulted in planning process for MOSS unless or until they give their blessing to a community process. And the same for other new or derived projects, I look at my OU role as that of a trustee providing services to others, rather than an owner myself.)

Lets start there... reactions? Refinements?

_R

Wow, that's a lot to read through.

The main benefit with the bitbucket system is ease of use for developers. Its history tracking for merged pull requests is minimal at best. If you feel it's important to have a review tool with a record of those sorts of things, then it's inadequate. I don't think that's necessary. It's better, IMO, to use the built-in tools available in the VCS (like mercurial's persistent branch names), and in cases where it's a cpde design point rather than a history one, anything weird should be well commented.

I honestly don't know what you can do to make the Atlassian tools more approachable. They're designed for a corporate environment, and I think it shows. If you're able to sit down and train a new developer with them, they're very very powerful. If you want developers to be able to show up and contribute, they're unintuitive and a barrier. This is the main reason I tend to disagree with the choice to use enterprise-targeted tools for open source projects in general.

With regards to early developers - I think you underestimate how familiar some of us are with Cyan's codebase . Paradox, Hoikas, Zrax, and myself are all fully qualified to watch over the repository. I can't speak for the others and what interest they might have in doing so, though. Beyond that, qualified developers tend to appear, and you'll know them when you see them. With PlasmaClient, I've had a policy of "show me a couple good patches and you get commit access". That's probably not a good policy for a mature codebase like this one, but I encourage policies that involve as little hassle as possible.

On the subject of PlasmaClient: the review process there has been fairly informal - mostly pastebinned code snippets and IRC discussions. What I've tried to do instead of a massive formal review process is to document with code comments why something non-obvious was done. To me this is a much more effective way of tracking decision making than a separate tool, as I work with the code every single day.


Thinking through things, and reading what you've said, here's one possible scenario:

A developer realizes how to fix the Garrison wall in MOULagain! She goes to bitbucket, and creates a fork of the cwe-compat repo. She clones her repo, and runs hg branch GarrisonWallFixes. She hacks on it for a little while, and gets it working succesfully. She pushes her changes back to bitbucket. Next, she sends us a pull request through the bitbucket interface.

I, as a maintainer, see that request. I look over the code and don't see any obvious flaws with it. I clone her repository, and push it to a (temporary) repository on Foundry. I start a new review request based on that repository. Since her email address is in her commits, her Foundry account is automatically given push access to the review repository. If she doesn't have a foundry account, an email is sent to her advising her that her changeset is being considered, and she should sign up.

The review is done, and there are some minor nitpicks. She fixes those and pushes them directly to the review repository on Foundry. When the reviewers are satisfied, a maintainer merges her GarrisonWallFixes branch with the cwe-compat tip.

At this point, she's now a known developer, with code in the repository. Rather than having to go through bitbucket, she can now choose to create her work branches directly on Foundry, and initiate review requests when she's ready. Of course she's still free to fork on BitBucket, if that's her preferred workflow.

In short: we get the benefits of the easy fork/pull system in BitBucket, along with the powerful Atlassian review tools. The tough part will be the server-side integration code to make all that work correctly. Spinning up clones on demand and whatnot. It ought not to be complicated code, but it needs to be integrated well with the permissions system and it needs to be performant.

This workflow is intentionally similar to the standard "email patches then get commit access" pattern that many open source projects use. It replaces email with the initial bitbucket clone, and raw commit access with local-fork privileges. One of the advantages of a DVCS is that direct commit access can be kept to a minimum, and we can force developers to go through a review/merge process.

I also think this workflow sucks. It's a lot of work for the maintainers, and its a lot of work for developers. I'd personally prefer a workflow where if someone comes with a good patch, we give them commit access and let them refine it *without* being babysat. Worst case we need to push in a revert commit. In my experience such cock-ups are rare, and it's overall less work for the maintainers than ubiquitous review. In the end, I'm just not a fan of ubiquitous review-everything-before-it-hits-mainline policies. You guys seem to be big on them, and I'm trying to find ways to make that as streamlined as possible... but there's just no easy way to do it.


EDIT: I just realized I missed some of the other Atlassian benefits... the CI stuff could prove to be pretty excellent, especially once CWE has been ported to multiple platforms. I've had a couple issues with breaking Mac OS in PlasmaClient, and a CI system with mac support would have caught those sooner. I think this is something we can benefit from *without* the need for the rest of the crucible/fisheye stuff. It has a lot of immediate benefits with the drawbacks that ubiquitous review systems can have for developers.