Work Flow Management

[DarK]

I am still catching up with things, but what I have managed to piece together is in documents scattered everywhere – in forum threads, wiki articles and on the foundry systems. This is not helping my work flow coming into the project, nor can I see it helping anyone else’s work flow in terms of fire fighting.

From my point of view, the repo issue is part of the CWE project and a key milestone - the problem is that I can't figure out at what stage we are at without spending days reading forum posts and mapping out key action from those discussions, then having to keep on top of the thread to see if the issue has moved on.

The other biggest issue I am having is finding a way to transpose discussion into issues, and then make them visibly seen so people can contribute without having to spend time becoming intimate with JIRA or the forums.

I am looking at confluence as a way to solve the issues above. Its feature rich, and would help integrate the raw code development with the management and documentation areas of the project.

http://www.atlassian.com/software/confl ... ftware.jsp

Using the following work flow with confluence may help people with communication:

Work flow:

Document is created when a discussion has been initiated on the forums, issues in JIRA need collaborative discussion to resolve, or formal documentation is required.

People are tagged in the document once they have participated in the discussion so that they can receive notifications of updates.

Document Templates are used to define layout so that formality is maintained to ease accessibility

Documents head through defined stages once opened – as an example I will use an issue raised via JIRA that needs collaborative discussion

Define – Problem definition layout with condensed discussion history if required

Design – key action points from the discussion on how to resolve the issue

Formalise – Final actions are agreed in consensus with those involved

Raised – actions in JIRA

Review – solution does not solve problem definition and review is needed

Or

Resolved – All issues in JIRA closed and definition resolved

This process has some key tradeoffs with that it is particularly self documenting, and that people can view progress almost immediately, with the additional tools in confluence.

You would not need to create a page for every issue that arrived in JIRA, just the relevant discussions that are needed to progress a milestone.

I’m still working this out, but it’s getting late here and I wanted to get something out for people to look at so I can get some feedback to see if it is worth continuing with.

Is it possible to get confluence set up at all ?!

[rarified]

DarK

Are there particular attribute(s) of Confluence that address the functions you describe, and that the existing Wiki does not?

We have the ability to set up Confluence, but had stayed with the exiting Wiki instance as it predated our acquiring Atlassian licenses for OU to use their tools. I'm not a wiki wrangler like Mac_Fife is, he could elaborate on the capabilities of the current Wiki software.

_R

[Mac_Fife]

@DarK: We have a license from Atlassian for Confluence so getting that set up is quite possible, if it fits some particular bill. I'm not a Confluence "expert" by any means, so if there is some additional integration features with the other Atlassian tools that would be of benefit that we can't easily get out of MediaWiki, then I'd be keen to hear about them.

We had a discussion about using Confluence as an interim wiki while we were preparing the open source release (you can kind of jump into the middle of the discussion here: viewtopic.php?p=3663#p3663). In that particular case, the intent was for the content to be migrated over into the main MediaWiki, and due to differences in the markup language between Confluence and MW we decided Confluence was too much hassle in that particular case. But we haven't ruled it out for other things: If there are code change/review discussions or similar that would work better in Confluence and don't really need to be routinely migrated to the main wiki, then we can do that, with the caveat that we need to be clear about what content goes in which wiki, as we couldn't really port everything we currently have from MW into Confluence.

[JWPlatt]

DarK,

In my ToDo notes (in a locked ToDo thread from prior to OS), I mention http://www.open.collab.net/community/subversion as an example of how to introduce and educate folks to help avoid the kind of confusion you are going through in your research. This means web pages with lots of graphical content and rich in well-organized information. Maybe that's Confluence, but there is also a need to put that kind of thing right up front on a website or at least one obvious click away. But there's only so much a bunch of engineers can do. We're counting on community participation of folks like you with the kind of talent you and others might have to produce a good, organized presentation or inspire more people to contribute in ways other than programming. Part of the plan here is to encourage people to emerge to either lead or do the work. There's no way around that. The biggest block will be convincing people it is up to them to contribute and that it won't happen automatically. Many may not even realize what they have to offer if they assume it's all just programming.

Others have tried to distill forum discussions into something presentable. It's very difficult. We need to learn that forums should be treated more like email, or texting, than a knowledgebase. To do that, we need to make the workflow easier than forums. If Confluence can do that, cool. We need to populate it and funnel people there, like a train ride through the workflow, but perhaps after given a web page of consolidated, summary information and places to go.

But that's not possible without contributors. I see a lot of folks expecting others to do the work, but it won't happen without those folks.

Maybe the first order of business is to make Confluence easier for you and others to find on the existing site.

[DarK]

Looking at your responses, I think I might be stepping too far forwards, and how I see various data points and systems, might not be how other people are seeing them.

For instance, the Wiki seems to be a front of house presentation tool, instead of a technical data store based on your posts.

Since a Picture says a thousand words aparently , I’ve produced some Data Flow Diagrams that will hopefully demonstrate what I am trying to achieve, and why confluence may be something to assist at least technically, to plan releases.

Colour Code:
Black : Already implemented
Red : Proposed
Orange : Question Marked process – unsure if it actually happens
Green : Automation

Bear in mind when reading these diagrams that an actual developer can be in all entity roles Community, testing, developer, planner and technical writer. The DFD’s may not be 100% accurate and may not fully reflect the actual process, so any feedback on changes is welcome.



This demonstrates the current work flow, from what I can tell with regards to CWE. As you can see it’s a detached model.

JIRA is the core of the development, yet only developers and testers feed information into it and nothing actually comes out at the moment. So developers are feeding information into a black hole!

Finally data is been stored in locations that are dead ends, they are not been utilised for the data they actually store.



This is a proposed work flow for CWE, as you can see entities and data stores are connected to one another and information is utilised by all to feed each other’s processes.

Based on this diagram the missing links are coordinating JIRA and technical information on the Wiki to produce information towards release.
Note: Process 6 has a larger remit in terms of inputs and outputs; however we are quite a way from deciding what constitutes a release yet and how it is decided.



This actually shows the recent repo discussion and the missing piece in that puzzle in terms of information flow, once commit details are passing into JIRA there is a reason to use it!



Finally, this shows the analysis process and how utilising forum discussion as an input can result in a direction for development, and documents to support it.

So how does confluence help?

It removes the need for commits to go via the technical writers, meaning that people can instantly get updated on the issues outstanding against current issues.

Planners will otherwise need to use JIRA directly, to filter out and build reports for proposed changes and then act upon them when trying to release (Process 6). However all planners really need is the status of the outlined issues from JIRA at that stage, which has automatically been produced by the automation.

This frees up the technical writers to actually be able to do their job, writing technical documents and documenting change from commit data

@JW – I understand what you mean in terms of man power and that not all jobs require you to write code.

Before the masses show up however, they needs to be at least some form of initial ground work in how they are to integrate into the rest of the development cycle, hopefully these DFD’s will go some way to defining a process that people can get involved in, particularly in relation to testing, documentation and analysis which are none code based tasks.

Edit: Original PPT is here : http://www.thedarkcloud.co.uk/images/OpenUru-DFD.ppt

[Mac_Fife]

Thanks for the effort you've put in here DarK. I need to take a little more time to absorb your later diagrams and in particular to understand the role of the Planner in your DFDs. Right now, my head's like mush and nothing is going in (I'm blaming the pollen from the rapeseed fields).

Initially, I was going to argue a little over your first diagram, but in hindsight it's probably correct. And in any case, it's the diagrams moving forward that are more important. I think I generally agree with the process model in your second diagram; it maybe depends on how fine-grained you want to make it, and whether you break out JIRA's component parts like Fisheye and Crucible.

But the main point was to let you know that what you've offered up here isn't being ignored .

[DarK]

No problem

The first diagram is what I have actually observed happening in terms of information exchange - as the recent repo development as an example.

The changes should be logged in JIRA at least, but by the looks of things rarified has enough on his plate as it is, without worrying about procedure on keeping a log of his actions.

Planners are hopefully going to be able to log change via forum discussion/other methods and produce issues in JIRA with regards to these discussions.

On the diagrams, it could be possible that JIRA could be replaced with Foundry if it makes more sense - I see JIRA as something that feeds everything else in terms of information.

Also these are only level 2 diagrams - only the proposed planning diagram is level 3, so you won't see individual components as such.

I think once the repo issue has settled down and the key contributors (GoW, H'uru etc) decide what is best for them, I can map process 4 out a bit more to show more detail.

Ideally though I want to remove the Forum and have a formal submittal process via some more manageable means of communication and documentation, mailing list - patch submission - repo pull requests etc

If openUru - CWE moves to a submittal process, unless there has already been some form of automated agreement where everything from a specific source is accepted - a developer will have to make a request and get the issue in JIRA before a commit can occur.

Swinging by the "social acceptance" lines - I'm not sure how such a process would be received by the masses, as a barrier?!

Again I agree that it’s not the current diagrams that are 100% important - it’s the progressive ones and how processes change over time.

[Mac_Fife]

"Foundry" is a term with too broad a scope to use in these diagrams: The Foundry includes the Atlassian tools (JIRA, Fisheye, Crucible), the repos and the Jenkins build engine. Basically where you're headed is to keep all the development communication within the tools that comprise the "Foundry" sub-domain and keep it away from the "front of house" forums and wiki. That's logical.

You won't be able to stop people discussing bugs and solutions on the general forums, so I think we have to accept that it will remain a source of information, albeit some kind of ad hoc input into another tool. As I said, we haven't looked too hard at how Confluence fits in with the other Atlassian tools at this point. What we had done, once an issue was recorded in JIRA, was to use comments to discuss the issue. The comments meant that the discussion was "in your face" when you looked at the issue record but maybe Confluence is a better place to have those discussions.

We need a very top level, simple, process map that goes onto the main wiki as a kind of "What do you want to do? OK, go here". Underneath, it needs to all hang together, but we need to get simple process descriptions for individual workflow steps, e.g. "How do I submit a bug?", "How do I claim and issue to work on it?", How do I submit my changes?", etc.

As an example, one thing I've been thinking about is the bug reporting: The reporter may recognise that it is a bug but not understand the nature of it (is the problem in the client, the server or the age?) which could make it hard to create the issue in the "right" project. OK, they can be moved around if necessary, but now I see why Cyan have the support ticket system as a front end. Record standard information, as minimum, for every report into a central pool, let someone with a bit of architecture knowledge validate and categorise the reports and convert them into JIRA issues in the appropriate project. This seems to be part of role of your Planner. The underlying mechanics will have some influence on what a particular role does in a particular process.

I'm feeling that some of the labels on the data flows may need to be refined slightly: We have "Issue Definition" and "Commit Details" in several places, but I think there may be subtle differences depending on the path we're considering.

[DarK]

"Foundry" is a term with too broad a scope to use in these diagrams: The Foundry includes the Atlassian tools (JIRA, Fisheye, Crucible), the repos and the Jenkins build engine. Basically where you're headed is to keep all the development communication within the tools that comprise the "Foundry" sub-domain and keep it away from the "front of house" forums and wiki. That's logical.

Yes - this tends to keep the emotional context that can sometimes come across from forums away from development. While disagreements will happen, having this kept out of data systems improves them.

You won't be able to stop people discussing bugs and solutions on the general forums, so I think we have to accept that it will remain a source of information, albeit some kind of ad hoc input into another tool.

Hopefully we can have some success with people having a discussion and inputting information via either a ticketed issue with testers, or directly on some form/wiki template for discussion to formalise into data that we can use for planning and development

As I said, we haven't looked too hard at how Confluence fits in with the other Atlassian tools at this point. What we had done, once an issue was rcecorded in JIRA, was to use comments to discuss the issue. The comments meant that the discussion was "in your face" when you looked at the issue record but maybe Confluence is a better place to have those discussions.

I tend to see comments as a sort of developer related chat - so a few people working on an issue leaving notes about what was done and what was committed, similar to what we see in JIRA now - Comments like: hack to tide us over - needs further development, can be picked up as something needing to be fed back into planning process. JIRA currently has no commit data, making it hard to see what code fixed the issue, or if there was any outstanding issues that still need addressing.

We need a very top level, simple, process map that goes onto the main wiki as a kind of "What do you want to do? OK, go here". Underneath, it needs to all hang together, but we need to get simple process descriptions for individual workflow steps, e.g. "How do I submit a bug?", "How do I claim and issue to work on it?", How do I submit my changes?", etc.

I will need to dig my old lecture notes out for that , not done a process map for a while. The reason for DFD's initially are that there easier to track information on than making on the spot decisions about processes. We know for example that issues go into JIRA but who needs, uses, manipulates that information and at what stage. Initially it’s a lot easier question to answer than "how do we submit a bug".

As an example, one thing I've been thinking about is the bug reporting: The reporter may recognise that it is a bug but not understand the nature of it (is the problem in the client, the server or the age?) which could make it hard to create the issue in the "right" project. OK, they can be moved around if necessary, but now I see why Cyan have the support ticket system as a front end. Record standard information, as minimum, for every report into a central pool, let someone with a bit of architecture knowledge validate and categorise the reports and convert them into JIRA issues in the appropriate project. This seems to be part of role of your Planner. The underlying mechanics will have some influence on what a particular role does in a particular process.

This is similar, if not the same as what I was thinking for Process 2, not sure about the planners as an entity been directly involved, but certainly the raw information can give some clues as to what needs most urgent action where from a client perspective. (That’s not me saying I will not answer tickets btw )

We have to remember that most of developers while moving to some similar goals will be scattered on what they are working on. so while some will be swatting critical bugs - others maybe working on less critical features.

If this was a commercial venture, there would be no question that everyone would be working on critical bugs, as this is open source we may need to make allowances for this in planning.

I'm feeling that some of the labels on the data flows may need to be refined slightly: We have "Issue Definition" and "Commit Details" in several places, but I think there may be subtle differences depending on the path we're considering.

I agree - in general terms of the information the tags are correct - but if we get down to level 4 DFD's there should be specifics on what exactly each information flow is - such as "Issue definition - assigned" to indicate that a developer has picked up the ticket and has altered the issue to be their responsibility.

I’ll attempt a testing level 3 (process 2), as I think I might have enough information on how that process will work – at least from the players point of view.

[Mac_Fife]

I will need to dig my old lecture notes out for that , not done a process map for a while. The reason for DFD's initially are ...

No, I understand the need for the DFDs. We need to have the mechanics of how the users and the tools interact with data before we can create meaningful process maps. Hopefully, things will be largely intuitive, but if they aren't then we need to be able walk people through the processes/tasks. My experience is that sometimes even technically smart people can get lost trying to use relatively simple tools.

This is kind of back to front compared to a traditional top-down approach, but we're not designing tools to fit a particular set of processes, we're trying to determine the best way to utilise the tools that we have for the tasks that we expect to perform. There's a bit-of top-down and a bit more of bottom-up involved here, I think.