Iterative and incremental development (IID) is the established practice for dividing software projects in time. IID is often interpreted to mean a succession of complete development cycles, meaning that each iteration includes within its temporal boundaries the analysis, design, implementation, and deployment of the features produced in that iteration.

But practically speaking, some analysis, and some interaction design generally precedes the bulk of the program design and construction work in each iteration. Customers don’t normally come to the planning game of an iteration with a blank slate. They do at least some homework on functions they need built, and on how users might interact with the system to access those functions. In fact, by the time we get to the planning game, there may well be a draft PRD outlining, albeit briefly, the required functions and user interactions for that iteration. I call the analysis of required functions and the design of the associated user interactions, functional design.

If we accept that it makes sense for functional design work to be front-loaded for each iteration, it is then a short step from there to a two-stage pipeline that includes a functional design stage, and a development stage. Because different sets of people generally do functional design and development, and because it makes sense for much of functional design to precede development for each iteration, the activities of functional design and development may, to a large extent, be parallelized in a two-stage pipeline.

In a recent project we put in place this kind of a pipeline, and we found it worked quite well for us in streamlining our work. During the development of the n’th iteration, the PRD for the n+1’st iteration would be researched and produced.

The practice of pipelining functional design and development is not uncommon in IID projects. But there is concern that pipelining can lead to disruptive cross-iteration interruptions. For example, while the functional designers are busy designing the functions of the next iteration, the developers need bandwidth from them for the clarification of the functions of this iteration. In our experience, interruptions in this direction, while frequent, were welcomed.

Interruptions in the other direction turn out to be more serious. Let’s see why, and consider how to reduce their impact.

As our functional designers come up with the functions of the next iteration, it is necessary at the same time to consider the technical feasibility of these functions. It would be futile to pursue the functional design of some cool feature, if its implementation is not feasible. That means that a level of engineering R&D work has to happen in conjunction with functional design. Who does engineering R&D? Developers, of course. That is, the same people who do development. But if the same people do engineering R&D and development, it becomes harder to parallelize these activities, and pipelining them can result in a loss of focus and productivity as developers multi-task between them for two successive iterations.

In order to allow for pipelined parallelism between engineering R&D and development with minimal disruption, we must free ourselves of the ingrained notion that all developers build production code all the time. As argued in an earlier blog on development roles, one way to do this is to use two distinct development roles: designer/builder and R&D engineer. Designer/builders concentrate on designing and developing the production system. R&D engineers concentrate on feasibility issues and exploring alternative solutions to hard technical problems. It is, of course, not necessary to typecast team members into these roles. People can move between them. What is important is that being responsible and accountable for different kinds of things, people in these roles are each allowed to focus on their areas of responsibility, and to work at a pace appropriate to their work product.

Now that the first stage of the pipeline includes both functional design and engineering R&D work, I prefer the name R&D stage for this first stage. And since development includes engineering R&D as well as program design and construction, the name development stage is no longer appropriate for the second stage of the pipeline. I will call the second stage the construction stage.

The R&D stage of the pipeline is the time to figure out the functions to be built, and to come up with feasible approaches to their implementation, away from the day-to-day pressures of building production code. This type of work is exploratory in nature, and it has a very different pace and expectation of success than the more or less clockwork pace of construction activity in most shops. The construction stage of the pipeline includes normal program design and construction.

Of course, not all risk can be eliminated before construction begins. But to the extent that conceptual and technical issues can be anticipated and explored before the construction stage of an iteration, interruptions, task switches, and tensions due to surprise under deadline pressure may be mitigated in the normal program design and construction work for the features of the iteration.

To recap: An initial R&D pipeline stage for each iteration enhances workflow by allowing work on functional design and normal development to proceed in parallel, and by reducing the risk of hiccups in the middle of normal program design and construction, so that most developers may work on designing and building the features of an iteration at a consistently high clip most of the time.

Similarly, a final deployment stage can help parallelize the nitty-gritty work associated with getting all acceptance tests to pass and deploying the final system, thus freeing most developers to start working on the features of the next iteration.

Here is a schematic of the resulting 3-stage development pipeline.

By the end of the construction stage, all automated tests, including automated acceptance tests, would be passing. What remains is for acceptance testers to run through the remaining manual tests one last time. This activity can be time-consuming. But if there is significant test coverage by automated tests, the bug yield from this last pass through manual tests is likely to be modest.

In that case, for a small development team, a single developer can usually be shunted to do release engineering work, including defect investigation, for the deployment stage of the pipeline. This release engineer (with expanded responsibility for defect tracking) buffers developers from interruptions by acceptance testers. The release engineer, being a developer, can also track down and isolate defects to particular areas of the code before involving the area specialists in the fix. That frees the area specialist from the burden of tracking down and pinpointing bugs, which is often the more time-consuming and disruptive part of bug fixing.

Of course, this function may be rotated between members of the team, to spread the pain, if that is how it is viewed.

© Copyright 2006 Bolour Computing.