Agile Business Change Blog Thoughts on Agile Strategic Business Change and Agile Delivery

I had the privilege of presenting a keynote at the Agile Business Conference last week "Delivering Agile in Government; Learning Lessons from the Commercial Sector", together with Jerrett Myer of the IfG (Institute for Government).

Jerrett first presented on the recent System Error report produced by the IfG, and the new UK government IT strategy, which are both promoting the use of Agile to address issues with public sector IT projects.  IndigoBlue was involved last year in the research phase for the System Error report by leading a project in a "problem area" so the IfG could determine if, how effectively, and with what challenges, Agile could be used in government.  The results were extremely positive and the project was delivered for a fraction of the estimate for the waterfall project.

Jerrett finished his part with a list of key challenges to overcome – challenges inherent to Agile rather than the significant challenges of creating change in such a large organisation, which is currently short of investment options.  These include cultural and mindset issues, requirements for effective governance, necessarily wide and disparate stakeholder participation, and government's procurement strategy and processes.

My section of the keynote addressed these challenges through lessons learned introducing Agile in large organisations in the private sector.  The point being that none of these challenges are specific to government – the differences are simply a matter of degree.

The first and most significant lesson from the private sector is that, as organisations get more complex, there is tendency toward Agile being implemented at the fringes of the IT change.  This is not the same problem as achieving some diluted, anaemic form of Agile, although that can be a problem too - this is where an organisation has a high-degree of capability to achieve remarkable results using Agile, but the use of Agile is limited to small low-risk projects, projects which are otherwise failing, or projects deemed as experimental.  Clearly an organisation operating this way gets very limited value from Agile, when frustratingly the large complex projects are where the most benefit is possible. This is particularly true of the massive government projects that hit the news because of colossal overspend and ultimate failure to deliver.

Here are two examples of how Agile ends up on the fringes:

1. On day one, Agile projects are inconsistent with the existing governance structures.  They are therefore given exemption.  Before long projects are being started using Agile with the express purpose of being exempt from governance.  This results in a corporate view that Agile projects are intrinsically ungovernable, thus risky and not suitable for large or complex projects.
2. Agile is often perceived as a well-defined thing, and therefore has prerequisites (eg co-location, or a clear product owner) that must be met to be used.  As organisations get more complex it gets harder to find projects that meet these criteria, so a lower proportion of smaller, simpler projects are deemed suitable.

For Agile to be at the heart of IT change it must add value universally, and preferably more value the more complex the problem is.  This means we need to be very clear about what Agile is, and how it addresses complexity.

There is a form of Agile which we refer to generically as "team in a room".  This form of Agile focuses on optimising quality and productivity through simplification: get the key skills and knowledge inside the single room, set a clear goal, remove impediments, and let them get on with it.  This is not to trivialise it in any way – it is hard, and can provide extraordinary results.  It suffers from two key problems though in the face of increasing complexity:

1. It is focused on optimising the development part of the system.  In general, optimising one part of a system is not necessarily optimal for the system as a whole.
2. What happens when the attempts to simplify fail, eg you can’t get a room (even metaphorically), there are too many to fit in the room, you can't the right people in the room, etc?

Team-in-a-room does not apply universally, and cannot be implemented without prerequisites.  It is brittle in the face of increasing complexity.  It is an ideal to strive for, but fortunately it is not the totality of where Agile adds value.

Another way of thinking about Agile is as "incremental delivery".  The team inside a room will (ideally) achieve incremental delivery as a matter of course, but we  promote it as a headline objective in its own right, which does add more value in the face of increasing complexity.  Incremental delivery can be utilised even without team-in-a-room, although it is less complex and more cost efficient when both are used in conjunction.

Incremental delivery is the process of looking at a problem as a journey of complete deliverables, rather than a single one-off deliverable.  Planning the journey becomes as important as the identifying the end state.  Note that this is an additional output of planning compared with a traditional project.  It can be done well or badly, and should therefore be subject of review and governance.  It also requires a new mindset and skill set.

Thinking incrementally enables, and forces, a project to actively seek simplicity.  It is the opposite of the forces that tend to make problems grow, by adding more objectives or more scenarios or more points of view.   It is constantly asking the question "is there something simpler which is of value?" without the emotional context of an equivalent descoping exercise.

Thinking of Agile as incremental delivery, ie many batches of work instead of one batch, helps understand the underlying governance challenges with Agile: traditional governance is focused on shaping the single batch, deciding whether to commit to it, and tracking what remains to complete it (ie realising the investment).  When using multiple batches, if they get small enough, it is of no interest what is happening inside a batch, and the decision to commit to the next batch is in principle much easier.  However, there are still issues that typically need to be addressed across multiple batches, such as funding commitments, compliance, architecture, business change and residual post-batch work. Ignoring this can lead to catastrophic project failure even when team-in-a-room is working effectively.

There is however an even more profound, and potentially difficult to digest, way of viewing Agile: as dynamic uncertainty management.  A key principle of traditional processes is that specific uncertainties are removed at specific times, and this is built directly into the process: define the requirements, then the functionality, then the technical solution; or provide costs at increasing levels of accuracy.  At the other extreme, "team-in-a-room" Agile enables software to be built in a more responsive way, so it is possible to implement "simple" Agile projects by dealing with all uncertainty in a just-in-time way per increment.  Both fixed approaches to uncertainty management are high risk within complex environments.

The important message is that projects can be massively optimised by taking a proactive and intelligent approach to uncertainty management, deciding explicitly why, when and how uncertainty should best be removed.  It is even possible to manage an uncertainty by putting in place mechanisms (eg design standards) that enables the uncertainty to be "carried" for longer.  A uniform approach to uncertainty is inherently poor: for example, spending time and effort getting costs accurate within 10% on a project that would be clearly cost-beneficial even if it cost twice as much, whereas in another case a 10% cost variance might affect the outcome of a borderline decision.

Like incremental delivery, dynamic uncertainty management is a universally applicable, additional planning tool and skill.  Different approaches to managing uncertainty can have a significant impact on risk and strategic decision-making, and it is essential that the planning is reviewed as part of governance procedures.

Returning then to the challenges that the IfG identified in adopting Agile in government, our experience is that incremental delivery and proactive uncertainty management provide a conceptual framework that enables the right context of engagement with wider stakeholders, and clarify how Agile is different from what they are used to.  Cultural change to accept and embed new ideas should not be just a question of "trust me ", or even "try it out and see".

Presentation Slides

Agile in Goverment; Lessons from the Private Sector