Agile software development is now the dominant approach to software engineering, with adoption rates reaching 55 percent in late 2016, according to research from Computer Economics of Irvine, Calif.
Today, the incremental development trend is everywhere – even in places you wouldn’t expect. “I was surprised at the degree to which agile software development and agile DevOps are being used in programs,” said Maj. Gen. Sarah Zabel, who recently became the Air Force’s director of IT acquisition process development to help the Air Force acquire systems faster. “F-35 is doing agile. F22 is doing agile. There are so many projects doing some type of agile development.… I’ve seen between 25 and 30 in the past couple of months.”
And she wants to see more.
Zabel’s job was created for her with a specific mission in mind: “It was an expression of the frustration from our secretary and chief of staff,” she said at the Defense Systems Summit in November. “Why does it take us eight to 10 years to develop systems that will be wickedly expensive and which we know won’t be what we need when it’s finally delivered?”
There are a host of reasons, of course. Risk-averse procurement officers, arcane procurement rules and grindingly slow requirements processes are prime causes, but so are old-fashioned waterfall development processes that separate users from developers and assume that nothing will change between the time the requirements are set and the system is delivered.
Agile development, by contrast, breaks down those requirements into smaller, more manageable pieces that can be completed in short “sprints” of one to four weeks. Daily scrums bring all interested parties together to share current task information and discuss any impediments. At the conclusion of a sprint, customers get an early look at functionality and the opportunity not only to approve, but to see possibilities they hadn’t imagined before. Other times, that early access affords the opportunity to “de-scope” requirements and eliminate waste; everyone becomes more vested – and accountable – in the development process.
“I’d have a very hard time going back to waterfall,” says Matthew Zach, director of software engineering at General Dynamics Information Technology’s Health Solutions, who has been working in an agile environment since 2009. “Once you transition teams to this, they like it. The customer likes it, too.”
The 2001 Agile Manifesto launched a revolution in customer-centered software development. In the process, its authors laid out 12 underlying principles that can be applied to software development and, indeed, to many other kinds of projects, as well:
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
- Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
- Business people and developers must work together daily throughout the project.
- Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
- Working software is the primary measure of progress.
- Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity – the art of maximizing the amount of work not done – is essential.
- The best architectures, requirements, and designs emerge from self-organizing teams.
- At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Teams, in fact, are a key to the agile philosophy. Close-knit teams work together like well-oiled machines. “If one person’s tasks for a sprint are done,” Zach said, “we want them to look at the board and see how they can help others on the team. It’s a high-trust, high-competence situation. We trust the team will produce. It all revolves around the team: the overall team succeeds or stumbles. Individual heroics are not as prevalent or necessary.”
The board is a means of communicating status and progress to the group. Agile practitioners may follow one of several methodologies to structure their projects. The most common of these, Scrum, divides project tickets into three categories – Ready, Doing and Done. Tickets move across the board showing progress from conception through testing and completion.
A second agile methodology is Kanban, which is derived from Toyota’s just-in-time delivery process of the same name. In Kanban, work flows through four stages: In progress, testing, ready for release and released. Only a certain number of items can be in each state at any one time. By limiting work in progress (WIP), the team can see when bottlenecks occur and rally to solve those problems, while otherwise managing a steady workflow.
“Teams using Kanban are typically mature teams. They can produce at such a predictable fashion, that they abandon the sprint concept and simply pull work off a prioritized list. These teams produce a small but continual stream of business value. Teams operating at this level have a high amount of automation built into the process to maximize efficiency and quality,” Zach said. “Engaged customers groom the backlog and ensure the most needed functionality is at the top of the list. They can then trust those items will be completed first.”
Customers on Board
That may be the biggest difference of all: Instead of ordering a completed product and then going away until it’s done, the product is in a continuous state of development and the customer is likewise continuously returning to the table, reviewing results, and offering feedback. That means fewer surprises and disappointments and more opportunities to refine ideas and clarify intent.
For the customer, the big change is near constant involvement. Customers are involved at the start so developers can better understand requirements, which are contained in user stories that describe the necessary task, and they are there, as well, at the end of each sprint to see, approve and comment on the work thus far.
It can be a hard adjustment for mission-focused customer agencies. “Officials from the Office of the CIO at three agencies (Environmental Protection Agency, General Services Administration and the Department of Labor) independently reported challenges related to organizational changes, such as staff adapting to the culture shift from being business customers to taking on a more active role as product owners and project managers in the software development process,” noted the General Accountability Office (GAO) in a November report on incremental development.
The agile movement started in 2001 with the creation of the Agile Manifesto, in which a group of software visionaries sought to change the way developers and their customers approached product development by placing people and interactions ahead of processes and tools; valuing working software over comprehensive documentation; encouraging customer collaboration over contract negotiation; and embracing the ability to respond to changes over slavishly adherence to plans. The group laid out 12 principles to developing better software (see box).
But agile is not all touchy-feely soft skills. At its root is order and organization. The up-front planning, daily scrum sessions, progress boards, one- or two- or three-week-long sprints – are all structures intended to enforce discipline and make teams work more efficiently.
“Think of it this way: If I want to make a low-cal birthday cake, I can’t bake the cake, frost it and then say, ‘Oh, I want to make it low-calorie.’ It doesn’t work that way,” says Paul Black, computer scientist at the National Institute of Standards and Technology. “I have to decide to replace oil with apple sauce before I bake the cake for it to turn out right. It’s the same with software. The planning up front is crucial.”
Yet that should not be construed as trying to plan everything all at once, notes GDIT’s Zach. The critical difference is that agile development teams break down those plans into small, manageable pieces.
Robert Binder, senior engineer in the Software Solutions Division at Carnegie-Mellon University’s Software Engineering Institute (SEI), says agile is here to stay. “You might say it’s all over but the shouting,” he told GovTechWorks. “Agile development is now pretty much the way most software gets done. There are some parts of the software development universe where it’s taken a while for that change to take root: Program offices working on systems that are very long-lived, [which] tend to be on the trailing end of that.”
Now comes the hard part, at least for those arriving late to the party: actually implementing agile processes and managing the changes they entail.
“The cultural piece does represent significant barriers, just because of the way we have traditionally executed software and system acquisitions,” says Eileen Wrubel, technical lead for agile in government at SEI’s Software Solutions Division. “In the interest of being good stewards of taxpayer dollars, [government program managers] have historically tried to nail down all the requirements up front, when, in reality, we operate in a world where the ground changes under us rather frequently. However, those behaviors are still engrained in the culture: The idea that we need to lock down as much as possible because all variability may be interpreted as bad is a cultural mindset that is changing over time.”
Every small program that adopts an agile process is a step in that direction, she says. “There’s been a lot of work to look at smaller programs and engaging testing and cyber security organizations earlier and more often. There’s been a lot of guidance to look at prototyping and innovative means of acquisition. But it has been a shift due to the inertia of how the acquisition system has operated in the past.”