Agile: A new way of Governing
A research paper in progress by Professor Dr. Ines Mergel, Professor Andrew B. Whitford and Professor Dr. Sukumar Ganapati
Abstract
The evolving concept of agile has fundamentally changed core aspects of software design, project management, and business operations. The agile approach could also reshape government, public management, and governance in general. Here, we introduce the modern agile movement, reflect on how it can benefit public administrators, and describe several challenges managers will face when expected to make their organizations more flexible and responsive.
Evidence for Practice
- Agile is antithetical to traditional hierarchical/bureaucratic line organizations.
- Agile requires a new form of self-selected, team-based organizational structures; it also requires leadership that serves such teams (e.g. servant leadership).
- Agile flourishes in innovation-oriented organizational cultures.
- Agile requires new, flexible forms of contracting and procurement.
Introduction
Government agencies are creating policies for agile government and introducing new practices and playbooks (e.g., U.S. GAO 2012, HUD 2018). For instance, digital service teams such as the United States Digital Service, the United Kingdom’s Government Digital Service, and the Canadian Digital Service have paved the way for working in an “agile way”. State and local governments have adopted agile and related practices through innovation labs and civic service design teams (e.g., the Georgia Technology Authority, New York City). In these specific cases, “agile” is a paradigm for better software development and project management to avoid large scale project failures at the end of the project and funding period (e.g., Mergel 2016; Project Management Institute, 2017). Agile is usefully contrasted with the traditional “waterfall” method, in which each project phase has to be carried out in sequence see, for example, Whitford 2020). Planning and building often takes years - and working software is often outdated when finally released. Agile project management values and techniques allow the project team to work on smaller increments, review their work often, and include feedback right away to avoid costly failures.
A quiet government transformation is already underway with practitioners who are investing heavily in working in agile environments and applying agile approaches from software development to other types of government problems. Traditionally, major changes in the way government works are either introduced through policy changes or public management reforms. Agencies are now changing their project management techniques and even procurement practices, incorporating new values and methods that are foreign to classical “bureaucratic” organizations. Historically, only emergency managers typically have to deal with crisis situations in an agile way to deal with the shifting ground realities. In both practice and academic settings, people often use “agility” interchangeably with more familiar terms such as responsiveness or adaptive governance - highlighting momentary change of the standard operating procedures, and leading to a conflation of the terms (e.g., Wise 2006; Janssen & Van der Voort 2016). Moreover, the canonical public administration literature has largely neglected agile and more fundamental changes it introduces to hierarchical and bureaucratic organizations. For instance, our search in Web of Science for the topic “agile” revealed that 26 research articles in “public administration”, most not related to the agile concept.
Based on years of collective experience interviewing practitioners and documenting these changes, this Viewpoint article aims to introduce the agile concept to the public administration community - bringing clarity to the use of the concept and integrating it with other more established concepts in public administration like responsiveness, resilience, and adaptability (in contrast to more monumental public management reforms such as NPM) (e.g., Greve et al. 2019). For clarity, academics can think of agile as a new package of routines and processes embedded within formal work groups and structures - as a pathway for “nudging” organizational behavior toward higher-valued outcomes. We proceed by describing its roots in the area of software design. We then highlight key advantages and challenges of an agile working environment in government beyond its origins in software development.
What Agile is and why it matters
Agile is such a recent phenomenon that most governments are still learning how to apply it when searching for innovation and performance improvements in government operations and service provision. Agile government is inspired by agile software development, but in wider administrative parlance it means responding to changing public needs in an efficient way. In the redesign and digitalization of public services, agile methods are applied in the initial requirement analysis. Service designers use ethnographic methods to understand user needs along the journey they take to access a public service. They interview process owners (to understand the formal requirements based on the law) and internal and external users (to understand their experiences throughout the full user journey). This reveals pain points but also things that work well, which shows how to design a better public service from a user perspective.
Service designers then compare these informal requirements with the formal requirements - and hand over a prototype to the software developers. These design steps emphasize inclusiveness and transparency - not only with respect to citizens, but also because public servants continue as core project team members and are not left out of the decision-making processes. In contrast, in “waterfall” or other traditional approaches involving vendors, contract officers help decide about the product’s attributes but rarely with the input of those who use the tools daily. Users, when they are involved at all, provide feedback or assessment information after the new process, software, or approach is deployed. In contrast to a traditional bureaucracy where decisions are made top-down and complaints from users emerge bottom-up, agile government procedures reframe traditional decision making by making internal and external users part of the process from day 1.
As such, agile is not in inherent conflict with democratic or other classical administrative values. In internal project management, agile is a method for improving the efficiency of service delivery. Yet, agile governance implies being responsive to public values like equality and social responsibility. This is because even if agency officials only care about how they deal with external users, agile shapes how the agency discovers and then integrates the changing needs of the public - itself a democratic value (Beck et al., 2001). Rather than focusing only on the provision of products or services, agile deeply values the voices of both team members and citizens. When present at all levels of the agency, successful self-governing agile teams prepare and make decisions, with the broad support of top management; this creates greater openness to the broader adoption of successful reform elements (Greve, et al., 2019).
The concordance of agile and classical public administration values can be seen in the role of the “Agile Manifesto” in the movement’s evolution (Beck, et al. 2001). In 2001, an array of software engineers and developers articulated four core agile values that they wanted to guide efforts to fix common problems in how virtually everyone builds and deploys complex software (see Table 1 below). Agile’s core values were established as a way of changing how software is produced. It is easy to see the main points here: users and makers were the main starting points; software should work first (even if not comprehensive); collaboration gets more done than a conflict-based process; and, change is inevitable - so responsiveness and adaptability are key:
- Individuals and interactions over processes and tools.
- Working software over comprehensive documentation.
- Customer collaboration over contract negotiation.
- Responding to change over following a plan.
From these four values, the architects of agile developed 12 key principles for software developers to follow when they were building and deploying complex software projects (see Table 2). These four values and 12 principles establish an infinite feedback loop. Developers are asked to figure out what the software should do, how it should be designed, how it should be built, and to test it all at the same time. Working software produced quickly is the most valuable outcome. Users then test that build as a way of guiding developers about what changes will most improve the overall project. Those user experiences as they interact with prototypes show developers what aspects they should fix first. After rapidly deploying new versions of working software to the user base, they then gather new information about what should be fixed next. The overall point here is that this cycle should take as many iterations as is needed to make user experiences better - and it may be that the feedback loop lasts forever.
- Seek to fulfill the customer’s needs. Do so through the early delivery of software. Continuously improve that software.
- Respond to the demand for changes to that software because doing so better positions the customer for success.
- Shorten the timescale for the delivery of working software. Deliver changes frequently.
- Developers should work hand-in-hand with business users.
- Center fabrication around individuals motivated to succeed.
- Emphasize face-to-face conversation within the team and between the development team and the broader organization.
- The most important benchmarks are working software.
- Sustainable development is the goal. All involved parties should be able to maintain a constant pace of engagement.
- Continuously focus on technical quality and good design.
- Emphasize simplicity.
- Self-organization in teams improves design and production.
- Regularly reflect on how to improve this process.
Because agile centers on working and learning from customers about how to build better software - about finding what core needs should be fixed next - it is easy to see how people in government have latched onto this approach for improving software development in the public sector. For instance, a 2014 IBM Center for the Business of Government report (Gorans & Kruchten 2014) focused on how waterfall approaches (the most common way government builds software and, indeed, most programs and projects) are rigid in their laying out of project requirements (often taking years) before they moved on to the actual design, development, testing, and deployment of working software.
While waterfall is slow and plan-based, agile is fast and light. Agile invests in initial planning but assumes that those plans will change as experience with working software provides new information about what users need. In many settings, those learning experiences are carried out by teams organized as “scrums” known for high-paced, rapid iteration - perhaps even with the goal of producing new versions of working software in less than a day. While waterfall approaches center on planning, agile succeeds when working software is deployed. Agile projects had four times more successes and one-third less failures than waterfall projects (Standish Group, 2015; Serrador & Pinto, 2015); this balance indicates why so many people are drawn to agile as a way of doing their work.
In some important ways, agile’s core assumption is that innovation is not linear - that it does not proceed in a rational, deterministic manner (e.g., see, Mackenzie and Wajcman 1999). Organizations, cultures, and need entwine when moving forward with an innovation; in many senses, the process from beginning to end is path-dependent and builds upon the actions and preferences of builders, operators, and end users.
Agile embeds this understanding - either purposively or intuitively - in its focus on rapid learning, which naturally means that agile requires high levels of collaboration between technical staff and the user base. For this reason, we now see agile as a way of doing more than just designing and deploying working software. Indeed, agile has evolved from a software design method to a central way of handling the design, production, and deployment of any customer-facing processes and includes not only IT specialists but also generalists, managers, and process owners at the user level. In this respect, while it is impossible to escape the social shaping of technology, agile respects those processes.
Because of its roots in and permeation of software design and deployment (to the point where it is difficult now to find software shops that are not agile in some shape or form), we see agile as more than just a way of making apps or bots. Yes, agile has continued to evolve - many organizations now use higher-level integrative practices (e.g., DevOps) for the continuous integration and delivery of code - we see the bigger picture here as being this new focus on all customer-facing processes. We know that “customer” is a loose word - in fact, users, employees, citizens, stakeholders, and any word we want to use can be the target of an agile design method to improve government; yet, agile works for both internal and external users.
Advantages of Agile
Overall, agile is a mindset that initiates a cultural change in bureaucratic command-and-control organizations. Agile administrations would be open to reforms, adapting to the changing environment, public values, and public needs. Here we outline how it can contribute to a more effective and efficient administration.
1. Agile assumes situations are fluid and change over time. As new information, constraints, or opportunities emerge, agile drives practitioners to revise and update early working versions to improve processes or services. Improvements might include speed to delivery, product or service quality, or to the program’s existence itself. While in traditional processes “results” are often mostly about “reporting,” in agile the goal is to satisfy constituents by solving their problems (outcomes are achieved) rather than just producing detailed documentation. Transparency and accountability are improved by knowing why things changed and who was in the room when decisions were made.
2. Agile privileges adaptive structure over hierarchies and silos. Unlike in traditional bureaucracies, in which administrative divisions are meat and bones of everything, in agile administrative divisions are only for backend purposes. External users do not know (or even need to know) administrative constraints. In this way, agile is like other cross-functional administrative arrangements intended to improve transparency, resource sharing, and capabilities. Like other arrangements (e.g., matrix designs), agile is intended to make it easier to know what works and avoid what does not work. In agile, policy or product designers need the input of users, citizens, or customers because their own experiences are not as useful or representative - so they have to leave their silos. In several ways this is what also lays behind the move toward single points of contact instead of dealing with multiple agencies for resolving issues (e.g., permitting in local governments, 311 call centers). In agile, coordination should be seamless at the frontend.
3. Agile emphasizes responsible individual discretion over bureaucratic procedures. Rather than following traditional standard operating procedures, workers with expertise are liberated to solve public problems in an efficient and effective manner. As discussed above, agile emphasizes bottom-up change over top-down direction (or even outside-in from vendors or consultants). Of course, this focus on the individual resonates with broader approaches to change management. Agile’s emphasis on the individual can improve public servants’ engagement if the organization’s culture values individual contributions. In agile, no worker is just “another cog in the wheel”, which changes the perceptions of those who are involved and might increase ownership and, subsequently, acceptance during the adoption.
4. Agile emphasizes continuous self-reflective learning processes. Agile assumes failure so agencies that experience failure in first iterations are better equipped to improve. As such, public managers are pushed to abandon a zero-failure culture so that workers are free to make mistakes. Of course, trial-and-error and experimental approaches can be difficult in environments that traditionally do not reward mistakes. Likewise, managers drive the process. For instance, agile “retrospectives” require managers and teams to revisit policies and procedures periodically to ask what happened and how things can improve. Agile cultures learn fast from past mistakes.
5. Agile increases knowledge about processes, procedures, and requirements for new processes and services. We know that agencies with agile purchasing processes - often “blanket purchasing agreements” (BPAs) - learn more about what they need and how things should work. For instance, co-authoring and adapting buying requirements for products and services often improve overall quality. In agile BPA processes, contract managers verify legal prerequisites but knowledge owners (who need to use the products and services) write content requirements. This helps ensure that products and services are usable, while maintaining oversight. Users then also assess what external contractors deliver. In domains like information technology, many vendors are required to follow agile principles in their work with private-sector entities, so an agile layer in the governmental sector is de rigueur. In turn, many agencies now require agile buying approaches, including the delivery of prototypes up front to demonstrate ability to deliver on tenders and improve transparency. We are already seeing this change in the service delivery industry (although it is still early stage days) (e.g., Whitford 2020).
Challenges in adopting Agile in Government
We also see several key challenges in importing agile into traditional agencies, especially in scaling new practices or experiments to the rest of the organization.
1. Agile is antithetical to many typical bureaucratic line organizations. In those settings, managers who need to take responsibility for actions may not be willing to do so given how cross-functional agile teams work. This is classical blame avoidance - a natural response in risk-averse settings when asked to take responsibility for new methods and especially their outcomes. If experimentation is not part of the toolbox, and if bureaus value organizational reputation and avoid public failures, agile never gains traction. Managers need to take responsibility for the final product and defend the outcome, even though it might not have been developed with their explicit input. Agile civil servants cannot rely on established standard operating procedures, so their actions run headlong into legalistic administrative culture. Because agile can also contradict administrative law, its application must be evaluated case-by-case. Moreover, in organizations largely characterized by “we have seen this before, let’s wait until the next wave comes in…”, traditionalists could see agile as another fad that will be replaced.
2. Agile requires a new form of leadership. Most notably, agile requires consensual decision making and acceptance of trial-and-error approaches - leadership styles not often represented among middle managers. Agile requires handing over decisions to groups of non-leaders; agile requires protection of teams from external political and other influences. In a number of ways, agile is consistent with servant leadership, which is less-represented in public administration (Greenleaf, 1970). Agile emphasizes putting subordinates first, helping them grow and succeed in the organization, empowering the team, and creating value for the community. Many private-sector information technology companies using agile emphasize servant leadership.
3. Agile requires new forms of contracting and procurement. Traditional contracting processes are oriented toward waterfall, which focuses on the delivery of specified products in a stepwise fashion. Agile does not quite fit the traditional processes since there is not a single finished product, process, or service - the thrust is on continuous improvement. Agile requires a contract management approach that is flexible and stretches beyond a fixed-price, one-time project. Such new modes of contracting and procurement focus on the holistic team, service delivery process, and creating long term trust relationships (Opelt et al. 2013).
Conclusion and future opportunities
Agile requires at its core a change of rigid bureaucratic cultures (top-down, zero-failure). This is not a small feat. Generations of civil servants were trained to follow the hierarchy principle. They have been told to obey a command-and-control structure without questioning the legitimacy of its decision-making model. They have learned to stick to their guns respectively in assigned roles and leave innovation up to the upper echelons. Change in organizational culture is hard, especially if there are no incentives to change. We know that there are many internal organizational and cultural features that influence these changes and adopting new reform elements (Greve et al., 2019; Jun and Weare, 2011; Venkatesh et al., 2003).
Agile culture turns traditional organizational principles of the bureaucracy upside-down. Agile values individual team members and teams. It requires responsible discretion - and great flexibility in organizational procedures and principles. Many organizational settings have experienced remarkable transformation through the application of agile principles. The same is possible in the public sector - if leaders embrace its advantages and are mindful of its challenges. Here, empirical research together with practitioners is necessary to understand the expectations of organizational members, such as necessary competences, decision-making structures, or expected outcomes.
Yet, we recognize that it is early days for our understanding of the prospects for agile in the public sector. For instance, one area ripe for theoretical and empirical contributions is the political aspects of agility. We know from our collective empirical understanding of the practice space that theoretical or empirical contributions that disentangle agility in policy making, agility in emergency policies, or other types of proactive or reactive policy making issues would be beneficial. An obvious example is that emergency management or public health responses may be especially attuned to agile principles. However, the development of such a contribution is beyond the scope of this short essay.
Another area that needs greater elaboration is how agile practitioners see the relationship between the framework they know best and the broader classical public administration values elaborated in this journal and taught in public affairs programs. A number of methodologies (e.g., Q-sorts) could be used to elaborate that relationship but to our knowledge no systematic information is currently available.
Even though our knowledge base remains incomplete, the practice experience makes clear that agile is gaining traction as a new way of governing. We hope that this contribution helps build a bridge for further collaboration between practitioners and academics in the search for new ways of enhancing public value.
Works cited
Beck, K., et al. 2001. The Agile Manifesto. Agile Alliance. http: //agilemanifesto.org/.
Gorans, Paul, and Philippe Kruchten. 2014. “A Guide to Critical Success Factors in Agile Delivery.” IBM Center for the Business of Government. 2014. Available online: http://www.businessofgovernment.org/sites/default/files/A%20Guide%20to%20Critical%20Success%20Factors%20in%20Agile%20Delivery.pdf
Greve, Carsten, Niels Ejersbo, Per Lægreid, and Lise H. Rykkja. 2019. Unpacking Nordic Administrative Reforms: Agile and Adaptive Governments. International Journal of Public Administration, DOI: 10.1080/01900692.2019.1645688
Greenleaf, R. K. 1970. The servant as leader. Newton Centre, MA: Robert K. Greenleaf Center.
Janssen, M., and H. Van Der Voort. 2016. Adaptive governance: Towards a stable, accountable and responsive government. Government Information Quarterly, 33(1), 1-5.
Jun, Kyu-Nahm, and Christopher Weare. 2011. Institutional Motivations in the Adoption of Innovations: The Case of E-Government. Journal of Public Administration Research and Theory, 21 (3): 495–519. DOI: 10.1093/jopart/muq020
Mackenzie, Donald and Judy Wajcman, Eds. 1999. The Social Shaping of Technology. Open University Press.
Mergel, I. 2016. Agile innovation management in government: A research agenda. Government Information Quarterly, 33(3), 516-523.
Opelt, A., B. Gloger, W. Pfarl, and R. Mittermayr. 2013. Agile Contracts: Creating and Managing Successful Projects with Scrum. Wiley.
Project Management Institute. 2017. Agile Practice Guide.
Serrador, P. and J.K. Pinto. 2015. Does Agile work? - A quantitative analysis of agile project success, International Journal of Project Management. 33(5), 1040-1051.
Standish Group. 2015. CHAOS Report 2015. https: //www.standishgroup.com/sample_research_files/CHAOSReport2015-Final.pdf
U.S. Government Accountability Office. 2012. Software development. Effective practices and federal challenges in applying agile methods. United States Government Accountability Office. Available online: https: //www.gao.gov/assets/600/593091.pdf
U.S. Department of Housing and Urban Development (HUD). 2018. Agile Methodology Policy, US Department of Housing and Urban Development (HUD). Available online: https: //www.hud.gov/sites/dfiles/OCHCO/documents/34301CIOH.pdf
Venkatesh, Viswanath, Michael G. Morris, Gordon B. Davis, and Fred D. Davis. 2003. User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly 27(3): 425-78. DOI:10.2307/30036540.
Whitford, Andrew B. 2020. “Transforming how Government Operates: Four Methods for Changing Government.” IBM Center for the Business of Government. http: //www.businessofgovernment.org/sites/default/files/Transforming%20How%20Government%20Operates.pdf
Wise, C. R. 2006. Organizing for homeland security after Katrina: Is adaptive management what’s missing?. Public Administration Review, 66(3), 302-318.
Prof. Dr. Ines Mergel
Professor Dr. Ines Mergel is full Professor of Public Administration in the Department of Politics and Public Administration at the University of Konstanz, Germany.
Prof. Dr. Whitford
Dr. Whitford serves as the Alexander M. Crenshaw Professor of Public Policy in the School of Public and International Affairs at the University of Georgia.