https://queue.acm.org/detail.cfm?id=3182626
“Software is eating the world.” —Marc Andreessen
“You can't manage what you don't measure.” —Peter Drucker
Organizations from all industries are embracing software as a way of delivering value to their customers, and we're seeing software drive innovation and competitiveness from outside of the traditional tech sector.
For example, banks are no longer known for hiding gold bars in safes: instead, companies in the financial industry are harnessing software in a race to capture market share. Using innovative apps, banks are making it possible for their customers to do most of their daily banking in a few swipes, from depositing checks to transferring money securely between bank accounts. Moreover, the banks themselves can improve their service in a number of ways, such as using predictive analytics to detect fraudulent transactions. Other industries are seeing similar changes: cars are now computers on wheels, and even the United States Post Office is in the middle of a massive DevOps transformation. Software is everywhere.
Leaders must embrace this new world or step aside. Gartner Inc. predicts that by 2020, half of the CIOs who have not transformed their teams' capabilities will be displaced from their organizations' leadership teams. And as every good leader knows, you cannot improve what you do not measure, so measuring the software development process and DevOps transformations is more important than ever.
Delivering value to the business through software requires processes and coordination that often span multiple teams across complex systems, and involves developing and delivering software with both quality and resiliency. As practitioners and professionals, we know that software development and delivery is an increasingly difficult art and practice, and that managing and improving any process or system requires insights into that system. Therefore, measurement is paramount to creating an effective software value stream. Yet accurate measurement is no easy feat.
Collecting measurements that can provide insights across the software delivery pipeline is difficult. Data must be complete, comprehensive, and correct so that teams can correlate data to drive business decisions. For many organizations, adoption of the latest best-of-breed agile and DevOps tools has made the task even more difficult because of the proliferation of multiple systems of record-keeping within the organization.
One of the leading sources of cross-organization software delivery data is the annual State of DevOps Report (found at https://devops-research.com/research.html).2 This industry-wide survey provides evidence that software delivery plays an important role in high-performing technology-driven organizations. The report outlines key capabilities in technology, process, and cultural areas that contribute to software-delivery performance and how this, in turn, contributes to key outcomes such as employee well-being, product quality, and organizational performance.
Bolstered by this survey-based research, organizations are starting to measure their own DevOps “readiness” or “maturity” using survey data. While this type of data can provide a useful view of the potential role that DevOps can play in teams and organizations, the danger is that organizations may blindly apply the results of surveys without understanding the limitations of this methodology.
On the flip side, some organizations criticize survey-based data wholesale and instead attempt to measure or assess their DevOps readiness or maturity using system data alone. These organizations, which are creating metrics based on the system data stored in their repositories, may not understand the limitations of that methodology, either.
By understanding these limitations, practitioners and leaders can better leverage the benefits of each methodology. This article summarizes the two separate but complementary approaches to measuring the software value stream and shares some pitfalls of conflating the two. The two approaches are defined as follows: