By Russ Cox, Robert Griesemer, Rob Pike, Ian Lance Taylor, Ken Thompson Communications of the ACM, May 2022, Vol. 65 No. 5, Pages 70-78 10.1145/3488716Comments
Credit: Andrij Borys Associates; Renee French (CC by 3.0)
Go is a programming language created at Google in late 2007 and released as open source in November 2009. Since then, it has operated as a public project, with contributions from thousands of individuals and dozens of companies. Go has become a popular language for building cloud infrastructure: Docker, a Linux container manager, and Kubernetes, a container deployment system, are core cloud technologies written in Go. Today, Go is the foundation for critical infrastructure at every major cloud provider and is the implementation language for most projects hosted at the Cloud Native Computing Foundation.
Early users were attracted to Go for a variety of reasons. A garbage-collected, statically compiled language for building systems was unusual. Go's native support for concurrency and parallelism helped take advantage of the multicore machines that were becoming mainstream at the time. Self-contained binaries and easy cross-compilation simplified deployment. And Google's name was undoubtedly a draw.
But why did users stay? Why has Go grown in popularity when so many other language projects have not? We believe that the language itself forms only a small part of the answer. The full story must involve the entire Go environment: the libraries, tools, conventions, and overall approach to software engineering, which all support programming in the language. The most important decisions made in the language's design, then, were the ones that made Go better for large-scale software engineering and helped us attract like-minded developers.
In this article, we examine the design decisions we believe are most responsible for Go's success, exploring how they apply not just to the language but also to the environment more broadly. It is difficult to isolate the contributions of any specific decision, so this article should be read not as scientific analysis, but as a presentation of our best understanding, based on experience and user feedback over the past decade of Go.
Go arose through experience building large-scale distributed systems at Google, working in a large codebase shared by thousands of software engineers. We hoped that a language and tools designed for such an environment could address challenges faced by the company and industry at large. Challenges arose due to the scale of both the development efforts and the production systems being deployed.
Development scale. On the development side, Google in 2007 had about 4,000 active users working in a single, shared, multi-language (C++, Java, Python) codebase.3 The single codebase made it easy to fix, for example, a problem in the memory allocator that was slowing down the main web server. But when working on a library, it was too easy to unwittingly break a previously unknown client because of the difficulty of finding all the dependencies of a package.
Also, in existing languages we used, importing one library could cause the compiler to recursively load all the libraries that one imported. In one C++ compilation in 2007, we observed the compiler (after #include processing) reading more than 8 GB of data when handed a set of files totaling 4.2 MB, an expansion factor of almost 2,000 on an already large program. If the number of header files read to compile a given source file grows linearly with the source tree, the compilation cost for the entire tree grows quadratically.
To compensate for the slowdown, work began on a new, massively parallel and cacheable build system, which eventually became the open source Bazel build system.23 But parallelism and caching can do only so much to repair an inefficient system. We believed the language itself needed to do more to help.
Production scale. On the production side, Google was running very large systems. For example, in March 2005, one 1,500-CPU cluster of the Sawzall log analysis system processed 2.8 PB of data.26 In August 2006, Google's 388 Big-table serving clusters comprised 24,500 individual tablet servers, with one group of 8,069 servers handling an aggregate 1.2 million requests per second.4
Yet Google, along with the rest of the industry, was struggling to write efficient programs to take full advantage of multicore systems. Many of our systems resorted to running multiple copies of the same binary on a single machine, because existing multithreading support was both cumber-some and low performance. Large, fixed-size thread stacks, heavyweight stack switches, and awkward syntax for creating new threads and managing interactions between them all made it more difficult to use multicore systems. But it was clear that the number of cores in a server was only going to grow.
Here too, we believed that the language itself could help, by providing lightweight, easy-to-use primitives for concurrency. We also saw an opportunity in those additional cores: a garbage collector could run in parallel with the main program on a dedicated core, reducing its latency costs.
Go is our answer to the question of what a language designed to meet these challenges might look like. Part of Go's popularity is undoubtedly that the entire tech industry now faces these challenges daily. Cloud providers make it possible for even the smallest companies to target very large production deployments. And while most companies do not have thousands of active employees writing code, almost all companies now depend on large amounts of open source infrastructure worked on by thousands of programmers.
The remainder of this article examines how specific design decisions address these goals of scaling both development and production. We start with the core language itself and work outward to the surrounding environment. We do not attempt to give a complete introduction to the language. For that, see the Go language specification18 or books such as The Go Programming Language.11