Cluster schedulers promise us ease of deployment with ultimate scalability. We designed an ambitious challenge to test these promises: schedule one million containers. We call this the Million Container Challenge (C1M).
HashiCorp prides itself on creating technically excellent software, and the C1M is a test to showcase this. We tested Nomad against the C1M to ensure that we meet the needs of our users at any scale.
A cluster of five Nomad servers scheduled one million containers in less than five minutes, a rate of 3,750 containers per second. Details and observations of this benchmark are explained below.
Thank you to Google for providing the credits and support necessary to run the infrastructure for the C1M on Google Cloud.
We ran both a C100K (100,000 containers) and C1M with the following cluster configurations. For each cluster configuration, we ran five Nomad servers. The cluster size below is number of Nomad clients, and does not include the additional servers.
Our partners at Google generously provided the credits to run this amount of compute on Google Compute Engine. The strong and consistent performance of Google Cloud made the entire testing process efficient. We used Terraform to spin up thousands of resources in minutes.
A job is a declaration of work submitted to the scheduler. A job is composed of many tasks. A task is an application to run, which is a Docker container running a simple Go service in this test. Nomad can also schedule other tasks such as VMs, binaries, etc.
We could have submitted 1 job with 1,000,000 tasks for the C1M. Instead, we broke down the tasks into many jobs in an even split in order to provide more strain on the scheduler as well as better represent real world scenarios where many jobs would be running.
To make the benchmark even more strenuous and realistic, we designed the jobs to have constraints on which nodes can run the tasks. This forces the scheduler to evaluate and check constraints in addition to pure binpacking.
For full technicals of the C1M setup, including the Docker images used, the Nomad job specification, Terraform scripts, and more please see the full technical README of the C1M. The linked repository can also be used to reproduce any of these results.
We begin by looking at the results for the C100K, since interesting observations can be drawn by comparing these results to the C1M results shown later.
The Y-axis is number of containers. The X-axis is time (milliseconds). There are three lines on the graph: