Fly runs apps (and databases) close to users, by taking Docker images and transforming them into Firecracker micro-vms running on our hardware around the world. You should try deploying an app: it only takes a few minutes.
This is a story about a cool hack we came up with at Fly. The hack lets you do something pretty ambitious with full-stack applications. What makes it cool is that it’s easy to get your head around, and involves just a couple moving parts, assembled in a simple but deceptively useful way. We won’t bury the lede: we’re going to talk about how you can deploy a standard CRUD application with globally-replicated Postgres, for both reads and writes, using standard tools and a simple Fly feature.
If you’ve build globally distributed apps in the past, you’re probably familiar with the challenges. It’s easy to scale out a database that’s only ever read from. Database engines have features to stand up “read replicas”, for high-availability and caching, so you can respond to incoming read requests quickly. This is great: you’re usually more sensitive to the performance of reads, and normal apps are grossly read-heavy.
But these schemes break down when users do things that update the database. It's easy to stream updates from a single writer to a bunch of replicas. But once writes can land on multiple instances, mass hysteria! Distributed writes are hard.
Application frameworks like Rails have features that address this problem. Rails will let you automatically switch database connections, so that you can serve reads quickly from a local replica, while directing writes to a central writer. But they’re painful to set up, and deliberately simplified; the batteries aren’t included.
Our read/write hack is a lot simpler, involves very little code, and it’s easy to understand. Let’s stick with the example of a standard Rails application (a difficult and common case) and dive in.
You can think of Fly.io as having two interesting big components. We have a system for transforming Docker containers into fast, secure Firecracker micro-VMs. And we have a global CDN built around our Rust front-end proxy.
We run all kinds of applications for customers here. Most interesting apps want some kind of database. We want people to run interesting apps here, and so we provide Fly Postgres: instances of Postgres, deployed automatically in read-replica cluster configurations. If your application runs in Dallas, Newark, Sydney and Frankfurt, it’s trivial to tell us to run a Postgres writer in Dallas and replicas everywhere else.
Fly has a lot of control over how requests are routed to instances, and little control over how instances handle those requests. We can’t reasonably pry open customer containers and enable database connection-switching features for users, nor would anyone want us to.
You can imagine an ambitious CDN trying to figure out, on behalf of its customers, which requests are reads and writes. The GETs and HEADs are reads! Serve them in the local region! The POSTs and DELETEs are writes! Send them to Dallas! Have we solved the problem? Of course not: you can’t look at an HTTP verb and assume it isn’t going to ask for a database update. Most GETs are reads, but not all of them. The platform has to work for all the requests, not just the orthodox ones.
So, short of getting in between the app and its database connection and doing something really gross, we’re in a bit of a quandary.
It turns out, though, that with just a little bit of cooperation from the app, it’s easy to tell reads from writes. The answer is: every instance assumes it can handle every request, even if it's connected to a read replica. Most of the time, it’ll be right! Apps are read-heavy!
When a write request comes in, just try to do the write, like a dummy. If the writer is in Dallas and the request lands in Frankfurt, the write fails; you can't write to a read replica. Good! That failure will generate a predictable exception. Just catch it:
rescue_from ActiveRecord::StatementInvalid do |e|
if e.cause.is_a?(PG::ReadOnlySqlTransaction)
r = ENV["PRIMARY_REGION"]
response.headers["fly-replay"] = "region=#{r}"
Rails.logger.info "Replaying request in #{r}"
render plain: "retry in region #{r}", status: 409
else
raise e
end
end
In 8 lines of code, we catch the read-only exceptions and spit out a fly-replay header, which tells our proxy to retry the same request in the writer’s region.
You could imagine taking this from 8 lines of code to 1, by packaging this logic up in a gem. That’s probably a good idea. The theme in this blog post though is that all the machinery we’re using to make this work is simple enough to keep in your head.
Our proxy does the rest of the work. The user, none the wiser, has their write request served from Dallas. Everything… works?
This design isn’t why we built the fly-replay feature into our proxy.