goal

To get two Caddy instances to share TLS certificate storage with a shared mounted volume of a Backblaze B2 bucket.

why?

Caddy is a production grade project. But it needs a simple working solution for clustering that doesn’t require setting up a whole Redis cluster, Postgres database or other complicated infrastructure just for shared storage of TLS certs. Backblaze B2 is cheap, reliable, easy to mount to commodity VMs, and Caddy is happy reading and writing files from a B2 shared volume (most of the time).

what is the problem to be solved?

There is a Caddy issue where, even when a domain is properly provisioned via the Caddy API, and there are valid certs downloaded that Caddy can read, Caddy will throw an SSL error instead of serving the site securely. Only a full service caddy restart will fix this, which results in some downtime.

what is the suggested solution?

I suggest updating Caddy so that, if Caddy gets a request for a domain, it checks for valid certs, and if they exist, then serve the site properly over TLS instead of throwing an SSL error.

background of Caddy test environment

Here’s what I’ve done so far:

• [ ] spun up two Linux Ubuntu servers (Digital Ocean VMs), one in San Francisco, one in New York
  • [ ] installed Caddy per the docs
• [ ] created a DNS load balancer in Cloudflare (”grey cloud” — not proxied) to balance requests for https://caddy-test.purple.uno between the two servers (for HA, obviously)
  • open for diagram
• [ ] generated a set of subdomains pointed to caddy-test.purple.uno for experiments
  • 500.greenbongo.com thru 599.greenbongo.com
• [ ] generated 100 test websites on the servers, examples:
  • /usr/share/caddy/500/index.html
  • /usr/share/caddy/599/index.html
• [ ] created a Backblaze B2 bucket and mounted it to both servers using s3fs
• [ ] set owner of all files in the bucket to caddy
• [ ] set permissions on all bucket subfolders to 777
• [ ] created an override file here:
  • /etc/systemd/system/caddy.service.d/override.conf
• [ ] created a simple Caddy autosave.json file here:
  • /var/lib/caddy/.config/caddy/autosave.json

initial setup steps