Pre-Design

Existing situation

All RPCs and validators keep snapshots of all solana accounts.

• Triton says: We have an archive with an approximately once hourly snapshot, and then foundation does as well.
• Size: A snapshot file is 40GB
• Snapshots can be downloaded from RPC servers using this repo https://github.com/c29r3/solana-snapshot-finder, Meyerbro (validator) uses it and says that it works perfectly
• Snapshots can be parsed and dumped into a DB (SQLite in the repo: https://crates.io/crates/solana-snapshot-etl ) - The author is also a contributor to Jump’s firedancer client. However, this operation take a long period of time (~ 1 hr on a good machine) and results in 80 GB sqlite DB with long query times and lots of unnecessary data.
• The final goal is to open source this as public goods snapshot parsing infrastructure

Required

Code

• All code should be open source and developed inside marinade-finance github space
• Implement https://github.com/c29r3/solana-snapshot-finder to reliably obtain a snapshot file, the initial required frequency is once per day
• Fork https://crates.io/crates/solana-snapshot-etl to extract needed values to a SQLite database (to be later inserted in a PostgreSQL DB that will hold snapshots extraction history). (E.g. we do not need to track all token accounts, we only need mSOL token accounts)
• Allow projects to add “plug-ins” so different amounts can be tracked and extracted into the SQLite DB.
  • First milestone is to include a set of plug-ins to be able to parse top 10 DEFI locations for mSOL. Initially we need to know wallet + datetime/slot + location + mSOL amount. Where “location” is a place where the mSOL might be: e.g. wallet, Solend, Marinade-mSOL-Farm, Orca pool, Open Book, Tulip, etc. The initial use we will give to that DB is to sum(mSOL Amount) by wallet+slot so we can get total mSOL held by the wallet at that particular slot
  • The “plug/in” option, should allow other projects to be considered, e.g. Francium, or Kamino writes another “location” plug-in, and now we can count mSOL in Francium too. The plug in should parse protocol accounts owned by a particular wallet in order to return Mint+amount of assets under that wallet control.

Infra

• The code should be compatible with “Marinade Cloud”. @lj1024 will set the standards required so Marinade can run Solana Snapshots in Marinade’s Cloud.
• A Dockerfile that can be used to build a container with the solution included inside is required.
• All logs from the project must contain ISO 8601 timestamp, log level, message and must fit in a single line
• The project will run as a CronJob resource inside a k8s cluster or as a long running job. If it is a long running job, allow user to expose prometheus metrics