This is a SOP for daily routine tests. We run multiple jepsen/longevity tests with different cluster topology and products on our local server. The existing test clusters can be found at ‣.

The routine tests consist of 2 parts,

  1. Longevity tests that need to be run every night.
  2. Performance regression test that needs to be run every weekend.

Current test cluster and topology:

Engine type cluster name topology test home dir eloqctl config
eloqkv jepsen-eloqkv-eloqstorecloud-single single tx qnode + eloqstorecloud /home/eloq/workspace/jepsen-eloqkv-eloqstorecloud-single
eloqkv jepsen-eloqkv-eloqstorecloud-standby 1 primary + 1 standby + 1 voter + eloqstorecloud(S3) /home/eloq/workspace/jepsen-eloqkv-eloqstorecloud-standby resources/eloqctl-jepsen-eloqkv-eloqstorecloud-standby.yaml
eloqkv jepsen-eloqkv-eloqstorecloud-three-ng 3 node cluster + 1 log service + eloqstorecloud(S3) /home/eloq/workspace/jepsen-eloqkv-eloqstorecloud-three-ng
eloqdoc single tx node + S3 /home/eloq/workspace/jepsen-eloqdoc
eloqsql single tx node + S3 /home/eloq/workspace/jepsen-eloqsql

Longevity test

Routine Test Staff Assignment

The whole process might take up to 1 hour and need to be done before you leave the office. Please start early so that you can leave office on time.

Routine Test step for EloqKV

  1. Login ci pipeline http://35.222.141.150:8080. user: eloq password: cz123321Cz

  2. Build a new debug version of the latest main branch.

    Trigger a new build for EloqKv at http://35.222.141.150:8080/teams/main/pipelines/eloqkv-debug-build/jobs/ubuntu2204-main/ (This build step may take up to 1 hour, better to trigger this pipeline 2 hours before you leave the office).

  3. Connect to our internal network with WIFI chengzhang-5G. Password: CZhanG2022. ssh to our local server.

    ssh -J [eloq@192.168.1.75](<mailto:eloq@192.168.1.75>) eloq@compute-1

    compute-1 is our control node for all tests that we need to run. The password and username are of both machines are eloq

  4. Go to ~/workspace and clear old logs with script bash {JEPSEN_WORKING_DIR}/clear_log.sh for each cluster.

  5. After the build on CI has completed, update test cluster binaries to the latest version.

    For EloqKv, you can check all test clusters with eloqctl list. Then update each test cluster with eloqctl update <CLUSTER> debug -f

    NOTE: After the update is successful, the cluster will be started automatically.

    NOTE: Should use the cluster with EloqStore-Cloud mode, the cluster name are: jepsen-eloqkv-eloqstorecloud-single , jepsen-eloqkv-eloqstorecloud-standby,and jepsen-eloqkv-eloqstorecloud-three-ng.

    NOTE: For jepsen-eloqkv-eloqstorecloud-single , during server startup, a logs/asan.${pid} file is generated, which contains an ASAN warning reported by boost::context. This ASAN file should be deleted before executing the Jepvisen test; otherwise, the Jepsen checker will consider it an error and exit directly.

  6. Check the all test nodes are clean and no server is running on them. Notify in team chat that nightly tests will start.

  7. Start the test clusters with eloqctl start <CLUSTER>