Opal’s backend services currently have a hard dependency on LaunchDarkly to start up. This will prevent our CronJobs from running, and will prevent any new pods from starting up - though any currently-running pods will still work with a cached set of feature flag data.

You can temporarily work around this dependency during the active LaunchDarkly incident by deploying a proxy service for LaunchDarkly and updating your instance of Opal to talk to that proxy service. We have pre-built images of this proxy service that contain a snapshot of our feature flag configuration.

To update your Opal deployment with this workaround, do the following:

1. Get kubernetes admin access to the cluster where your instance of Opal is deployed

2. Run this to download the helm chart for LaunchDarkly’s proxy service:

   helm repo add ld-relay <https://launchdarkly.github.io/ld-relay-helm>
   

3. Add the secret to the ld-relay service account:

   kubectl patch serviceaccount ld-relay -n opal-onprem \\
       -p '{"imagePullSecrets": [{"name": "opal-onprem-registry"},{"name": "kotsadm-replicated-registry"}]}'
   

4. Create a ld-relay-values.yaml file locally with these contents:

   image:
     repository: proxy.replicated.com/proxy/opal-onprem/179751979675.dkr.ecr.us-east-2.amazonaws.com/opal-ld-relay-proxy
     tag: 1.1074.1-sha-6d38ffe-prod
     pullPolicy: Always
   relay:
     environment:
       FILE_DATA_SOURCE: /relay-proxy-archive/relay-archive.tar.gz
   securityContext:
     capabilities:
       drop:
         - ALL
     runAsNonRoot: true
     runAsUser: 1000
   replicaCount: 2
   
   

5. Run this against your kubernetes cluster to deploy the ld-relay chart:

   helm install -n opal-onprem ld-relay ld-relay/ld-relay --version 3.3.0 --values ./ld-relay-values.yaml
   

   Verify that the ld-relay pods start successfully

6. Run this to modify Opal’s ConfigMap:

   kubectl -n opal-onprem edit cm opal-web
   

   Add this to the set of values:

   RELAY_PROXY_HOST: "<http://ld-relay:8030>"
   

7. Manually restart any Opal pods that are in CrashLoopBackoff state with kubectl delete pod. They should start up successfully, as they’ll begin talking to ld-relay instead of LaunchDarkly itself

<aside> ‼️

Once you do this, please avoid deploying new versions of Opal until the incident from LaunchDarkly is fully resolved.

</aside>

We will follow-up with a proper fix for this dependency soon after this incident.