Mastodon sidekiq autoscaling with the KEDA operator

This article assumes Mastodon is installed as a containerized setup on top of e.g. Kubernetes (k0s/k3s/k8s), Rancher, Openshift, etc.
Creating a kubernetes cluster or deploying Mastodon on kubernetes via helm is out of scope of this article.

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
kubectl create namespace keda
helm install keda kedacore/keda --namespace keda

Creating the HPA objects

Initially, we will need a way to authenticate to Redis. This can be achieved by creating a triggerAuthentication object which refers to the secret holding the Redis database password.

KEDA uses so called scaledObject definitions, which in turn will create horizontalPodAutoscalers  to scale our workloads.

In the example configuration, we start to scale up our deployment when the amount of jobs in the default queue reaches 1500.

If this is not enough to reduce the jobs in the queue below the threshold, every other minute another pod will be deployed, up to a maximum of four.

If, for any reason, the HPA should fail, the deployment will be scaled to two pods to avoid an outage of the service until the issue can be inspected and rectified.

Example triggerAuthentication :

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-auth-redis-secret
  namespace: mastodon
spec:
  secretTargetRef:
  - key: redis-password
    name: mastodon-database-secrets
    parameter: password

Example scaledObject :

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  labels:
    scaledobject.keda.sh/name: mastodon-sidekiq-worker-default
  name: mastodon-sidekiq-worker-default
  namespace: mastodon
spec:
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          policies:
          - periodSeconds: 60
            type: Pods
            value: 1
          stabilizationWindowSeconds: 300
  cooldownPeriod: 300
  maxReplicaCount: 4
  minReplicaCount: 1
  pollingInterval: 30
  fallback:
    failureThreshold: 3
    replicas: 2
  scaleTargetRef:
    kind: Deployment
    name: mastodon-sidekiq-worker-default
  triggers:
  - authenticationRef:
      name: trigger-auth-redis-secret
    metadata:
      address: 10.10.10.50:6379
      listLength: "1500"
      listName: queue:default
    metricType: Value
    type: redis

These are just some examples! 
Adjust name, namespace, IP address and port of Redis, etc. to your specific setup.

Testing the setup

After 

Additional caveats

Depending on setup specifics, certain parts of the configuration should be adjusted.

Considerations regarding Redis namespaces:

Considerations regarding pods with multiple Sidekiq roles:

It is a common practice to have the push and pull queues handled by one pod. In this case, triggers for both queues must be defined in the scaledObject - triggers is a YAML list and can hold more than one entry.

Considerations regarding performance and pod count:

If REDIS_NAMESPACE is defined in the environment, listName must be adjusted to reference the namespace:
listName: <REDIS_NAMESPACE>:queue:default

E.g. if REDIS_NAMESPACE=masto5 , then set listName to:
listName: masto5:queue:default

Just add additional definitions in the triggers part of the definition: 

  triggers:
  - authenticationRef:
      name: trigger-auth-redis-secret
    metadata:
      address: 10.10.10.50:6379
      listLength: "1500"
      listName: queue:push
    metricType: Value
    type: redis
  - authenticationRef:
      name: trigger-auth-redis-secret
    metadata:
      address: 10.10.10.50:6379
      listLength: "1500"
      listName: queue:pull
    metricType: Value
    type: redis

The referenced values for pod count, amount of jobs in the queue, when to scale up or down, or the time between scale operations are highly dependent on the underlying hardware and overall utilization of the cluster.
They have proven sufficient in the author's setup and can be used as a starting point, but most likely will need to be adjusted.