Production Planning

A production installation of Catalyst Enterprise Self-Hosted runs on a dedicated Kubernetes cluster with access to external PostgreSQL and, optionally, Kafka. Use the guidance below to size your infrastructure for the expected workload.

Cluster requirements

• Kubernetes 1.24 or later, installed in a dedicated cluster.
• A CNI that supports NetworkPolicy resources.
• At least 3 worker nodes for production, to support High Availability of the system components.
• Outbound connectivity to the Diagrid Cloud endpoints listed in Architecture.

Reference sizing profiles

The following profiles are verified minimum configurations for running Catalyst. Pick the profile that best matches your workload; individual components can be scaled independently from there.

Profile	Use case	Kubernetes worker nodes	AWS equivalent	Workflow PostgreSQL	AWS equivalent
Dev / PoC	Evaluation and development. Not suitable for production.	2 × (2 vCPU, 4 GiB)	c5.large	2 vCPU, 4 GiB	db.t3.medium
Small production	Low-volume workflows.	3 × (8 vCPU, 16 GiB)	c5.2xlarge	4 vCPU, 16 GiB	db.m5.xlarge
Large production	High-volume workflows with independent scheduler state.	3 or more × (8 vCPU, 16 GiB)	c5.2xlarge	16 vCPU, 64 GiB	db.m5.4xlarge

For the Large profile, we recommend a separate PostgreSQL instance for the Dapr Scheduler (8 vCPU / 32 GiB, e.g. AWS db.m5.2xlarge) so that scheduler state does not contend with workflow state.

Component resource footprint

The following are the default resource requests and limits shipped with the Catalyst Helm chart. Use them to validate that your node pool has sufficient capacity. All values can be tuned via Helm; refer to the Helm Reference for the full set of options.

Component	Replicas	CPU (request / limit)	Memory (request / limit)
Agent	1	40m / —	500Mi / 1200Mi
Management	2	40m / —	100Mi / 1200Mi
Gateway (Envoy)	1 (2 with HA)	100m / 1000m	512Mi / 2048Mi
Gateway (Control Plane)	1 (2 with HA)	50m / 100m	50Mi / 100Mi
Identity Injector	1	50m / 200m	64Mi / 128Mi
Dapr Server (per app)	per app	10m / 300m	25Mi / 256Mi
OpenTelemetry Collector (metrics)	per project	75m / —	500Mi / 1100Mi
OpenTelemetry Collector (logs)	per project	50m / —	500Mi / 750Mi
Dapr Scheduler	1 (3 with HA)	— / —	130Mi / 175Mi
Dapr Sentry	1	— / —	— / 100Mi

Dapr Servers and OpenTelemetry Collectors are provisioned per app and Project at runtime, so their aggregate footprint scales with the number of Apps and Projects you deploy.

Scale limits

Each Catalyst installation enforces the following internal limits to prevent resource exhaustion:

Limit	Default	Helm value
Projects per installation	50	agent.config.placement.max_project_count
Apps per installation	300	agent.config.placement.max_appid_count

High availability

It is recommended to enable High Availability (HA) in production installations. Set gateway.ha.enabled: true in your Helm values to run two replicas of the Gateway Envoy and the Control Plane. The Management service runs two replicas by default.

It is also recommended to run the following external dependencies across multiple Availability Zones (AZs):

• PostgreSQL database, for the Workflow state store
• PostgreSQL database, for the Dapr Scheduler service when using the Large size profile
• Kafka, when Managed Pub/Sub is enabled

See the Helm Reference for the full set of HA-related options as well as the configuration of external PostgreSQL and Kafka instances.

High availability here is defined as surviving the loss of a Kubernetes node or AZ within a single Catalyst region. It does not include surviving the loss of the region itself given the cluster, database, and gateway are all deployed in the same geographic location. To enable multi-regional availability add the region to a region group, and see what survives what for which failures each mechanism covers. This allows multiple regions to run in an active-passive configuration with only one region active at a time, sharing one replicated database, behind a single endpoint. Both mechanisms are worth having, and adding a region group does not replace deploying multiple replicas as described above.

Under load

The Helm chart defaults are tuned for typical non-production workloads. High-throughput deployments — many concurrent workflows, frequent reminders and jobs, or large bursts of application or agent activity — surface bottlenecks that the defaults do not anticipate. The guidance below comes from load testing the Large production profile and documents the adjustments that had the largest impact, giving recommendations so you can apply them before hitting the same limits.

The settings below are independent — apply the ones that match the pressure you observe.

Create a dedicated database for the Scheduler service

The Dapr Scheduler manages reminders, jobs, and workflow timers, and is backed by PostgreSQL by default. Out of the box it reuses the global PostgreSQL instance configured under global.postgresql (postgresql.use_global: true) in the Helm chart and shares a database server with the workflow state store. For high job and reminder volume, give it a dedicated PostgreSQL instance so its writes do not contend with the workflow state store: set postgresql.use_global: false and provide the connection yourself. It is also recommended to raise the database connection pool well above the default of 5 connections per instance, and increase the worker count that drains the job queue. See the following Helm values as an example:

agent:
  config:
    internal_dapr:
      scheduler:
        backend_type: postgresql
        postgresql:
          use_global: false
          max_conns_per_instance: 100
          connections:
            - existing_secret_name: scheduler-external-postgresql
              existing_secret_namespace: cra-agent
        workers: 8192
      scheduler_resources:
        requests:
          memory: 256Mi
        limits:
          memory: 1Gi

The referenced Kubernetes secret must contain the keys host, port, username, password, and database.

The scheduler's database must allow logical replication

The PostgreSQL scheduler uses a logical replication connection, so the database behind it needs wal_level = logical. Managed databases turn this off by default. Turn it on with rds.logical_replication = 1 on Amazon RDS and Aurora, cloudsql.logical_decoding = on on Cloud SQL, or wal_level = logical on Azure Flexible Server. On RDS this is a static parameter, so you have to reboot an existing instance before it takes effect.

You need this with the default setup too, not only with the dedicated instance above. By default the scheduler shares the global PostgreSQL instance, so set the parameter there. Without it, every scheduler replica crash-loops with FATAL: no pg_hba.conf entry for host ... SSL encryption. The error names the host, not the setting, so it's easy to misread. The region then starts with no jobs and no actor reminders. The Catalyst agent waits for the scheduler pods, so what you actually see is the agent restarting and the gateway stuck in Init.

Running the scheduler on its own instance, separate from the workflow store, keeps the two from contending — see the dedicated scheduler instance in the Large production profile. The default scheduler memory limit of 175Mi is also too low for this volume; the values above raise the request to 256Mi and the limit to 1Gi.

Tune the workflow state store

Workflow execution state is persisted to PostgreSQL, configured under global.postgresql in the Helm chart. For production installations use an external PostgreSQL instance, give the database a connection pool sized for the number of concurrent workflows — well above the chart default of 2 connections — and enable write batching:

global:
  postgresql:
    create: false
    external:
      existing_secret_name: catalyst-external-postgresql
      existing_secret_namespace: cra-agent
      max_conns: 100

agent:
  config:
    project:
      workflow_store_batch_size: 100
      workflow_store_flush_interval: 100ms

workflow_store_batch_size and workflow_store_flush_interval control how workflow execution updates are written to PostgreSQL. By default each update is written eagerly, which generates a write per state transition. Batching coalesces up to workflow_store_batch_size updates, flushing whenever the batch fills or workflow_store_flush_interval elapses, whichever comes first. This trades a small amount of write latency for a large reduction in database write pressure — the single most effective change for sustaining high workflow throughput.

Scale and resource the sidecars

Each App ID runs a Dapr sidecar. The default CPU request of 10m is sized for idle sidecars and starves the runtime under load. Raise the request to a full CPU, enable autoscaling so busy App IDs add replicas, and lower the log level to cut logging overhead on the hot path:

agent:
  config:
    sidecar:
      log_level: error
      autoscaling:
        enabled: true
        min_replicas: 1
        max_replicas: 10
        target_cpu_utilization_percentage: 80
      resources:
        requests:
          cpu: 1
          memory: 50Mi
      soft_mem_limit:
        override: "0"

Setting soft_mem_limit.override: "0" disables the Go soft memory limit (GOMEMLIMIT) while keeping the container memory limit in place. The soft limit forces the garbage collector to run more aggressively as memory approaches the cap, which adds CPU overhead under sustained load. Disabling it favors throughput, at the cost of the sidecar being more likely to be OOM-killed if it exceeds its memory limit — size resources.limits.memory accordingly.

Enabling High availability also matters under load: the gateway is the first component to saturate as request volume grows, and running at least two Envoy replicas removes both the throughput ceiling and the single point of failure on the request path.

Private container images

If you are installing in an environment without access to public container registries or prefer to use your own container registry, you can pull the artifacts from our public registry, re-tag them, and push them to your private registry. Then, you can configure the Helm chart to use your private registry by setting the appropriate values. See the Helm Reference for the chart values.

We have provided a script and documentation on how to achieve this in the Catalyst Enterprise Self-Hosted Helm Chart repository.

Next steps

• AWS Deployment — reference architecture on AWS (VPC, EKS, RDS, Bastion host).
• Azure Deployment — reference architecture on Azure (VNet, AKS, Azure Firewall, management VM).
• Multi-region high availability — run a project across several regions so it survives the loss of one.