This guide outlines a step-by-step process for estimating resource allocation based on workload, including calculating resource requirements per pipeline/schema, provisioning VMs, and updating StatefulSets for pod resource modifications. It emphasizes using default values unless calculated resources exceed them and provides examples for handling the largest project scenarios.
Steps to allocate resources
| Step Name | Action | Purpose | Details and Considerations |
|---|---|---|---|
| Step 1: Estimate the workload | Identify the number of pipelines or schemas involved in each Data Integration use case (such as Continuous Replication, Mainframe Replication, and Fast Load). | Establish the scale and complexity of the integration workload to guide accurate resource planning in the following steps. | Count the number of pipelines or schemas by use case. This figure will be used to scale the resource requirements per pod in subsequent steps. |
| Step 2: Per pipeline/schema resource requirements | Refer to predefined tables that list pod-level vCPU, JVM memory, pod memory, and disk usage per pipeline or schema for each use case. | Understand the base resource footprint of each pipeline or schema and identify the relevant pods per use case. | Use per-unit pod resource values for Continuous Replication, Mainframe Replication, and Fast Load. These are the inputs for total resource calculation in Step 3. |
| Step 3: Calculate total resources | Multiply per-pipeline resource values by the total number of pipelines (from Step 1); apply fixed memory adjustments; add OS and agent resources; round off totals. | Generate complete resource requirements, including overhead and adjustments, to ensure the environment can support your workload efficiently. | Includes: Multiplying pipelines × per-pipeline pod values, adding fixed values (e.g., listener memory, cloud-applier off-heap), adding default pod memory for unused pods, including OS and agent resource needs. Final totals are rounded (e.g., 9826 millicores → 10 vCPUs). |
| Step 4: Provision the VM | Allocate a virtual machine with CPU, memory, and disk capacity based on the totals calculated in Step 3. | Ensure the infrastructure supports the workload with sufficient resources to avoid performance degradation. | Provision VM using final totals (e.g., 10 vCPU, 24 GB memory, 1 TB storage). Larger VMs do not automatically increase pod limits; manual configuration may still be needed. |
| Step 5: Update stateful sets to modify pod resources | Manually update the pod resource limits via StatefulSets if your calculated requirements exceed the default allocations. Reapply changes after upgrades. | Ensure that each pod has sufficient resources to handle the workload and avoid being limited by defaults. | Only required when calculated resources exceed defaults. Pod memory must always be greater than JVM memory. Changes will reset after an upgrade and must be reconfigured. |