A step-by-step process for estimating and configuring resource allocation for Data Integration workloads. Use default values unless your calculated requirements exceed them.
Steps to allocate resources
| Step Name | Action | Purpose | Details and Considerations |
|---|---|---|---|
| Step 1: Estimate the workload | Identify the number of pipelines or schemas involved in each Data Integration use case (such as Continuous Replication, Mainframe Replication, and Fast Load). | Establish the scale and complexity of the integration workload to guide accurate resource planning in the following steps. | Count the number of pipelines or schemas by use case. This figure will be used to scale the resource requirements per pod in subsequent steps. |
| Step 2: Per pipeline/schema resource requirements | Refer to predefined tables that list pod-level vCPU, JVM memory, pod memory, and disk usage per pipeline or schema for each use case. | Understand the base resource footprint of each pipeline or schema and identify the relevant pods per use case. | Use per-unit pod resource values for Continuous Replication, Mainframe Replication, and Fast Load. These are the inputs for total resource calculation in Step 3. |
| Step 3: Calculate total resources | Multiply per-pipeline resource values by the total number of pipelines (from Step 1); apply fixed memory adjustments; add OS and agent resources; round off totals. | Generate complete resource requirements, including overhead and adjustments, to ensure the environment can support your workload efficiently. | Includes: Multiplying pipelines × per-pipeline pod values, adding fixed values (e.g., listener memory, cloud-applier off-heap), adding default pod memory for unused pods, including OS and agent resource needs. Final totals are rounded (e.g., 9826 millicores → 10 vCPUs). |
| Step 4: Provision the VM | Allocate a virtual machine with CPU, memory, and disk capacity based on the totals calculated in Step 3. | Ensure the infrastructure supports the workload with sufficient resources to avoid performance degradation. | Provision VM using final totals (e.g., 10 vCPU, 24 GB memory, 1 TB storage). Larger VMs do not automatically increase pod limits; manual configuration may still be needed. |
| Step 5: Update stateful sets to modify pod resources | Manually update the pod resource limits via StatefulSets if your calculated requirements exceed the default allocations. Reapply changes after upgrades. | Ensure that each pod has sufficient resources to handle the workload and avoid being limited by defaults. | Only required when calculated resources exceed defaults. Pod memory must always be greater than JVM memory. Changes will reset after an upgrade and must be reconfigured. |