Pipeline engines are processing engines that are necessary for running quality pipelines within the platform. They enable the execution of pipelines by providing the required computational resources and processing capabilities.
About pipeline engines
Pipeline engines are created and managed from the page. Each pipeline engine is created based on a connection. When setting up a connection, you have the option to either create a new pipeline engine or select an existing one. Supported connections for the pipeline engine include: Databricks, Google Dataproc, Precisely Agent, and Snowflake. When working with quality pipelines, you create a run configuration that utilizes the pipeline engine to execute the pipeline.
Why are pipeline engines important?
Pipeline engines are essential for running quality pipelines because they allow users to execute data processing and quality checks efficiently. By configuring and managing pipeline engines, users can ensure that their data pipelines run smoothly and meet their specific processing requirements. Pipeline engines also support different connection types, offering flexibility in how data is processed and managed.