Operational lineage harvesting captures current data lineage from Databricks and Snowflake by querying their system tables and access history views. This ensures your lineage views always reflect the actual state of your data pipelines without accumulating stale or phantom relations.
Operational lineage harvesting automatically captures and updates lineage based on actual data activity in your source platform. Unlike static lineage, which is derived from database structures and metadata, operational lineage reflects how data is actually used and moved through queries and workloads.
When enabled for a connection, operational lineage is refreshed during each catalog run to help ensure lineage views remain accurate and up to date.
Key capabilities
-
Operational lineage is refreshed during each catalog run. The system queries the audit tables within the time window you specify in the Runtime Lineage Window (Days) field. Lineage relationships from operations outside this window won't be captured, helping ensure you see only the lineage data relevant to your configured scope.
-
Harvest all available runtime lineage or limit harvesting to a specific time period using the Runtime Lineage Window (Days) field to help manage source platform query costs.
- Works with selective cataloging to focus lineage harvesting on selected assets that were created within your configured time window.
Requirements and prerequisites
- You must have the Advanced Connectivity Bundle (ACB) entitlement.
- Operational lineage is supported for Databricks and Snowflake.
Enable operational lineage on a connection
Operational lineage is configured on individual Databricks and Snowflake connections through the Runtime Engine section of the connection configuration dialog.
For detailed configuration instructions, refer Databricks connection configuration or Snowflake connection configuration
When to use operational lineage
Operational lineage is useful when you want to:
- Understand how data is actually used across your environment.
- Track data flows in complex pipelines and workloads.
- Keep lineage information current as data processes change over time.
- Reduce harvesting costs by limiting lineage collection to a specific time period.
- Capture lineage for Snowflake stored procedures without granting execute permissions to the service account.