Describes the required Snowflake roles, permissions, and objects needed to execute Data Quality pipelines with controlled access to compute resources, schemas, tables, and staging areas.
Overview
By creating a dedicated role and assigning the appropriate privileges, administrators can ensure that the pipeline has controlled access to compute resources, schemas, tables, and staging areas required during execution.
The configuration enables the pipeline to:
- Access source data for rule evaluation
- Create and manage intermediate processing objects
- Store temporary execution files in internal stages
- Write processed results to target tables
- Create supporting objects such as file formats and functions
Temporary objects and staged files may be created during pipeline execution to support processing. These objects are used only for the duration of the pipeline run and are automatically cleaned up after completion to maintain a secure and controlled environment.
This setup ensures efficient pipeline execution while maintaining proper access control and data governance within Snowflake.
Setup Steps
| Step | Action | Description / Command |
|---|---|---|
| 1 | Create role | Create a dedicated role used to execute the Data Quality
pipeline. |
| 2 | Grant warehouse usage | Allow the role to use the warehouse responsible for running pipeline
workloads. |
| 3 | Grant database access | Allow the role to access the database used by the
pipeline. |
| 4 | Grant schema access | Grant access to the schema where the pipeline reads and writes
data. |
| 5 | Grant object creation permissions | Allow the pipeline to create processing objects such as stages, file
formats, and
functions. |
| 6 | Grant read access | Allow the pipeline to read data from existing and future
tables. |
| 7 | Grant write access to existing tables | Allow the pipeline to insert, update, delete, or truncate
records. |
| 8 | Grant write access to future tables | Ensure future tables automatically inherit write
permissions. |
| 9 | Grant table creation permission | Allow the pipeline to create new tables when
required. |
| 10 | Create internal stage | Create an internal stage used for temporary pipeline
files. |
| 11 | Grant stage permissions | Allow the role to read and write files in the
stage. |
| 12 | Create pipeline user | Create a dedicated Snowflake user for pipeline
execution. |
| 13 | Assign role to user | Grant the role to the pipeline
user. |
| 14 | Allow internal stage unload | Allow the pipeline to unload intermediate results to internal
stages. |
Why These Permissions Are Required
- CREATE STAGE: Allows the pipeline to create temporary storage locations for intermediate execution files.
- CREATE FILE FORMAT: Enables the pipeline to define formats required for reading and writing staged data.
- CREATE FUNCTION: Allows creation of user-defined functions used during rule evaluation or transformations.
- SELECT: Provides read access to source tables required for rule evaluation.
- INSERT, UPDATE, DELETE, TRUNCATE: Allows the pipeline to write results and manage intermediate tables.
- CREATE TABLE: Enables creation of result tables or temporary processing tables.
- READ and WRITE on Stage: Allows uploading and retrieving intermediate files used during execution.
- The pipeline may create temporary stages, file formats, or functions during execution.
- Intermediate data may be written to the internal stage while rules are evaluated.
- Temporary files and objects are used only during pipeline execution.
- Cleanup of most temporary artifacts is automated after the pipeline run completes.
- The staging schema and stage object remain available for reuse across pipeline runs.