To create a table in target BigQuery, SQL server, or Snowflake and catalog the data connection, follow these steps:
Note: The clustering option is now available for BigQuery target
tables in continuous replication pipelines.
- Select a schema and the tables will populate.
- Click Next, if the target table(s) have a red icon (X), the tables do not exist.
- Click the Create target tables if it does not exist option. The icons change from a red (X) to a yellow (!).
- If you want to modify column mapping for a particular
table, click Map Fields to open the Map
Fields page and change options in the
Action column and click
Save.Note: When creating a single table, table creation is initiated after you configure Map Fields (if Create table if not exists is selected) and click Save.
- Click Next to perform a bulk table creation on the dataset.
- The Creating Datasets page displays with a status for the number of target tables created. If the cataloging failed, go to Connections > Data Catalog Connections and verify the cataloging status.
While the Creating Datasets page is processing, the navigational is grayed-out. If you want to navigate to other sections, click the Data Integrity Suite banner.
Note:
BigQuery clustering behavior
When you create target tables in BigQuery, clustering works differently depending on how the tables are created. If you are creating more than one table and clustering is enabled:
- Tables are created with a
CLUSTER BYclause. - The cluster by fields are taken from up to the first four columns of the source table’s unique index or primary key. If there are fewer than four columns, only the available columns are used.
- These fields are automatically selected based on the source table’s key or index.
Single table creation
When creating a single table:- Clustering fields are based on the first four columns of the DKey.
- The DKey columns can be chosen by you.
- This is different from bulk creation, where clustering is always based on the source table’s key or index.
For Kafka targets:
- The following options are currently not supported:
- Create topic if it does not exist
- Create subject if it does not exist
You must configure these options before you proceed.