To create a table in target BigQuery, SQL server, Snowflake, or Databricks and catalog the data connection, follow these steps:
- Select a schema and the tables will populate.
- Select Next. If the target tables have a red icon (X), the tables do not exist.
- Select the Create target tables if it does not exist option. The icons change from a red (X) to a yellow (!).
-
If you want to modify column mapping for a particular table, select
Map Fields
to open the
Map Fields
page and change options in the
Action
column and select
Save
.
Note: When creating a single table, table creation is initiated after you configure Map Fields (if Create table if not exists is selected) and select Save.
- Select Next to perform a bulk table creation on the dataset.
- The Creating Datasets page displays with a status for the number of target tables created. If the cataloging failed, go to Connections > Data Catalog Connections and verify the cataloging status.
While the Creating Datasets page is processing, navigation is unavailable. To open other pages, select the Data Integrity Suite banner.
BigQuery clustering behavior
When you create target tables in BigQuery, clustering works differently depending on how the tables are created. If you are creating more than one table and clustering is enabled:
- Tables are created with a
CLUSTER BYclause. - The cluster by fields are taken from up to the first four columns of the unique index or primary key for the source table. If there are fewer than four columns, only the available columns are used.
- These fields are automatically selected based on the key or index for the source table.
Single table creation
When creating a single table:- Clustering fields are based on the first four columns of the DKey.
- The DKey columns can be chosen by you.
- This is different from bulk creation, where clustering is always based on the key or index for the source table.
For Kafka targets:
-
The following options are currently
not supported
:
- Create topic if it does not exist
- Create subject if it does not exist
You must configure these options before you proceed.
Databricks target tables
When a mapped Databricks target table does not exist, select Create target tables if it does not exist and then select Next. The status for each table appears while the tables are created.
To create one target table while you update its column mappings, open Map Fields and select Save. If a target table already exists, resolve the conflict by selecting a different target table or removing the existing table before you try again.