Organizations often have diverse datasources that they need to access from the Data Integrity Suite. Connecting to data is the first step in processing, analyzing, and storing information.
The data connection is an interface between Data Integrity Suite, supported source and target databases. The datasource interface defines credentials, source and other specific information necessary to connect to external data. After you establish a connection, you can catalog that connection. A cataloged connection allows Data Integrity Suite services to catalog and access associated databases or data warehouses.
The Connections page serves as a centralized location for managing all connections associated with the datasource. Access this page by navigating to ConfigurationDatasources. Then select a datasource to view its associated connections and the following details for each connection:
- Type: Specifies the type of datasource.
- Description: Specifies the details about the datasource.
- Connections: Specifies the total number of connections in the datasource.
- Datasets: Specifies the total number of datasets associated with the datasource.
- Fields: Specifies the total number of fields associated with the datasource.
- Edit Details: Enables editing of the name and description of the datasource.
- Search: You can efficiently locate a specific connection by entering its name, hostname and connection type in the search bar.
- + Add : Button to add a new datasource. Once created, it will appear in the datasource list and you can use it to import data.
- Other actions: On this page, you also have the option to Test, Duplicate or Remove a datasource.
Connections page
The main section of the Connections page contains a list of all configured connections for a datasource. For each connection, you can see details like:
- Connection: Specifies the name of the data connection.
-
Catalog Summay: Displays the current cataloged status of the data connection. Cataloging information updates automatically every 30 seconds.
- Failed with <number of errors> errors: Displayed when an error occurs during cataloging. You can click this status link to view the error details.
- Partially Cataloged: Displayed when few tables within the schema has cataloged information but not all.
- Cataloging: Displayed when the cataloging process is ongoing.
- Starting: Displayed when the resources needed to catalog takes time to get started.
- Cataloged: Displayed when the schema is fully identified and cataloged, meaning all necessary metadata and information about them is available in the catalog.
- Not Cataloged: Displayed when the schema is not yet been identified and cataloged.
- Datasets: Specifies the number of datasets associated with the connection.
- Fields: Specifies the number of fields associated with the connection.
- Host or URL: Specifies the database name or address, along with the IP/hostname port combination. For Kafka data connections, brokers' IP/hostname port combinations are displayed in a comma-separated list.
- Schedule: Specifies whether a schedule exists for this datasource. For example: Every month at 6:45pm.
Once the datasource or connection is established, the related datasets and their fields can be accessed from the Catalog section in the main navigation menu.
After a datasource is configured, it becomes available for both data quality and data observability tasks. For data quality, you can ensure that data is accurate, consistent, and fit for purpose across operational and analytical systems. This involves identifying and resolving data issues, observing trends, and detecting anomalies. In data observability, you can track and monitor the health of data, assess data freshness, and measure processing performance. By scheduling data profiles and reviewing connection setups on the Connections page, you can ensure a comprehensive view of data quality and operational performance, facilitating timely interventions and maintaining robust data integrity.