Add datasource and connection

Add a new datasource

Add a connection to the datasource:

1. Configuring Connections:
   1. On the Add Datasource wizard, provide a Name and Description for the connection.
   2. Specify properties for the connection, depending on your Datasource type.
      Note: A connection type can have its data in either Cloud or On-Premises.

      • For Cloud connector type, by default, the selection for the Agent field is none.
      • For On-Premises connector type, if there are multiple registered agents, you can choose the appropriate agent for this connection. If there is only 1 agent available in the list, it will be selected by default.
   3. Click Test to verify the connection.
   4. Upon successful connection, a confirmation message will appear. Select Next to proceed.
2. Adding Data Assets: When adding Data Assets to a datasource, a list of available schemas for the selected datasource will be displayed.
   1. Cataloging options:

      You can choose to catalog all datasets or select specific datasets.

      1. Catalog All Datasets:

         1. In the Data Assets step, check the box next to All datasets to catalog.

         2. This selects all available data assets to associate with the datasource.

      2. Selective cataloging datasets:

         1. In the Data Assets step, all available datasets are displayed by default.

         2. Click All datasets to open the Datasets to Catalog window.

         3. From this window, select the specific datasets you want to catalog.

         4. Click Save to apply your selections.

      Note: The selective cataloging feature is only available for RDBMS, Snowflake and Databricks datasources.
   2. Click Next to proceed.
3. Analyzing Insights: During the datasource configuration, you can analyze data by enabling profiling and quality rule execution to gain insights into patterns, statistics, and data quality scores.
   1. Configure pipeline engine:
      1. In the 'Pipeline engine for performing data analysis' dropdown, a list of compatible pipeline engines for the selected datasource is displayed.

         Note: Under Insights, the Pipeline Engine dropdown now displays all pipeline engines associated with the same user and the same host as the current connection.

      2. Once a pipeline engine is selected, the Edit button is enabled.

      3. Click Edit to open the Edit Pipeline Engine window and configure engine settings.

      Tip: If no compatible engine is available, click Create Pipeline Engine to add a new engine.
   2. Enable data analysis options:
      1. Profile Datasets:

         1. Turn on the Profile Datasets toggle to initiate profiling on the selected data assets.

         2. Profiling helps you understand various characteristics of your data, such as data types, distributions, and completeness.

         3. The profiling duration depends on the volume of data in the selected assets.

      2. Run Quality Rules and Calculate Scores:

         1. Turn on the Run Quality Rules and Calculate Scores toggle to evaluate data quality.

         2. This action runs predefined data quality rules and generates quality scores at the field and dataset levels

         3. The scores help assess the overall quality and trustworthiness of the data.

      3. Semantic type identification:^{* Limited Availability} Enabling this toggle detects the semantic type associated with the fields upon successful discovery. Semantic type identification will run once for each asset and will not be re-run during subsequent discovery processes.
         Note: When you turn on the profile datasets toggle, the semantic type detection toggle will be disabled, since profiling automatically collects semantic type information. If you don't want full profiling and only need semantic type detection, simply disable the profile toggle and enable the semantic type identification toggle instead.

      Click Next to proceed.

      Limited Availability: This feature is currently available only in select workspaces and might be subject to change before general availability.
4. Scheduling Updates:

   1. Schedule Discovery: To activate schedule discovery, toggle the Schedule Discovery switch to ON. This would initiate the discovery recurringly at the configured schedule.

   2. Schedule Insight: To activate scheduling for the execution of rules and profiling, turn on either the Profile Datasets or Run Data Quality Rules toggle, or both in the insights page. This enables the insight scheduling option. Profiling and execution of rules is carried out at the configured insight schedule.

      Note: To access the insights schedule, the Schedule Discovery toggle must be enabled when a new data source is added.

   Table 1. Scheduler settings

   Occurrence	Description
   Daily	Specifies settings for the scheduler to run every day. Local time: The time at which scheduling starts (hh:mm). Note: The scheduler uses a 12-hour time format and the default time is 1:00 am. Example: If the schedule is selected for daily local time<hh:mm>, then scheduler will run daily at <hh:mm>.
   Weekly	Specifies settings for the scheduler to run every seven days. Days: The day(s) in the week that scheduler will run. Local time: The time at which scheduling starts (hh:mm). Note: The scheduler uses a 12-hour time format and the default time is 1:00 am. Example: If the schedule is selected On<selected days> and local time<hh:mm>, then scheduler will run on the <selected days at <hh:mm> for every week.
   Monthly	Specifies settings for the scheduler to run every month. TheSelect dayof every month at<time>: The scheduler will run on the every selected date of the month. Example: If the schedule is selected as The12of every month at 11:00 AM, then profiling or observer will run on every 12th of the month at 11:00 AM. The<first or last><day>of every month at <time>: The scheduler will run on the selected date of the month. Example: If the schedule is selected for The firstWednesday of every month at 11:00 AM, then profiling or observer will run on the first Wednesday of every month at 11:00 AM. Note: The scheduler uses a 12-hour time format and the default time is 1:00 am.

   Note: Use the Edit Schedule feature to edit the schedule for discovery and insight. If the toggle is disabled for insight schedule while establishing a new datasource, the edit option will be disabled for insight schedule. In this case, only the discovery schedule is shown by default.

   Tip: On the Schedule page, you can choose to use the same schedule for insight as discovery by selecting the Same as discovery schedule checkbox in the insight schedule page. For existing connections that are already added to the workspace, the discovery schedule is automatically applied to the insight schedule only if either the Profile Datasets or Run Data Quality Rules toggle, or both are turned ON. You can run profiling based on the insights schedule independent of the configured discovery schedule.
5. To activate scheduled discovery and scheduled insight, turn ON the Schedule Discovery and Schedule Insight toggles. You can enable recurring discovery using the Enable discovery toggle and recurring execution of rules and profiling using Enable insights toggle. When the Start discovery now and Start Insight now toggles are turned ON, the system immediately begins discovery, cataloging, profiling and execution of rules of all selected data assets upon adding the datasource.
   Note: It is observed that when default rules are executed in the Onboarding flow for Databricks connections that have large datasets, the rule execution fails for the selected datasets.
6. Select Finish to complete the process.

Once created, you can add additional connection to the datasource. The datasource will now include all associated connections. You can easily access any of these connections directly from the datasource without having to search for the individual connections.

Add connection to an existing datasource

Additional connections can be added to existing datasources. You must have an existing datasource configured with at least one connection.

To add a new connection to an existing datasource:

The datasource will now include all associated connections. You can easily access any of these connections directly from the datasource without having to search for the individual connections.