Quality jobs provide real-time visibility into the progress and outcomes of profiling, quality rule evaluations, and pipeline executions. Efficient job management helps maintain data integrity and supports informed decisions based on current results.
Manage assessment jobs
- Go to and select
Assessments. The
Assessments tab displays
the following details for jobs.
- Job type: Displays
the current job as an assessment job. A
scheduler icon indicates whether the job is scheduled.What's changed: In the new user experience, Job type appears as Assessment, and the Assessments tab includes All statuses and All users filters, and a User column. For more information on the new experience, see About the new user experience.
- Datasource: Displays the datasource associated with the job.
- Connection: Displays the name of the connection associated with the job.
- Started: Indicates when the job was started.
- Duration: Indicates the total duration taken for job completion.
- Status: Indicates the
status of job completion.
- Complete: Indicates that the assessment job completed successfully.
- Failed: Indicates that the job failed due to an underlying cause.
- Partial: Indicates that part of the assessment job completed successfully.
- In Progress: Indicates that the job execution is in progress.
- Stopping: Signifies that the process to halt the job has commenced and it is currently in progress.
- Stopped: Denotes that the job execution has been halted.
Note:- Whenever an in progress job is stopped, the system shows a stopping state to provide feedback, especially when the process takes significant time.
- If a partial set of jobs fails within the application, the job run status is marked as failed. By hovering over the status indicator, you can view the number of jobs that succeeded, the number of jobs that failed, and the number of jobs that were stopped.
- If any job is stopped in between, the overall status will be displayed as stopped, regardless of whether other jobs have succeeded or failed.
- Job type: Displays
the current job as an assessment job. A
scheduler icon indicates whether the job is scheduled.
-
On the assessment jobs page, select the required job. The assessment page displays the following details:
- Job Type: Type of job, which can be a profiling job, rule execution job, or a job associated with an observer rule.
- Started: Indicates the timestamp when the job started.
- Duration: Indicates the total duration of the job run.
- Status: Indicates
the status of job run.
- Complete: Indicates that the job processing was successful.
- Failed: Indicates that the job processing failed due to underlying reasons.
- In Progress: Indicates that the job execution is in progress.
- Stopping: Signifies that the process to halt the job has commenced and it is currently in progress.
- Stopped: Denotes that the job execution has been halted.
- Dataset path: Indicates the path of the dataset associated with the assessment job.
- For default rules, the job name is fetched from
the associated connection and datasource
details. Job naming convention:
<Datasourcename_Connectionname_Assessment> - For custom rules, the job name is fetched from
the associated rule name. Job naming convention:
jobs:
<Rulename_assessment> - For those jobs that are triggered via developer
portal, the job name is displayed in the format
below. Job naming convention:
<Custom_Assessment_CorelationID>
- To stop all underlying in progress jobs in assessment: Click the ellipsis next to the associated job and select Stop all jobs.
- To selectively stop assessment jobs: For a job type, open the Assessments page, select the checkbox next to the associated job, and click Stop selected. This will stop the selected in progress job in the application.
Understand pipeline job types
Pipeline jobs include different job types to support specific evaluation and analysis tasks. Each job type enables you to perform targeted assessments and ensure quality.
- Profile: A job related to profiling datasets. Triggered when the Profile Datasets toggle is enabled during onboarding.
- Rule: A job related to executing data quality rules. Triggered when the Run Data Quality Rules and Calculate Scores toggle is enabled during onboarding.
- Profile, rule: A job that includes both profiling and data quality rule execution. Triggered when both Profile Datasets and Run Data Quality Rules and Calculate Scores toggles are enabled during onboarding.
- Schema drift: A job related to detecting schema drift changes. Triggered during onboarding when schema drift detection is configured.
- Failed records: A job related to generating failed records samples. Displayed when the Show failed records sample toggle is enabled in a custom rule.
- Freshness: A job related to detecting data freshness. Included in the assessment when the freshness rule is selected during observer setup and triggered during scheduled execution.
- Semantic Type* Preview: A job dedicated to identifying semantic types. Initiated exclusively when semantic type detection is run on the profile page.
Manage pipeline jobs
When you create a pipeline run configuration, you define the environment, input and output data assets, connection, and pipeline engine. Each time a pipeline runs, the system logs it as a job with a specific job type. The Pipelines tab displays all pipeline jobs with the following controls:
- Click a column heading to reorder jobs in ascending or
descending order.
- ID: A sequential integer assigned when the job starts. Click the ellipsis to delete the job or start a quick run.
- Pipeline: Pipeline on which the job ran. Click the pipeline name to view the pipeline definition.
- Run Configuration: Environment in which the job ran.
- Start Time: Date and time when the job started.
- Duration: Time required to complete the job in HH:MM:SS format.
- User: Username associated with the job.
- Status: Current status of
the job. Possible values include:
- Ready: Prepared and waiting to run.
- Pending: Queued for execution.
- Running: Currently running.
- Paused: Running is paused.
- Successful: Completed without errors.
- Failed: Encountered an error and did not complete.
- Terminating: In the process of stopping.
- Cancelled: Manually cancelled.
- Unknown: Status cannot be determined.
- Click the Filter button in a column heading to filter jobs by column values or clear an existing filter.
- Select the check box next to the ID column to filter or delete jobs.
- Click the Refresh button on the toolbar to refresh the job list.
- Cancel jobs: You can cancel any job with a status
other than Successful, Failed, Terminating, or
Cancelled.
- Select the check box next to one or more jobs you want to cancel.
- On the toolbar, click the Cancel selected job execution button.
- Alternatively, click the ellipsis next to the
job ID and select
Cancel.
The job status changes to Cancelled for the selected job.
- Delete job history: Remove job records from the
history to clean up the job list.
- To delete a single job, click the ellipsis in the ID column and select Delete.
- To delete multiple jobs, select the check boxes next to the jobs you want to delete, and click the Delete button on the toolbar.
- Run a job: Run a pipeline job immediately using the same configuration as a previous run. Click the ellipsis next to the job ID and select Quick Run. The pipeline job runs immediately using the same run configuration as the selected job.