Set up automated profiling schedules to run data quality checks at regular intervals. Scheduling is optional and runs in GMT format.
By default, scheduling is off during profile creation. All schedules
use GMT format.
Plan your schedule: How frequently you want a profile to run depends on your needs and the resources available to you. You can experiment with different settings to find an optimal schedule that meets your requirements.
Consider these factors as you decide:
- The size of your data: If you have a large dataset, running a profile daily may consume a lot of resources. A weekly schedule might be more efficient.
- The type of data: If you're monitoring critical data, you may want to run the profile more frequently to detect anomalies as early as possible.
- The specific use case: You may want to run the profile on specific days or at specific times. For example, if you're monitoring sales data, you may want to schedule the profile to run on Monday morning to analyze the previous week's sales data.
- The cost: Balance the benefits of more frequent monitoring with usage based pricing models popular with cloud infrastructure vendors. Increased frequency may also result in more alerts you need to review.
- Go to .
-
To enable profile scheduler, select the
Scheduler status for the profile.
- Click the Enable Schedule toggle to turn it on.
- Review the Occurrence settings.
- Click Close. You return to the Profilers page and the profile saves.
- Click Save & Run to save and run the profile immediately. It will continue running on your selected schedule.
-
To edit profiling scheduler:
-
To disable profiling scheduler:
- In the Scheduler column, click the Enabled status for the profile you want to disable.
- Click the toggle button under Enable Schedule. The scheduler is now disabled.
Scheduler settings
Scheduler settings define when and how profiling runs. The
profiler checks data for completeness, uniqueness, and frequency
distribution based on your configured rules.
- Daily: Run profiling every day at a specified time.
- at: Time when profiling starts (hh:mm).
- Example: Daily at 09:00 runs profiling every day at 9:00 AM.
- Weekly: Run profiling on selected days each week.
- On: Day(s) of the week when profiling runs.
- at: Time when profiling starts (hh:mm).
- Example: On Monday, Wednesday, Friday at 09:00 runs profiling on those days at 9:00 AM each week.
- Monthly: Run profiling on a specific date or day pattern each
month.
- The day of every month at
time: Profiling runs on the selected
date each month.
- Example: The 12th of every month at 11:00 AM runs profiling on the 12th at 11:00 AM each month.
- The first or last
day of every month at
time: Profiling runs on the selected
day pattern each month.
- Example: The first Wednesday of every month at 11:00 AM runs profiling on that day at 11:00 AM each month.
- The day of every month at
time: Profiling runs on the selected
date each month.
Note: The scheduler uses 12-hour format and
displays your system time by default.
View scheduler status and run history
Monitor the status of your profiling schedules and review past run information to track profiling performance and troubleshoot issues.
You can now see the complete run history
for the selected profile's schedule, including when it ran, how long it
took, and whether each run succeeded or failed.