Freshness rule

Data Integrity Suite

Product

Spatial_Analytics

Data_Integration

Data_Enrichment

Data_Governance

Precisely_Data_Integrity_Suite

geo_addressing_1

Data_Observability

Data_Quality

dis_core_foundation

Services

Spatial Analytics

Data Integration

Data Enrichment

Data Governance

Geo Addressing

Data Observability

Data Quality

Core Foundation

ft:title

Data Integrity Suite

ft:locale

en-US

PublicationType

pt_product_guide

copyrightfirst

2000

copyrightlast

2025

Data freshness is an essential part of data quality. Freshness refers to the degree to which your data is up to date and accurate. Stale data may be outdated, incomplete, or inaccurate. You can configure a freshness rule to generate alerts when your data fails to update at the expected frequency.

For example, imagine you expect your sales data to update every three days. If the update fails and misses the three-day frequency mark, the system will alert you to the failure. Now you have an opportunity to refresh or update the data before individuals use it for decision making.

You can configure a confidence-based or a threshold-based freshness alert.

Considerations for generating freshness alerts

Using Databricks datasource

To capture Freshness alert for the Databricks connections, the DESCRIBE HISTORY feature has to be enabled in Delta Lake. The DESCRIBE HISTORY command reads the log and returns a history of operations on the table. Databricks stores the last modified table information for up to 30 days only. So, the freshness alert for monthly interval is generated upto 30 days only. For more details, see DESCRIBE HISTORY.

Only WRITE, UPDATE, and DELETE DML operations are used to calculate the last modified time of the table.

Execute the following SQL query to retrieve data:

select max(timestamp) as PRECISELY_VALUE from (DESCRIBE HISTORY <database_name><schema_name><table_name>) where operation='WRITE' or operation='UPDATE' or operation='DELETE'

Supported Operations

WRITE: A new write operation to the table.

MERGE: A merge operation to perform upserts (insert, update, delete).

DELETE: A delete operation.

UPDATE: An update operation.

OPTIMIZE: Optimizes the table by performing compaction on small files.

VACUUM: Cleans up older data files that are no longer needed.

Supported Versions for Describe History

Databricks Runtime 7.x	Delta Lake 0.8.x	Supports DESCRIBE HISTORY, basic Delta Lake functionality.
Databricks Runtime 8.x	Delta Lake 1.0+	Supports DESCRIBE HISTORY, time travel, and Delta optimizations.
Databricks Runtime 9.x+	Delta Lake 1.0+	Full support for DESCRIBE HISTORY, time travel, schema evolution, and performance improvements.

Using Redshift datasource

Redshift stores the last modified table information for up to seven days only. So, the freshness alert can be generated only for daily and weekly intervals.
The Redshift user must be a superuser or have access to the SVV_TABLE_INFO table. To permit a user to query the view, grant SELECT permission on SVV_TABLE_INFO to the user. For more details, see SVV_TABLE_INFO.
Only DML operations are used to calculate the last modified time of the table. The DDL query operations will not generate freshness alerts.

Using Snowflake datasource

In Snowflake, you use the LAST_ALTERED column to get freshness at table level. This supports the Add, Delete, Update operation.

Execute the following SQL query to get information on Freshness

SELECT TO_TIMESTAMP(LAST_ALTERED) AS PRECISELY_VALUE, NULL AS PRECISELY_COMMENT FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = <schema_name> AND TABLE_NAME = <table_name> UNION ALL SELECT NULL AS PRECISELY_VALUE, 'table or view does not exist' AS PRECISELY_COMMENT WHERE NOT EXISTS (SELECT 1 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = <schema_name> AND TABLE_NAME = <table_name>)

The query, when executed successfully, returns the LAST_ALTERED date. This date is useful for tracking the Freshness metric.

Configure a confidence-based alert

Set a confidence-based alert if you want to get alerts based on how certain the system is about the freshness of the data. You set the range of confidence you desire, and the Observability Adaptive Model will alert you when it is certain that the table has failed to update at the expected frequency.

Use these steps as you create an Observer to configure a freshness rule with a confidence-based alert. You can also edit existing Observer rules.

After you select the data assets to observe on the Create Observer page, click Next.
In the Freshness row, click the gear icon in the Configure column.
Select the confidence level for generating an alert:
- Drag the lower value slider and higher value slider to set a range of percentages, or
- Enter the lower and higher value in the text box to set the range of percentages.
Click Save.

Note: By default, a critical alert generates when the certainty is greater than 80%. You can only adjust the yellow alerts or warnings. You can set the yellow range from 20% to 100%. Below 20%, the confidence level is so low that it will generate a lot of noise in your alerts.

For example, set the warning range at 60% to 80% to get warning alerts when the confidence of failure to update the table is detected in that range. You will get critical alerts when the certainty is above 80% and no alerts will be generated when the certainty is below 60%.

Note: The alerts will not be generated for lower values if the percentage overlap, the alerts will be generated only for values beyond the higher range. It will set to a default range if your range overlaps at 100%.

For example, alerts will not be generated below 80% for the range selected as 80% to 80%. Alerts will be generated only above 80%. If you set the range as 100% to 100%, the range will revert to default range as 60% to 80%.

Configure a threshold-based alert

Set a threshold-based alert if you want to manually set limits to trigger alerts. You set a custom threshold to define the expected frequency of updates.

For example, if you set the Freshness Frequency to three days, alerts will be generated when the table fails to update every three days.

Note: Threshold-based alerts will always generate as critical alerts.

Use these steps as you create an Observer to configure a freshness rule with a threshold-based alert. You can also edit existing Observer rules.

After you select the data assets to observe on the Create Observer page, click Next.
In the Freshness row, click the gear icon in the Configure column.
Click the Threshold-Based Alerts radio button.
In the Freshness Frequency field, type a number or use the incremental arrows to set the expected schedule for the table update.
Click Save.

Note: If your freshness rule is not generating expected alerts, check the Observer scheduler. An Observer must run as frequently or more frequently than the freshness rule.

For example, if your freshness frequency is set to run every four days, but your Observer only runs every seven days, it will not detect whether the table failed to update at the expected frequency.