Configure transformation steps - Precisely Data Integrity Suite

Data Integrity Suite

Product

Spatial_Analytics

Data_Integration

Data_Enrichment

Data_Governance

Precisely_Data_Integrity_Suite

geo_addressing_1

Data_Observability

Data_Quality

dis_core_foundation

Services

Spatial Analytics

Data Integration

Data Enrichment

Data Governance

Geo Addressing

Data Observability

Data Quality

Core Foundation

ft:title

Data Integrity Suite

ft:locale

en-US

PublicationType

pt_product_guide

copyrightfirst

2000

copyrightlast

2026

Transformation steps clean and standardize your data. Add multiple steps to build a pipeline that solves your data quality challenges. Steps run sequentially. The output of each step becomes the input for the next step, with the final output coming from the last step. Each step has configurable settings and output parameters you can preview before saving.

On the pipeline edit page, use the following controls:

Add Step: Select to add a step at the end of the pipeline.
Step menu (vertical ellipsis): Select a step, then choose the ellipsis to add, edit, rename, move, or delete it. Choose to insert before or after the selected step.
Insert between steps: Point at the connection between two steps and select the add button.
Sample data: Select a step to view sample data after it completes. Select Original Dataset to view original data.
Needs Data Description icon: A step requiring a data subscription displays this icon. Hover to see details.
Search by column: Enter a column name to select it. The selection is added to the Columns box when editing a transformation.
Search by data: Enter data to filter rows in the inspection table.
Sample data table: Each column heading shows the field name, base type (string, integer), and semantic type (First Name, Postal Code). Select a heading to choose the column. Hold Ctrl and select additional headings to choose multiple columns.
Step Preview: Turn on to see data before and after transformation for any step. Select Preview while editing a step to see its effect on your data.

Add transformation steps

Transforms are configured to form transformation steps in a pipeline. These steps change the format, structure, or values of data uploaded by a dataset. A pipeline executes transforms in the order you specify.

Go to Quality > Pipelines.
Find the pipeline you want to edit in the Name column. Use the Search box to filter by pipeline name.

Tip: Scroll the Dataset column to find the dataset for a pipeline.
Select the ellipsis in the Name column, then choose Edit. Alternatively, select the pipeline name.
Choose how to add a step to the pipeline:

Append a step: Select Add Step after the last step.

Insert before a step: Hover over the step, select the shortcut menu, then choose Add Step > Before.

Insert after a step: Hover over the step, select the shortcut menu, then choose Add Step > After.

Insert between two steps: Hover over the connection line between steps and select the add step button.

Insert at start or end: Hover over a step, select the shortcut menu, then choose Add Step > To Start or Add Step > To End .
Note: These options do not appear on the first or last step.
In the Add Step dialog, select the transformation you want to add.
Options for the selected transformation appear.
Edit the Step name to describe this transformation step.
Configure the remaining options for the transformation. Options depend on the selected transformation. See Data Quality pipeline steps reference for details.
Optional: Select Preview to view affected column values before and after the transformation. Optionally, select Transformation Preview to view sample data.
Select Save.
The transform is added to the selected location in the pipeline.
Optional: Drag a transformation step left or right to change its run order. Alternatively, choose the vertical ellipsis and select Reorder > Move To > Start, Right, Left, End.

View errors in the Data Quality pipeline

After you select Confirm to save changes, affected steps show error and warning symbols in the confirmation dialog. A red box shows the total number of errors and warnings in the pipeline.

Select a warning or error symbol on a step to view the list of errors to fix.
To view all errors and warnings, select the error and warning count box to expand the Errors and Warnings list panel.
This panel shows errors and warnings for every step. It is useful for complex pipelines with many steps.
Scroll the list and select any step label to expand and edit settings for that step.

When a step displays the error symbol, you cannot view the outcome of the step or any subsequent steps. If you select the Transformation Preview checkbox to view the affected columns, they no longer show any sample data. When a step displays the warning symbol, you can view the outcome of the step, but only if it is not preceded by any steps that display the error symbol.

View data errors

A box on the top right corner of the pipeline panel shows the total number of data errors in the sample dataset.

Select the box to expand the Issues panel.
The Issues panel shows the total number of issues. The Data Errors page identifies and counts each error type.
For each error type, use the directional control buttons to move focus between affected fields.

Group transformation steps

Grouping steps helps you manage large pipelines efficiently. You can group steps by function, create groups within groups, and apply conditions to groups. This simplifies navigation and provides a focused view of your pipeline.

For example, use parsing steps to clean customer names, general steps to amend fields, addressing steps for location insights, and enrich steps to enhance data. Group these by function to eliminate horizontal scrolling and improve clarity.

You can also create groups with conditions. Steps in a conditional group apply only to fields that satisfy the group's condition.

For example, create a group with condition Name EQUAL Literal John. All parsing steps in this group apply only to fields named John.

With groups, you can:

Create multiple groups
Select steps and move them to a group or add them to a new group
Update or delete groups

Considerations while working with grouping:

A group must contain at least one step.
Open a group to drill down into a focused state.
Use drag-and-drop to reorder steps or groups.
The previous step or group's data is input for the next step or group.
The output preview for a group shows the preview of the last step in the group.
You cannot group both a step and a group together into another group.
You cannot group steps located on the left and right sides of a group.

Note: When working inside a group, use the last operator in that group to configure the step and investigate results.

Open the pipeline you want to edit.
Select the checkbox for each step you want to combine into a group.
The checkbox for a step appears when you point at the step in the pipeline editor.
Select Group in the top right corner of the pipeline editor.

The selected steps combine into a new group in the pipeline. To add more steps to an existing group, select the checkboxes for the group and the additional steps, then select Group again.

Apply conditions to transformation step or a group

Apply conditions to transformation steps or groups to target specific fields in your dataset.

Set conditions on a transformation step or group to apply transformations only to fields that satisfy the condition. Understand your dataset to target the fields you want to transform.

For example, use Parse Email with condition StateEqualLiteralNew York to parse emails only for users from New York. You can enter a literal value or select a field.

You can apply conditions to groups of transformation steps. The output reflects each transformation based on the group condition.

Choose between basic and advanced conditional evaluation:

Basic conditions perform straightforward data evaluation based on simple conditions. Use for simple tasks.
Advanced conditions handle complex data evaluation when basic conditions are insufficient. They provide greater flexibility for complex data transformations.

Considerations for applying conditions to transformation step or group:

You can generate conditional output for all transformation steps except: Copy Field, Filter Field, Rename Field, and Filter Row.
If a group contains steps that do not support conditions, applying conditions to the group generates conditional output based on the group condition.
If conditions exist on both individual steps and a group, the group condition intersects with individual step conditions using AND. For example, group condition Age = '30' and step condition FIRSTNAME=ucase('AMY') returns results only for Age = '30' AND FIRSTNAME=ucase('AMY').
Switching from advanced to basic conditions deletes your advanced conditions.

Create a new pipeline or open an existing one.
Open the transformation step or group to which you want to apply a condition.
Select the Condition tab on the right pane.
Select Basic or Advanced condition.
Select the condition for basic evaluation or type the condition expression for advanced evaluation.
Select Save to apply your changes.

To rename a step: Select the step you want to rename, edit and press Enter.
To preview the outcome of any step: On the pipeline pane, select the step and turn on Step Preview. Select any step in the pipeline to view its outcome. Select Original Dataset to view the input schema or choose the last step to view the output schema.
To reorder or delete transformation steps: Select Edit for a pipeline. A preview table shows sample data in the selected columns after the transformation step completes. Edit the name or other settings for the transform step and select Save .
- Choose Preview to view affected column values before and after the transformation or select Transformation Preview to view changes to columns acted upon by the operation.
- Drag a transformation step left or right to change its run order. Alternatively, choose the ellipsis and select Reorder > Move To > Start, Right, Left, End.
To delete a transform step: On the transformation steps panel, select the transformation step you want to delete. Choose the vertical ellipsis and select Delete and confirm with a Yes.

Basic condition

Basic condition performs data evaluation based on simple conditions.

Provide conditions in two ways:

Select <field>, <operator>, <literal> and type a value to match the condition.
Select <field>, <operator>, <field> and select a field to match the condition. Compare fields of the same data type. Use for all available data types.

Available operators depend on the selected field's data type.

For example, condition FirstnameequalJoe applies the transformation to fields where Firstname equals Joe.

Numeric and temporal field operators

Use numeric operators to evaluate values in Integer, Long, Float, Date, DateTime, and Time fields.

Operator	Binary or Unary	Expression true when
Equal	Binary	Field value equals a literal or field value.
Not equal	Binary	Field value does not equal a literal or field value.
Null	Unary	Field value is null.
Not null	Unary	Field value is not null.
Greater than	Binary	Field value is greater than a literal or field value.
Greater than or equal	Binary	Field value is greater than or equal to a literal or field value.
Less than	Binary	Field value is less than a literal or field value.
Less than or equal	Binary	Field value is less than or equal to a literal or field value.

String field operators

Use string operators to evaluate values in textual fields.

Operator	Binary or Unary	Expression true when
Equal	Binary	Field string value matches a literal or field value.
Not equal	Binary	Field string value does not match a literal or field value.
In	Binary	Field string value matches any value in a specified list. Type and press Enter to add each literal value.

Logical field operators

Use logical operators to check values in fields that represent either true or false conditions.

Operator	Binary or Unary	Expression true when
Equal	Binary	Field value equals True or False or the value in another Boolean field.
Null	Unary	Field value is null.
Not null	Unary	Field value is not null.

Advanced condition

Advanced condition handles complex evaluation operations. Type expressions using available operators and functions. The syntax is color-coded for clarity. Errors flag when you mistype or misuse an operator or function.

Note:

When using advanced conditions, use left and right operands with the operator.
Field names are case-sensitive. Incorrect casing does not flag an error and will not apply the condition.
For example,
                      ucase(FirstName)=true applies when you
                    have FirstName field. ucase(FIRSTNAME)=true
                    does not apply if your dataset lacks FIRSTNAME
                    field and no error flags.

Operators and their description

Operator	Description
= (equals)	Selects records where a field equals a specified value. Example: `Age = '30'` selects records where age is 30.
!= (not equal to)	Selects records where a field does not equal a specified value.
> (greater than)	Selects records where a field is greater than a specified value. Example: `Price > '50'` selects records with prices higher than 50.
< (less than)	Selects records where a field is less than a specified value.
>= (greater than or equal to)	Selects records where a field is greater than or equal to a specified value.
<= (less than or equal to)	Selects records where a field is less than or equal to a specified value.
IN	Selects records where a field matches any value in a specified list. Example: `City IN ['New York', 'Los Angeles']` selects records with cities matching either value.
IS NULL	Selects records where a field has a null or undefined value. Example: `Phone Number IS NULL` finds records where the phone number field is empty.
IS NOT NULL	Selects records where a field has a defined (non-null) value. Example: `Email IS NOT NULL` finds records where the email field is filled in.
AND	Logical operator combining multiple conditions. Returns results satisfying all conditions. Example: `Age > '25' AND Salary > '50000'`.
OR	Logical operator combining conditions. Returns results satisfying at least one condition. Example: `Category = 'Electronics' OR Price < '100'`.
!	Logical operator negating a condition. Returns results not meeting the criteria. Example: `State != 'New York'` selects data where state is not New York.

Functions and their description

Function	Description	Usage	Example
ucase	Selects a string with uppercase letters.	Standardize character case in a text field.	`FIRSTNAME=ucase('AMY')`
lcase	Selects a string with lowercase letters.	Standardize character case in a text field.	`LASTNAME=lcase('stamp')`
tcase	Selects a string with title case (uppercase first letter of each word).	Update text to title case.	`tcase('Mrs Mary Smith')=Name`
trim	Removes leading and trailing spaces from a string.	Clean data, especially user-inputted text with unnecessary spaces.	`trim('routley')=LastName`
ltrim	Removes leading (left-side) spaces from a string.	Clean data by eliminating leading spaces for consistent formatting.	`ltrim('AMY')=FirstName`
rtrim	Removes trailing (right-side) spaces from a string.	Standardize data by eliminating trailing spaces affecting comparisons or display.	`rtrim('stamp')=LastName`
lpack	Left-packs a string, removing leading spaces and adjusting remaining characters left.	Align text left while eliminating unnecessary leading spaces for data formatting.	`lpack('AMY')=FirstName`
rpack	Right-packs a string, removing trailing spaces and adjusting remaining characters right.	Align text right, removing trailing spaces for consistent formatting.	`rpack('stamp')=LastName`
mask	Applies a mask to a string, replacing certain characters with specified placeholders.	Verify datatypes in a generated pattern. Pattern AAANN indicates A contains alphabets and N represents numeric values.	`mask(Country)='AAAANN'`
substr	Extracts a substring from a string based on specified starting position and length.	Extract specific portions of text from a longer string. Two ways: `substr(string data, int beginIndex)` and `substr(string data, int beginIndex, int endIndex)`	`FIRSTNAME=substr('TAmy',2)` `FIRSTNAME=substr('TAmyT',2,5)`
concat	Joins two or more strings without space.	Combine information from multiple fields into a single string.	`concat('AMY',' ')=FirstName`
regexp	Applies a regular expression pattern to a string for pattern matching.	Complex pattern recognition and extraction in textual data.	`regexp(Email,'@.')='@g'`
replace	Replaces occurrences of a specified substring or pattern in a string with another substring.	Clean and update data, replacing specific elements with desired values.	`replace('Mircosoft','rco','cro')=Company`
split	Splits a string into an array of substrings based on a specified separator character.	Separate text data into individual components for analysis or processing.	`split('Sweden:BasicCountry',':',0)=Country`
length	Selects a string having the specified length (number of characters).	Know the size of a string, often used for validation.	`length(CollectionNumber) > 1`
to_number	Selects a numeric string having a specified numerical value.	Transform string representations of numbers into numeric formats for mathematical operations.	`to_number('100')=CustomerId`
to_int	Selects double value as per specified integer value.	Maintain data consistency and facilitate numerical calculations.	`to_int(22.53)=Currency`
occstr	Returns the position of the first occurrence of a specified substring in a string.	Locate specific substrings within text data.	`occstr(Email,'.','L')=3`
now	Selects the current date and time.	Provide a dynamic timestamp for real-time data capture or time-sensitive operations.	`now()=Totaltime`
isalpha	Selects a string containing only alphabetic characters.	Verify if a text field consists solely of letters.	`isalpha('test')`
isnumeric	Selects a string containing only numeric characters.	Facilitate validation of numeric data integrity in text fields.	`isnumeric('100')!=CustomerId`
isnull	Selects fields with the absence of any value.	Check fields for missing or unknown data.	`isnull(NAME)`
isempty	Selects an empty string.	Check that text fields are not null or empty.	`isempty('')!=PostalCode`
contains	Selects a string containing a specific substring.	Search for a particular set of characters within a larger string.	`contains('@', Email) != 0`
to_date	Selects the specified ISO format date (YYYY-MM-DD).	Transform date representations in string format for date-related operations.	`to_date('1973-06-05') = Date`
to_time	Selects the specified ISO format time (HH:MM:SS).	Transform time representations in string format for time-related calculations.	`to_time('09:04:43')=Time`
to_datetime	Selects the specified ISO format datetime (YYYY-MM-DDTHH:MM:SS).	Integrate date and time information from string formats into datetime values for temporal analysis.	`to_datetime('1973-06-05T09:04:43')=DateTime`

Select Preview to review the results. If results meet your expectations, select Save to preserve the conditions. Output is then generated based on your selected transformation step and conditions.

AI Assist

Use AI Assist to create pass conditions when setting up a condition for a Data Quality transformation step.

Configure any transformation step.
Go to Condition and select the AI Assist icon.
In the dialog, enter the rule in simple text. Example: Date Created and Date Modified should not differ by more than 3650 days.
Select Generate to produce the pass condition.
Review the output and apply it to the pipeline condition.

Add multiple inputs to quality pipeline

Add multiple inputs while building your Data Quality pipeline. A simple input uses data directly from a database connection. A complex input requires additional steps to process data before merging. Merge inputs as union or joins. An output step is added for each input with runtime configurations.

For example, an e-commerce platform collects customer data from online registration, purchase histories, and customer service interactions. Multiple inputs bring these sources together to build complete customer profiles.

Warning: You cannot add multiple connections from different connection types. For example, you cannot merge Databricks tables with Snowflake running on Snowflake connection.

Create a Quality pipeline.
Select Add Input on the top-right.
Select the dataset you want to use for the pipeline.
Select Generate Sample or upload sample data.
Enter a value for Number of rows to include in the sample (1 to 2000 rows; default is 100). The sample retrieves records sequentially starting with the first record.
Select the fields you want to include in your sample and choose Generate Sample.
Choose Select Dataset.
The selected dataset is added to the pipeline.

Continue performing transformations on the data. If you have multiple datasets, combine them by appending or joining their outputs. Appending adds rows from one dataset to another. Joining merges datasets based on common fields. This consolidates all relevant data into a single output dataset.