Custom coding

Data Integrity Suite

Product
Spatial_Analytics
Data_Integration
Data_Enrichment
Data_Governance
Precisely_Data_Integrity_Suite
geo_addressing_1
Data_Observability
Data_Quality
dis_core_foundation
Services
Spatial Analytics
Data Integration
Data Enrichment
Data Governance
Geo Addressing
Data Observability
Data Quality
Core Foundation
ft:title
Data Integrity Suite
ft:locale
en-US
PublicationType
pt_product_guide
copyrightfirst
2000
copyrightlast
2025

The Custom Coding Step in the Data Quality Suite enables you to address advanced use cases or requirements that cannot be fulfilled using the existing Quality pipeline steps. This feature is particularly beneficial for users who prefer coding or scripting and need to implement complex logic such as conditional statements, loops, or integrations with scripts from other vendors or solutions.

Note: The coding language supported by the custom coding step is JavaScript.
Key Features
  • Custom Coding Capability: Solve use cases beyond the scope of pre-defined Quality pipeline steps.
  • Flexibility with Existing Scripts: Leverage custom scripts or existing code from other solutions.
  • Enhanced Editing Environment: Provides a dedicated code block with auto-complete (IntelliSense) and error validation.

AI Capabilities

The Custom Coding step in the Quality pipeline now supports AI-powered Natural Language Processing (NLP) capabilities. This allows users to describe logic in plain language, which the system automatically converts into equivalent JavaScript code.

Key Benefits

AI capabilities are useful when users need to define logic in the Custom Coding step but may not have scripting expertise or be familiar with the proprietary syntax. This feature offers the following benefits:
  • Reduces time spent translating business requirements into code
  • Enables non-technical users to contribute logic using natural language
  • Minimizes errors by automating syntax and logic generation
Example Use Cases
Table 1.
Scenario Natural Language Input Generated JavaScript
Data Validation Check if ID is greater than 18 and mark as valid
var id = input.get('ID'); 
  try { 
    // Explicitly convert to BigInt to handle integer input 
    var numId = BigInt(id); 
    // Check if ID is greater than 18 
    var isValid = numId > 18n; 
    // Set the output variable 
    output.set('IsValid', isValid); 
} catch (error) { 
    // Handle any conversion or processing errors 
    output.set('IsValid', false); 
} 
String Normalization Trim and convert full name to lowercase.
var fullName = input.get('FullName'); 
// Check if input is null or undefined 
if (fullName === null || fullName === undefined) { 
    output.set('FullName', null); 
} else { 
    // Trim whitespace and convert to lowercase 
    var processedFullName = fullName.trim().toLowerCase(); 
    // Set the processed fullname as output 
    output.set('FullName', processedFullName); 
} 
Date Transformation Convert all dates in birth_date column YYYY-MM-DD format
var inputDate = input.get('DateOfBirth'); 
try { 
    // Check if input date is null 
    if (inputDate === null) { 
        output.set('New_DoB', null); 
    } else { 
        // Use Java's DateTimeFormatter to format the date 
        var formattedDate = java.time.LocalDate.from(inputDate); 
        output.set('New_DoB', formattedDate); 
    } 
} catch (error) { 
    // Handle any formatting errors 
    output.set('New_DoB', null); 
} 

Setup Instructions

To configure AI capabilities for your workspace for custom coding operator ensure that you've enabled the feature via AI Manager, on detailed steps on how to configure it for your workspace refer to Features under AI documentation.

How to configure the custom coding step

Step Name: Defines the name for the step. Provide a meaningful name so that others editing the pipeline can easily identify its purpose.

Input Fields and Output Fields: The inputs and output field types are mapped to JavaScript types:

Table 2.
Field Type JavaScript Type Description Example
String String Represents text values. if (input.get('name') === 'John')
Boolean Boolean Represents true/false values. if (input.get('isActive') === true)
Integer BigInt Used for large whole numbers. Requires n suffix for literals.
Note: When working with BigInt, use the n suffix for number literals.
if (input.get('intField') > 100n)
Float Number Represents floating-point numbers. let price = input.get('price') + 2.5;
Double Number Similar to Float but with double precision. let total = input.get('total') * 1.1;
Decimal BigDecimal

High-precision decimal values. Uses Java’s BigDecimal class.

if (input.get('decimalField').equals(new BigDecimal('100')))
Datetime LocalDateTime Represents date and time without a timezone. let timestamp = input.get('timestamp');
Date LocalDate Represents a calendar date (year, month, day). let today = input.get('dateField');
Time LocalTime Represents a time of day (hours, minutes, seconds). let currentTime = input.get('timeField');
Note:
  • In the case of DateTime, Date, and Time datatypes, the Java Time API should be used instead of JavaScript Date. Example: var currentDate = LocalDate.now() // assigns today’s date to variable currentDate, LocalDate is a class present in Java Time api
  • In case you want to use classes that are not whitelisted and still wish to use them, then you can do so by writing the complete classpath. Example: java.time.temporal.ChronoUnit.YEARS.between(startDate, endDate); // temporal package is present inside java time api and can be used in this way if package is not whitelisted
Referenced Input Fields: Select the input fields from the dataset that your script will reference. Fields added here will be available for IntelliSense. Follow these steps to configure referenced input fields:
  1. Select multiple fields at once if needed.
  2. Click on an input field name in the list to insert a "get" snippet with the correct syntax into the code.
Warning:
  • The input object is for read/input only, i.e., input.set(‘field’, value) is not allowed and will throw an error.
  • If an input field is removed from the list but still referenced in the code, validation errors will occur.
Output Fields: Declare any new fields created or existing fields transformed in the script.
Note: Always set the output field values, otherwise their values will be null. The output.putAll(input) method can be used to initialize the output field values from the input values at the beginning of the script.
Follow these steps to configure the output fields:
  1. Field Name: Assign a unique name for new fields or select an input field to overwrite it.
  2. Datatype: Specify the data type for the field.
  3. Click the list item to insert a "Put" script snippet at the cursor point in the code.
Warning: The script can only access the input fields and output fields as defined, accessing other fields not defined as input or output will cause a runtime error.
Note: If the script throws an error at runtime, then none of the output values will be produced, and the input row will not be changed.
Code: A dedicated block where you can write your custom code.
  • The script is executed against each row of input
  • Full-screen mode is available for a better coding experience.
  • Syntax errors are highlighted in real time within the editor.

Example: Creating a user-defined isPrime() function to find prime numbers from an existing column.

function isPrime(num) {  
    // Handle numbers less than 2  
    if (num <= 1) {  
        return false;     
    }  
   // Check divisibility from 2 to the square root of num  
    for (let i = 2; i <= Math.sqrt(num); i++) {  
        if (num % i === 0) {  
            return false; // num is divisible by i, so it's not prime  
        }  
    }  
    return true; // num is prime  
    }  
output.set('IsPrime', isPrime(Number(input.get('CustomerId')))) 
Tip: If necessary, use the try-catch syntax in the script to explicitly handle script error conditions, e.g.
try { 
    //script code 
    } catch (error) {   
    output.set(‘field’, -1); 
    } 

AI Assist: It is an NLP-based tool that helps you get started with AI capabilities in the Custom Coding step. It allows you to enter natural language instructions, which are then converted into JavaScript code snippets. Follow the steps given below to use AI Assist for the code:

Note:
  • Before using AI Assist, ensure that the AI feature is enabled in your workspace. For more information, refer to Features under AI documentation.
  • You must declare both Input and Output fields before using AI Assist. If these fields are not declared, the AI Assist dialog box will be disabled.
  1. In the dialog box, enter instructions in natural language for the AI to generate the required JavaScript code.
  2. Click Generate to create the code snippet.
  3. You can choose to:
    • Click Apply to insert the generated code directly into the pipeline.
      Note: If you click Apply and the code editor already contains existing code, a confirmation prompt will appear. You must confirm before the existing code is replaced with the AI-generated code.
    • Copy the code for later use.

Additional Features

  • Data Preview: Use the data preview feature to validate that your script works as intended.
  • Issues Tab: Syntax errors are highlighted in real time within the Monaco editor. If a script has errors but you still want to preview it, navigate to the Issues Tab to review and resolve the errors.

Best Practices

  • Always test the script using the data preview functionality before finalizing.
  • Ensure referenced fields are correctly declared to avoid validation errors.
  • Use meaningful and consistent naming conventions for steps, input fields, and output fields.