Upload sample data to configure and preview pipelines - Precisely Data Integrity Suite

Data Integrity Suite
Product
Spatial_Analytics
Data_Integration
Data_Enrichment
Data_Governance
Precisely_Data_Integrity_Suite
geo_addressing_1
Data_Observability
Data_Quality
dis_core_foundation
Services
Spatial Analytics
Data Integration
Data Enrichment
Data Governance
Geo Addressing
Data Observability
Data Quality
Core Foundation
ft:title
Data Integrity Suite
ft:locale
en-US
PublicationType
pt_product_guide
copyrightfirst
2000
copyrightlast
2026

Upload or select sample data to define the pipeline input structure, validate data compatibility, and preview how the pipeline processes your data before running it on the full dataset.

Preview: This feature is currently in Preview for select workspaces and might be subject to change before general availability.

Sample data is required when you set up or modify a pipeline. It defines the pipeline input structure and validates whether your data matches expected fields and formats. It also lets you preview how the pipeline processes data before full execution. You can provide sample data in two ways: upload a new CSV file directly or select an existing dataset from your data catalog. The system automatically validates sample fields against pipeline input requirements and reports mismatches clearly.

Pipeline data sample profiling: Profiling analyzes sample data in each field and identifies anomalies that need cleanup before production data flows through your pipeline. The system validates each field's contents against its base and semantic type. For example, alphabetic entries in a Telephone Number field are flagged as problems. In an Email Address field, missing domain names and consecutive dots are flagged as problems.

  1. Go to Quality > Pipelines and click Create Pipeline.
  2. Choose a sample source.
    • Upload a new sample file: Click Upload Sample and then choose a file, or drag and drop a CSV or TXT file into the upload area.
    • Select a sample from a cataloged dataset: Click Browse Dataset to view available datasets in your catalog.
  3. Upload a new CSV or TXT file to the system to use as sample data for your pipeline.
    1. In the Upload Sample dialog, click choose a file or drag and drop a CSV or TXT file into the upload area.
    2. Select your file from your local system. The file size appears after selection.
    3. In the Give sample a unique name field, enter a descriptive name for the sample.
    4. Click Upload & Preview Sample.
    5. Wait for the system to validate the uploaded file against the pipeline input requirements.

    If validation passes, the sample is configured and ready for preview. The system displays:

    • Sample Name: The unique identifier you provided (editable).
    • Sample Source: Indicates the sample is from a standalone upload.
    • Record Count: The number of records in the sample.
    • Input Fields: A list of all fields with their data types.
    • Stored in: Storage location and retention period (for example, "Precisely Cloud (Data sample will be purged in 89 days)").

    If validation fails, an error message appears. Review the error and act accordingly.

    Click Create Pipeline to create a pipeline.

  4. Browse and select an existing dataset from the catalog to use as sample data for your pipeline.
    1. In the Select or Add a Data Sample dialog, click Browse Dataset.
    2. The Sample from cataloged dataset view opens, displaying available datasets.
    3. Optionally, use the Filter options to narrow results by datasource type or storage location.
    4. Optionally, use the Search field to find a specific dataset by name.
    5. Review the dataset details displayed in the right panel.
      • Dataset name and description
      • Storage location
      • File format and properties (encoding, separators, headers)
      • Partitioning configuration
      • Number of records
    6. Click the radio button next to the dataset you want to use as a sample.
    7. Click Next.
    8. Wait for the system to validate the dataset fields against the pipeline input requirements.

    If validation passes, the sample is configured and ready for preview. The system displays the sample details, including name, source, record count, input fields, and storage information.

    If validation fails, an error message appears. Select a different dataset with matching fields or modify your pipeline input configuration.

  5. Change an existing sample.
    1. In the pipeline input panel, locate the Sample section.
    2. Click Change Sample.
  6. Preview sample data in the pipeline editor.
    After you configure a sample, preview the data in the pipeline editor to see how it flows through each transformation step.
    1. After sample validation passes, the pipeline editor displays the sample data in table format in the lower panel.
    2. Each column represents a data field, and each row represents a sample record.
    3. The top row shows the field name, semantic type, and field type for each data field.
    4. Select a transformation step in the upper panel to highlight the columns affected by that step.
    5. Optionally, select the Transformation Preview checkbox to display sample values as they appear before and after the transformation.
  7. After you provide a sample, the system automatically validates the sample fields against the pipeline input requirements.
    1. The system checks that all sample fields match the pipeline input fields in name and data type.
    2. If validation passes, the system confirms compatibility and lets you proceed with preview and configuration.
    3. If validation fails, the system displays an error message indicating which fields do not match.
    4. Resolve field mismatches by using one of the following options.
    • Upload or select a different sample with matching fields.
    • Modify your pipeline input configuration to match your sample data.
  8. Click the View pipeline input fields link in the error message to review the expected field structure and ensure your sample aligns with it.

Consider the following when selecting or uploading sample data:

  • File format: The upload feature currently accepts CSV and TXT files only. If your sample data is in a different format (Excel, JSON, Parquet), convert it to CSV before uploading or select a pre-cataloged dataset in the desired format.

  • Sample size: Use a representative sample that includes at least 50-100 records to provide meaningful preview data. Include all field types present in your full dataset and edge cases or quality issues you want to test.

  • Field matching: Sample fields must match the pipeline input fields exactly. If you receive a field mismatch error, upload or select a different sample with matching fields, or modify the pipeline input configuration.

  • Data retention: Uploaded samples are stored temporarily in Precisely Cloud and are automatically purged after a retention period (typically 89 days). If you need longer retention, consider cataloging the sample as a permanent dataset.

  • Cataloged dataset filtering: When browsing cataloged datasets, use the Filter options to narrow results by datasource type or storage location. This helps you find the right dataset quickly, especially in large catalogs.

  • Sample vs. Full data: The sample is used only for preview and configuration. When you run the pipeline, it processes the actual datasource, not the sample. Click Run to execute the pipeline on your full dataset.