Managing Data Pipelines

To use this feature, you must have the System Administrator access role.

Please reach out to your customer success and support teams for further assistance.

A Data Pipeline defines how a data file will be processed by Envizi.

There are two types of Data Pipelines - File Loading and Data Export.

File Loading

A File Loading Data Pipeline has the following configuration:

Type - File Loading
Target System - determines which data loader program to use to load the file into Envizi
Filename pattern - determines which file to download from data source
Data Source - determines which data source to connect to download files
Data Transformer - determines which data transformation service, e.g. IBM IS App Connect, to use to transform the original data file to “standard” file specification. Alternatively, files may be passed through without transformation.
Additional Parameters - advanced feature used to specify additional file processing parameters.

Currently all files loaded through Data Flow Automation will require a pipeline to be processed.

A Data Export Data Pipeline has the following configuration:

Type - Data Export
Dataset - lists available datasets for export (currently limited to monthly dataset)
Destination - determines which data service to export dataset files to