Data Pipeline Monitoring, File Downloading and Logging

To use this feature, you must have the System Administrator access role.

Please reach out to your customer success and support teams for further assistance.

The File Delivery Status page provides access to original and transformed files, as well as information and error logs to help troubleshoot file delivery, processing and loading issues.

File Status

1. Navigate to Admin → Data Flow Automation

2. The system will navigate to the landing page of Data Flow Automation, which is the Data Services page.

3. From Data Services page, navigate to File Delivery Status.

4. 

   The File Delivery Status grid shows the following columns:

   1. Filename - the original filename of the file

   2. Delivered On - the date and time the file was shared by the client on Amazon S3

   3. Delivered From - the name of the data source used in the data pipeline

   4. Delivered Via - the name of the data service type used in the data pipeline’s data source, e.g. Amazon S3

   5. Delivery Status -

      1. if the file has been downloaded successfully from the data source

      2. if downloading the file from the data source encountered an error

   6. Processing Status

      1. if the file requires transformation and was successfully transformed, and ready for file loading

      2. if the file does not require transformation but was successfully pushed for file loading

      3. ⏳ if the file has been transmitted to IBM App Connect but the transformed copy is not yet available

      4. if transmission or transformation of file encountered error

   7. Loading Status

      1. if the file has loaded successfully

      2. if the file encountered an error while loading
         

5. On the File Delivery Status page, click on the preview panel icon to see details of the pipeline which processed each file. This is useful to verify the filename pattern which the file matched on.

   

File Download

Downloading files is only available where files have been validated to contain data for your organization. If the download options are greyed out, please reach out to your support team for assistance.

Downloading a copy of the original file

1. On the File Delivery Status page, choose a file

2. Right-click and select Download Original File from the action menu

3. The copy of the file will start downloading on browser
   

Downloading a copy of the transformed file

1. On the File Delivery Status page, choose a file that has been transformed

2. Right-click and select Download Transformed Copy from the action menu

3. The copy of the transformed file will start downloading on the browser

Data Pipeline Logs

The File Delivery Logs page shows list of events that occurred when processing the file.

1. From the File Delivery Status page, select a file and right-click and choose Delivery Logs from the action menu.

2. The File Delivery Logs page shows the following information:

   1. Log Type

   2. Event On - date and time the process happened

   3. Event - a description of the event

   4. Pipeline Name - pipeline that were used to process the file

   5. Service Name

   6. Service Type

   7. Filename Pattern

   8. Target System

   9. Filename

3. Additional log details, such as source and destination of files can be seen on the File Details preview panel on the right

File Loading Logs

The Files Loading Logs page shows the list of events that occurred when loading the file.

1. From the File Delivery Status page, select a file and right-click and choose Loading Logs from the action menu.

2. The File Loading Logs page shows the following information:

   1. Filename

   2. Uploaded On

   3. Processed By

   4. File Status

   5. Compilation Status

   6. Relates To

   7. Records In

   8. Records Out

   9. Updated On

   10. Result

3. Parse Errors and Load Errors can be viewed by selecting these options from the actions menu for each log entry. For more information on this refer to Check File Loading Status