File loading through connectors is one of the main methods to get Account data into the platform. In certain scenarios, system configuration work can also be performed in bulk via file loading. This page documents the best practice for checking file loading status, and troubleshooting steps for Accounts and Setup data. Tools presented on this page are available to System Administrator users of Envizi only.

Check File Loading Status

Once a file has been sent to Envizi for loading (via email, web upload or FTP), you should be able to find it in the Files Processed - Accounts & Setup page. Access this page from main menu Manage and then under the Data Files section.


By default files loaded in the Last 24 hours are displayed on the page. To view more files by clicking either Last 7 days, Last 30 days, Last 90 days or All Items button on the page.

Basic Checking

If you have limited time available, these are the recommended steps to check the file loading status at a minimum level.

Step 1: Look for the file in the page

Once the file has been received and picked up by the server to process, you should be able to find it here.

If the file cannot be found, it means it was not picked up by your connector. This may occur for one of the following reasons:

  • File Name does not conform to the connector design document - e.g., looking for extra space, spelling check or additional 's' in the file name

  • For Excel file, Tab name does not conform to the connector design document

  • Column headings, order of columns and number of columns in the file do not conform to the connector design document - any of these could prevent the file from being picked up by your connector

  • Your file is still in the queue to be processed  - check out this page later

Step 2: Check File Status

When you find the file on the page, look for the column File Status.

A Loaded status means the connector has completed processing the file and its data has been loaded into the system.

An <*Error*> status indicates the file loading has failed and it needs to be checked and probably re-submitted. Some common reasons for file loading failures are but not limited to:

  • File contains data within your organization's data lock down period

  • None of the records in the file are loaded due to various reasons, e.g., invalid account style

  • Invalid data format present in the file, e.g., garbage text in a date column

You can find the error details in the Show Parse Errors and Show Load Errors pages which are covered later in this document.

Info

In some cases a file can still be loaded (with a 'Loaded' file status) but with some of the records being flagged as errors. In most cases these errors are considered as minor (such as checking for account's open/close dates) and does not prevent the rest of the records in the file to be loaded.

In other cases the entire file is failed (with an 'Errror' file status) due to some errors that are considered as 'critical' which prevent any other records in the file to be loaded. These 'critical' errors would need to be examined and corrected before the file can be re-submitted for loading.

The types of 'critical' errors that fail the entire file load is connector dependent - each connector is designed differently and has different loading behaviour. 

Generally if a file has failed to load completely, a comprehensive checking should be carried out in several areas in the following order: Load Errors → File Details  → Parse Errors. The details of each area are covered later in this document.


Step 3: Check Records In and Records Out

Records In - is the number of rows in the file that are considered to be valid records and picked up by the connector for processing. This excludes rows that have been deliberately designed to be skipped, rows that does not conform to the format defined in connector design document, the column header row or any empty rows in the file.

Records Out - is the number of new records that are actually getting loaded and written into the system. For a straight forward data loading connector, generally the Records In should be equal to Records Out. For example, if you have 10 electricity bills in a file to be loaded, you should see Records In equals to Records Out, which is 10.

However in many cases, Records Out'does not necessarily equal to Records In, and is sometimes less than Records In.

This may be due to a variety of reasons, depending on your connector. For example:

  • It is a multi-read connector, which means several rows of records are combined into one record and then loaded

  • The connector is unable to find the Location for the Account specified in the file - this needs to be investigated

  • The connector is unable to find the Account for the record specified in the file - this needs to be investigated

  • For system configuration (setup connectors), it is an UPDATE operation rather than a CREATE operation, which means 0 new record is created and 'Records Out' will show up as 0

Step 4: Check File Details

You can check for additional details about the file loading on the right hand side preview panel, such as:

  • Total Rows

  • Skipped Rows

  • Load Errors

  • Loaded Rows

  • Result

For setup connectors, the Result field will also show additional diagnostic information in regards to the configuration. For example, number of new accounts created will be shown here if it is a Setup Accounts connector.

The File Details is only available to files loaded in the last 48 hours.

Info

During file loading, each row in the file is first scanned by the connector and allocated into several categories - it could be a valid data row to be loaded, an empty row, a column header row, a row meant to be skipped or a row in a format that cannot be recognized by the connector. The connector will only try to load rows that are identified as valid data rows (Records In) into the system but not rows in other categories. During the loading process, if it encounters an error the row will be flagged as 'Load Errors' and not loaded.

The relationship can be represented by the following:

Total Rows = Process Errors (column header, empty rows, rows not recognized, etc.) + Skipped Rows + Rows In (Records In) + Load Errors + Loaded Rows (Records Out)

A general rule of thumb is that if there are more than 2 Process Errors, or there are any number of Load Errors, then it may warrant a further investigation to ensure the file loading is complete and as desired.

Comprehensive Checking

By completing the basic check you should have a good understanding of the file loading status. However you may still have the risk of an incomplete file load without completing a comprehensive check. If time allows or you have noticed there are issues in the file loading, you should perform the following steps for a comprehensive check.

Step 5: Check Compilation Status

For account data loading, Account stats need to be generated after the file loading. The generation of stats are being processed separately from the file loading process and thus may have a delay from the file being loaded physically. The 'Compilation Status' column allows you to check the status of the stat compilation jobs.

The status will go through the following change for a successful file loading:

  • n/a (when the file is still being physically loaded) → 0%; Estimated to start in 5 minute(s) → 65% → 100%

If there is error encountered during compilation stage:

  • n/a (when the file is still being physically loaded) → 0%; Estimated to start in 5 minute(s) → 65%; 53 Success, 1 Fail → 100%; 87 Success 1 Fail

If there are other files ahead of the current file which requires compilation, an estimated compilation start time will be provided for the file. The estimation is based on the assumption that one account would typically require one second of time to compile its entire stats data (including accruals). 

Visibility of compilation errors are also available in the Preview Panel on the right. Although the likelihood of getting errors from stats compilation process is very rare, but it does happen sometimes, especially with corrupted or bad data entered in the record.

Step 6: Check Show Parse Errors

Files are first scanned by the connector before they are loaded, and if there are any issues found during the 'scanning' process, they are captured in the Parse Errors page. You can access this page by right clicking on the file and then select Show Parse Errors.

Typical errors reported here include, but are not limited to:

  • Record rows that do not match any of the row specifications as defined in the connector design document

  • Invalid date format for date fields

  • Empty rows

  • Skipped rows as defined in the connector design document

  • Column header row

  • A summary line of total number of rows in the file

Info

For Parse Errors and Load Errors, a maximum number of 10 items of the same error type will be shown on the page. The number of total errors of the same type will be displayed in each message as well, e.g. ,  8 of 24 errors of this type.


Step 7: Check Show Load Errors

Once files have been scanned and valid records identified, the file will be going through the 'loading' process to be loaded into the system. Any issues identified during the 'loading' stage are recorded in the Load Errors page. You can access this page by right clicking on the file and then select Show Load Errors.

Typical errors reported here include, but are not limited to:

  • Trying to load record into a closed account

  • Trying to load record before locked down date

  • Non numeric values for consumption or cost fields

  • Unable to find the Location for the record to be loaded

  • Unable to find the Account for the record to be loaded

Correct the reported issues in the file if any and re-submit the file for loading if required.

Info

Different connectors may have different file loading behaviours. If a file was loaded but with mismatching numbers of Records In and Records Out, it may still be a valid scenario for this connector provided there is no unusual Parse Errors, Load Errors and issues appearing in Files Details - Result field.