Data Loads
A Data Load represents a task/process for data manipulation between Analytics objects, such as uploading data from Data Feed to Data Source, deleting rows from Datamart or calculation of new values of the fields.
Most Data Loads are created automatically (when you deploy a Data Source or Datamart) but you can also create them manually (e.g., a calculation Data Load to manipulate data).
Data Loads provide the following actions:
Type | Description | Available for |
|---|---|---|
Purge Cache | Clears the cache. The cache applies to queries on Data Sources and Datamarts, from the UI and from logics. There is no expiration, but it is limited in size (default is 1000 entries / partition). It is a LRU (Last Recently Used) cache. It gets invalidated after a data change (load, flush, refresh). Cache purging is a recovery mechanism. |
|
Truncate | Deletes (all/filtered) rows in the target. Note: When a Data Source is deployed, the Truncate Data Load of the linked Data Feed is updated with a filter to include only rows previously successfully flushed to Data Source and it is scheduled to run once a week. This applies only if there is no other filter or schedule already defined. Incremental mode is no longer available for Truncate jobs. For older jobs (created before upgrade to Collins 5.0 release) where this option was enabled, it will stay enabled. If you disable the Incremental option the check-box will become non-editable and you will not be able to enable the option again. For Data Loads saved with the Incremental option off, the check-box is completely hidden. |
|
Upload | Loads an XLS/XLSX sheet or a CSV/TXT file. Supports scientific notations of numbers: 1e02, 1E02, 1e+02, 1E+02, 6e-04, 6E-04. |
|
Mass edit | Allows you to apply mass changes to editable attributes. See Common Table Actions for details. Whenever you perform a Mass Edit operation in Datamart, remember to run a default Publish Data Load to make the edited data available in your queries. For more information about Data Flow see Data Manager. |
|
Flush | Copies data from the Data Feed into the Data Source. It can also convert values from string to proper data types set in the Data Source. It can copy everything or just new data (i.e., incremental Data Load). The Flush Data Loads can be executed in the following types of the Load Mode:
For more information see Load Mode. |
|
Refresh | Copies data from Data Sources (configured in the Datamart fields) into the target Datamart. It can copy everything or just new/modified data (i.e., incremental Data Load). If a source filter is set on the Source tab, only the filtered rows are loaded to the Datamart. Rows that do not meet the filter’s condition and are present in the Datamart are automatically deleted from the Datamart when the Refresh job is run. If you want to run a non-incremental refresh but avoid the costly merging of almost the same data, you can truncate the DM first – set the advanced configuration option 'truncateDatamartOnNonIncRefresh' to true. Since Godfather 8.1, rows updated during Refresh behave differently: their calculated fields are cleared to NULL instead of being persisted. Custom Refresh Data Loads are not supported since Rampur 13.0. To address this, we have implemented a warning message. If the response of a Data Load detail contains an explicit warning with Warning Message 
|
|
Calculation | Applies a logic (defined in Configuration) to create new rows, or change/update values in existing rows in the target Data Source or Datamart. The calculation can take data from anywhere, e.g., Master Data tables. Example usage:
Note that the Calculation Data Loads may show a Ready status while displaying a progress percentage lower than 100%. This occurs because the progress reflects the fraction of target rows actually updated by the calculation logic, which may not cover every row, depending on how the logic is constructed. This is normal and expected. Success or failure should be judged based on the Data Load Status and the Calculation Messages, not just by the progress bar. This is the case only for Non-Distributed Calculations. Distributed Calculations always reach 100% at the end as they have batch-based progress. Example Calculation Message Target rows before Calculation: 2,119,802
Duration (ms) Count Execution element
---------------------------------------------------------
58139.25 21% ██░░░░░░░░ 2042855 │ │ ├── datamartRowSet.addOrUpdateRow |
|
Calendar | Generates rows of the built-in Data Source "cal" and you get a Gregorian calendar with US locale. (If you need any other business calendar, just upload the data into the "cal" Data Source from a file or via integration and do not use this Data Load). | |
Customers | Special out-of-the-box Data Load which copies data from the Master Data table "Customer" into the Data Source "Customer". |
|
Products | Special out-of-the-box Data Load which copies data from the Master Data table "Product" into the Data Source "Product". |
|
Simulation | Applies a logic to the data as defined by the simulation for which the Data Load was created. |
|
Sellers | Special out-of-the-box Data Load which copies data from the Master Data table "Sellers" into the Data Source "Seller" |
|
Internal Copy | Copies data from a source into the Data Source table. The source here can be:
The easiest way to create this type of Data Load is to create a new Data Source from Template and deploy it; this automatically creates the Data Load and pre-fills the columns. The incremental mode in Internal Copy tasks is not exactly the same as in the Refresh or Calculation type. Here, incremental means the Data Source will not be truncated before the copy, i.e., it will keep old records instead of being a true copy. |
|
Index Maintenance | This task can be run to repair indexes associated with the target Data Source or Datamart, typically after backend DB migration. The task should be run only in these special circumstances, not on a regular or scheduled basis. We also strongly recommend consulting Pricefx support before you run this task. |
|
Distributed Calculation | Allows you to split a large set of data into batches and process them independently. See Distributed Calculation in Analytics for details. |
|
Publishing | Publishes data after Refresh which makes new data accessible by queries. It is system generated and cannot be created manually. |
|