...
Type | Description | Available for | ||||||
|---|---|---|---|---|---|---|---|---|
Purge Cache | Clears the cache. The cache applies to queries on Data Sources and Datamarts, from the UI and from logics. There is no expiration, but it is limited in size (default is 1000 entries / partition). It is a LRU (Last Recently Used) cache. It gets invalidated after a data change (load, flush, refresh). Cache purging is a recovery mechanism. |
| ||||||
Truncate | Deletes (all/filtered) rows in the target. Note: When a Data Source is deployed, the Truncate Data Load of the linked Data Feed is updated with a filter to include only rows previously successfully flushed to Data Source and it is scheduled to run once a week. This applies only if there is no other filter or schedule already defined.
|
| ||||||
Upload | Loads an XLS/XLSX sheet or a CSV/TXT file. Supports scientific notations of numbers: 1e02, 1E02, 1e+02, 1E+02, 6e-04, 6E-04. |
| ||||||
Mass edit | Allows you to apply mass changes to editable attributes. See Common Table Actions for details.
|
| ||||||
Flush | Copies data from the Data Feed into the Data Source. It can also convert values from string to proper data types set in the Data Source. It can copy everything or just new data (i.e., incremental Data Load). |
| ||||||
Refresh | Copies data from Data Sources (configured in the Datamart fields) into the target Datamart. It can copy everything or just new/modified data (i.e., incremental Data Load). If a source filter is set on the Source tab, only the filtered rows are loaded to the Datamart. Rows that do not meet the filter’s condition and are present in the Datamart are automatically deleted from the Datamart when the Refresh job is run.
Note: Since Godfather 8.1, rows updated during Refresh behave differently: their calculated fields are cleared to NULL instead of being persisted. For details see the release notes. |
| ||||||
Calculation | Applies a logic (defined in Configuration) to create new rows, or change/update values in existing rows in the target Data Source or Datamart. The calculation can take data from anywhere, e.g., Master Data tables. Example usage:
|
| ||||||
| Generates rows of the built-in Data Source "cal" and you get a Gregorian calendar with US locale. (If you need any other business calendar, just upload the data into the "cal" Data Source from a file or via integration and do not use this Data Load). | |||||||
Customer | Special out-of-the-box Data Load which copies data from the Master Data table "Customer" into the Data Source "Customer". | |||||||
Product | Special out-of-the-box Data Load which copies data from the Master Data table "Product" into the Data Source "Product". | |||||||
Simulation | Applies a logic to the data as defined by the simulation for which the Data Load was created. |
| ||||||
Internal Copy | Copies data from a source into the Data Source table. The source here can be:
The easiest way to create this type of Data Load is to create a new Data Source from Template and deploy it; this automatically creates the Data Load and pre-fills the columns.
|
| ||||||
Index Maintenance | This task can be run to repair indexes associated with the target Data Source or Datamart, typically after backend DB migration. The task should be run only in these special circumstances, not on a regular or scheduled basis. We also strongly recommend consulting Pricefx support before you run this task. |
| ||||||
Distributed Calculation | Allows you to split a large set of data into batches and process them independently. See Distributed Calculation in Analytics for details. |
| ||||||
Publishing | Publishes data after Refresh which makes new data accessible by queries. It is system generated and cannot be created manually. |
|
If you want to run a non-incremental refresh but avoid the costly merging of almost the same data, you can truncate the DM first – set the 
The incremental mode in Internal Copy tasks is not exactly the same as in the Refresh or Calculation type. Here, incremental means the Data Source will not be truncated before the copy, i.e., it will keep old records instead of being a true copy....
Options – Allows you to make Data Load settings:
Target Datamart – Specifies the target of the data coming from the Data Load operation.
For the Flush operation, the target is one of the Data Sources.
For Refresh, the target is one of the Datamarts.
For Calculation, the target is either Data Source or Datamart which you would like to update or enrich with columns.
+ Add Filter – You can filter data on the Target tab using the Advanced Filter.
Data Source – Specifies the source of data.
For the Flush operation, it will be a Data Feed.
For Refresh it is not specified because there can be more sources for one target Datamart. Those source Data Sources are specified in the columns definition of the Datamart.
+ Add Filter – You can filter data on the Source tab using the Advanced Filter.
Anchor AllowBatching AllowBatching Allow batching – Large amounts of data (more than 5 million rows) can be processed in batches. By default, batch processing is enabled for Flush operations and disabled for Calculation. You can override this setting here. (The batch size is an instance parameter and the default value is 2 million rows. There must not be any dependencies between rows belonging to different batches.)
Since version 13.3 - Rampur, when the Allow Batching option is set on a Calculation Data Load (Anchor AllowBatching AllowBatching DL.allowBatching) that is configured with feeder logic, thePACalculationTaskwill accumulate the target rows produced by the logic across feeder items until a batch is filled. Only then are the rows uploaded to the target. Previously, the rows were uploaded after the evaluation of each feeder item.Validation Logic – You can select a validation logic that will validate the target data after refresh. The following rules apply:
The target Datamart name is available in the validation logic through the "dmName" binding variable:
Code Block language groovy def dmName = api.getBinding("dmName")Use the name to query for the Datamart's data and apply custom validation rules.
Raise a warning with a custom validation message when a validation rule is not satisfied:
Code Block language groovy api.addWarning("Missing value in field1")When api.addWarning() is invoked, the data validation is considered as failed and the Data Load's status is set to Error. Note that the validation logic result does not affect the Data Load process itself as it is run after the Data Load is completed.
Validation messages passed from validation logic are present among the Data Load's calculation messages and can be viewed in the UI.
Overview – Summarizes the basic information on the Data Load.
Schedule – Here you can also schedule the Data Load manually (described below). The Job/Task Tracking section at the bottom shows the status for each task of the Data Load. When you click the
icon, the Job Details screen appears.
Target – Displays the complete target data set. The available options are the same as for Data Sources. View preferences are not available in this table. Instead, quick and advanced filter and sorting are saved automatically when you save the Data Load.
Source – Displays the complete source data set. The available options are the same as for Data Sources. View preferences are not available in this table. Instead, quick and advanced filter and sorting are saved automatically when you save the Data Load.
Calculation – Specifies the logic to be executed. Such logic is set up in Administration > Logics > Analytics. It can manipulate the data coming from the Source in many ways, e.g., filter the incoming rows from Source, create new lines for Target, modify/enrich/transform the data being copied from the Source into the Target. Click Default Formula to open the currently active Analytics default logic in an editor.
If you leave the Target Date field empty, the calculation will use "today" as the target date.Compare, Staging, Published – In Publishing type, these tabs display unpublished, refreshed data (Staging), published data and their comparison.
| Info |
|---|
When you add a new Data Load, a warning icon |
Schedule a Data Load
To schedule a Data Load:
...