Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: PFUN-26667

...

  • Options – Allows you to make Data Load settings:

    • Target Datamart – Specifies the target of the data coming from the Data Load operation.

      • For the Flush operation, the target is one of the Data Sources.

      • For Refresh, the target is one of the Datamarts.

      • For Calculation, the target is either Data Source or Datamart which you would like to update or enrich with columns.

      • + Add Filter – You can filter data on the Target tab using the Advanced Filter.

    • Data Source – Specifies the source of data.

      • For the Flush operation, it will be a Data Feed.

      • For Refresh it is not specified because there can be more sources for one target Datamart. Those source Data Sources are specified in the columns definition of the Datamart.

      • + Add Filter – You can filter data on the Source tab using the Advanced Filter.

    • Anchor
      AllowBatching
      AllowBatching
      Allow batching – Large amounts of data (more than 5 million rows) can be processed in batches. By default, batch processing is enabled for Flush operations and disabled for Calculation. You can override this setting here. (The batch size is an instance parameter and the default value is 2 million rows. There must not be any dependencies between rows belonging to different batches.)
      (info) Since version 13.3 - Rampur, when the Allow Batching option is set on a Calculation Data Load (DL.allowBatching) that is configured with feeder logic, the PACalculationTask will accumulate the target rows produced by the logic across feeder items until a batch is filled. Only then are the rows uploaded to the target. Previously, the rows were uploaded after the evaluation of each feeder item.

    • Validation Logic – You can select a validation logic that will validate the target data after refresh. The following rules apply:

      • The target Datamart name is available in the validation logic through the "dmName" binding variable:

        Code Block
        languagegroovy
        def dmName = api.getBinding("dmName")
      • Use the name to query for the Datamart's data and apply custom validation rules.

      • Raise a warning with a custom validation message when a validation rule is not satisfied:

        Code Block
        languagegroovy
        api.addWarning("Missing value in field1")
      • When api.addWarning() is invoked, the data validation is considered as failed and the Data Load's status is set to Error. Note that the validation logic result does not affect the Data Load process itself as it is run after the Data Load is completed.

      • Validation messages passed from validation logic are present among the Data Load's calculation messages and can be viewed in the UI.

  • Overview – Summarizes the basic information on the Data Load.

  • Schedule – Here you can also schedule the Data Load manually (described below). The Job/Task Tracking section at the bottom shows the status for each task of the Data Load.

  • Target – Displays the complete target data set. The available options are the same as for Data Sources. View preferences are not available in this table. Instead, quick and advanced filter and sorting are saved automatically when you save the Data Load.

  • Source – Displays the complete source data set. The available options are the same as for Data Sources. View preferences are not available in this table. Instead, quick and advanced filter and sorting are saved automatically when you save the Data Load.

  • Calculation – Specifies the logic to be executed. Such logic is set up in Administration > Logics > Analytics. It can manipulate the data coming from the Source in many ways, e.g., filter the incoming rows from Source, create new lines for Target, modify/enrich/transform the data being copied from the Source into the Target. Click Default Formula to open the currently active Analytics default logic in an editor.
    If you leave the Target Date field empty, the calculation will use "today" as the target date.

  • Compare, Staging, Published – In Publishing type, these tabs display unpublished, refreshed data (Staging), published data and their comparison.

...