Plasma Transactions Technical Documentation
- Ondřej Tesař
Modules:
- 1 Extractor
- 1.1 Configuration Tables
- 1.2 Data Sources
- 1.3 Data Loads
- 2 Distiller
- 2.1 Data Sources
- 2.2 Datamarts
- 2.3 Data Loads
- 3 Harvester
- 3.1 Data Sources
- 3.2 Datamarts
- 3.3 Data Loads
- 4 Steps Sequence
Data objects structure and calculations: Plasma Data Objects .
Extractor
Extractor gathers the customer's data into standardized Plasma data. Extractor is deployed on the customer's partition. It can be configured using Pricing Parameter tables to map the customer's data to Plasma data, and extract the data from the customer's partition to the Plasma Data Sources and Datamarts, via Data Loads.
Transactions extractor works by creating temporary in-memory tables for every source and finally joins them using the joining key. There is a limit of 100,000 rows for the in-memory tables, so if the transactions data exceeds this limit, then we can configure feeder to partition the data.
Configuration Tables
All the Plasma confifuration tables can be found here: Plasma Configuration Tables .
Data Sources
Plasma_TX – Contains Plasma fields which will be loaded by the Plasma_Transactions Data Load using the mapping defined in the configuration tables.
Data Loads
Plasma_Transactions – This is the Calculation type Data Load. Uses the PriceAnalyzer logic "Plasma_Data_Extractor", takes the Extractor name as an input and the feeder can be configured to partition the data. The feeder logic is "Plasma_Data_Extractor_Feeder". The target is Plasma_TX Data Source. It runs daily with an incremental load.
Distiller
Distiller calculates previous prices and aggregates the Plasma data rolled up by month into standardized metrics. Distiller is deployed on the customer's partition.
Before going to the distiller, previous prices are calculated and updated in the Plasma_TX Data Source. Then currency exchange rates are applied and loaded into Plasma_Transactions Datamart, then the monthly aggregated data is written into the Plasma_Output_Transactions Datamart. Only data which is the specified number of months older than current date will be taken into consideration whenever this aggregation happens.
Data Sources
Plasma_TX - Contains Plasma fields which will be updated by previous prices.
Plasma_Currencies - Contains Plasma specific currency exchange rates.
ccy (available by default)
Datamarts
Plasma_Transactions - Contains Plasma fields sourced from the Plasma_TX Data Source, after the Plasma_Transactions_PreviousPrice Dataload Calculation. Exchange conversion rates are applied while loading into this Datamart with Plasma_Transactions_Exchange dataload.
Plasma_Output_Transactions – Contains aggregated Plasma data rolled up by month. Fields are not sourced from any data source.
Data Loads
Plasma_Transactions_PreviousPrice – This is the Calculation type Data Load. It uses the PriceAnalyzer logic "Plasma_Transactions_PreviousPrice", runs on the source Plasma_TX Data Source and updates the same.
Plasma_Transactions_Exchange - This is the Calculation type data load. Uses the PriceAnalyzer logic “Plasma_Transactions_Exchange ”, runs on the source Plasma_TX Data Source and loads the target into Plasma_Transactions Datamart.
Plasma_Output_Creation – This is the Calculation type Data Load. It uses the PriceAnalyzer logic "Plasma_Transactions_Creation", targeted to update the Plasma_Output_Transactions Datamart.
Plasma_Output_Enrichment – This is the Calculation type Data Load. It uses the PriceAnalyzer logic "Plasma_Transactions_Enrichment", runs on the source Plasma_Output_Transactions Datamart and updates the same.
Harvester
General Harvester description: https://pricefx.atlassian.net/wiki/spaces/PSP/pages/2347532480.
Data Sources
Plasma_Transactions_Harvester – Contains Plasma standardized metrics data gathered from several customers' partitions.
Plasma_Entities – Contains demographic entity data of each customer with various validFrom dates.
Plasma_Taxonomy - Contains taxonomy for all possible industry hierarchy levels. Currently goes up to 5 levels.
Plasma_KPI - Contains the KPI data distributed to all the customers partitions by Harvester
Datamarts
Plasma_Transactions_Harvester – Contains the complete n metrics. The fields are sourced from the Harvester, Plasma_Entities and Plasma_Taxonomy Data Sources.
Plasma_Transactions_Validations - Contains all data about failed validations, with the description which validation rule catched it. The rows are added by Plasma_Transactions_Validator logic.
Data Loads
Plasma_Transactions_Harvester – This is the Calculation type Data Load. It uses the PriceAnalyzer logic "Plasma_Harvester" targeted to update the Harvester Data Source.
Plasma_Transactions_Harvester – This is the Refresh type Data Load. It refreshes the data from the Plasma_Transactions_Harvester , Plasma_Entities and Plasma_Taxonomy Data Sources into the Harvester Datamart. This automatically joins Plasma_Transactions_Harvester and Plasma_Entities data sources using Dem_Entity_Code; also joins Plasma_Entities and Plasma_Taxonomy data sources using Dem_Industry_Level_5.
Steps Sequence
Plasma transactions steps are sequenced with a combination of Plasma_Transactions_Sequencer logic, Plasma_Transactions_Sequencer_Reset CalculationFlow and Plasma_Transactions_SequenceControlTable PP table. More details: https://pricefx.atlassian.net/wiki/spaces/PSP/pages/3745054872/Plasma+Logics#Plasma_TX_Sequencer, Plasma Transaction Processor Package Configuration.
.