In Data Sources you can structure, clean up and validate data. It is a data container defined by the end-user to store specific master and transactional data, such as Customer master, transactional data (invoice line items) etc.

The Data Source detail view has three tabs:

Fields – Add new fields manually or from a data sample. Import or export Data Feed definitions in the JSON format. You can select a logic that will transform the data already present in the Data Source.
Data Entitlement – Define accessibility of data for different user groups. See How to Set Data Entitlement for details.
Data – View and edit data. By default, the table displays up to 12 columns, with the following priority: fields that have a function (e.g., key, alt-key), fields that are dimensions and then all other fields up to the limit. If you want to see more fields, add them using the Select Fields to Display option or create and save a view preference. Admins can also increase the default limit using the maxPATablesFetchLimit advanced configuration option.
The available options are:
- Purge Cache – Clears the cache. The cache applies to queries on Data Sources and Datamarts, from the UI and from logics. There is no expiration, but it is limited in size (default is 1000 entries / partition). It is a LRU (Last Recently Used) cache. It gets invalidated after a data change (load, flush, refresh). Cache purging is a recovery mechanism.
- Truncate – Deletes all/filtered rows from the Data Source.
- Upload – Loads an XLS/XLSX sheet or a CSV/TXT file. Supports scientific notations of numbers: 1e02, 1E02, 1e+02, 1E+02, 6e-04, 6E-04.
- Mass edit – Allows you to apply mass changes to editable attributes. See Common Table Actions for details.
- Status History – Shows information on the Data Source status history – a list of Data Load actions that have been executed. Clicking on the Info icon opens a pop-up window with Data Loads messages (if available).
- Export Excel, CSV – Exports the Data Source data to the XLSX or CSV format. For data tables containing a large amount of data, asynchronous download is available.

Default Data Sources

The following Data Sources are in Pricefx by default:

Product – Product master data. Usually a copy of data from Master Data.
Customer – Customer master data. Usually a copy of data from Master Data.
uom – Unit of measure (needed to compare different units of measure and recalculate values, e.g., kg to gram). For details on UOM, see How to Load Units of Measure.
ccy – Currency conversion table to recalculate money values to reporting currencies. For details on currencies, see How to Set up Currencies.
cal – Calendar to create week, month, quarter and year fields from the date fields. In every new partition, this Data Source is present. It is used for filling the Time Dimension columns in Datamarts (usually transactions) which are marked as Pricing Date (during refresh). By default this Data Source is empty. To generate rows for it, you need to run the Data Load DMDS.cal of the type Calendar. This Data Load is present by default as well. By default, this Data Load generates date rows 4 years back and 2 years to the future. This is configurable on the server. You should schedule this Data Load to run periodically, otherwise you will run out of dates some day in the future.

Multiple Data Sources can be joined into Datamarts.

Templates

On the Templates tab you can create a new Data Source from:

Product Extensions
Customer Extensions
Company Parameter matrix tables
Price Records table (one template for both Quoting and Agreements & Promotions – the desired type is selected in the Data Load)
Rebate Records table
Rollups

A template is created automatically for each of these tables. See the Type column for the type code of the source.

Click a Template's label to view the fields and click the Data Source button to create a new Data Source from this template.

When you create new Data Source from a Company Parameter table, the Data Source will remain linked to the Company Parameter table from which it was created. If you create a new version of a Company Parameter (the same name but a different valid-after date), you must recreate the Data Source from the template.

Loading Process

When you create a Data Source you must define the data structure. You can either import the structure definition in JSON format or you can do it manually from a data sample or add individual fields.

The data is loaded via Data Feeds using Excel Client or File Loader. For each Data Source (once it is deployed), a Data Feed and Data Load are automatically generated. Loading into the source is actually loading into the staging table of that source – new records are not visible in the source until the staging table is flushed into source.

After clicking the Status History button on the Data tab, you can see Data Load history from the staging table. The current status is indicated in the Status column. The possible values are: Draft, Pending (previously loaded data is being flushed), Scheduled, Processing, Scheduled dirty, Ready (all data flushed, source is up-to-date), Cancelled, Error.

If you have a large data set, you can disable automatic loading of the data grid content on the Data tab. Then you can set up a filter first and download only a subset of the data.

Notes

When working with Data Sources, the following applies:

The Data Source name must not contain any spaces or special characters (only [a..z], [A..Z] and [0..9] are allowed).
Name and Label are not editable for system-generated sources.
Name is not editable for sources that have previously been deployed.
There is no other option than the Deploy action to save the Data Source in draft.
Before you successfully activate a Data Source, the changes and additions are not available in Analytics.
When deleting a Data Source, you need to delete the Data Feed in the Data Feed tab as well.

In Data Sources, you can also:

For a comprehensive example on using Data Sources, see How to Upload CSV Data into Analytics.