In this section:

Upgrade to Version 8.0 and Higher

Date Type Change

Description of the Backend Change

In 8.0 Godfather release, there was a low-level change regarding the datetime (timestamp) fields representation in the core code, domain objects. While it was a big low-level change, it has kept the backward compatibility. To sum up the change, in the server Java code Pricefx now uses LocalDateTime (more modern representation of datetime values in Java JDK) instead of the traditional java.util.Date.

We invested a major effort to minimize possible impacts. So for example, all JSON serialization/deserialization should be backward compatible, as well as processing the domain objects in the Groovy logics (that means that even if the LocalDateTime is stored internally in the Pricefx core server code, the existing logics still get the values represented as java.util.Date to stay compatible with older Pricefx versions).

Reasons/Motivation

The reasons were related to performance. This change was based on the recommendation from the Hibernate project team with the goal to improve performance / lower CPU utilization.

Impact on Logics

There can be some corner cases. For example, when the Groovy logic relies on the default behavior of conversion of Date to string using the toString() method - i.e. without some formatter (e.g. SimpleDateFormat) and when this string representation is later parsed into object again.

For example, if some of the table fields were originally returned as java.sql.Date (that is subclass of java.util.Date), and now they return differently, then the behavior of their toString() methods differs.

In Godfather, both are internally replaced with LocalDateTime, but for the backward compatibility, they are converted into java.util.Date in the logic. So there could be a change in the default toString() behavior.

How to Avoid Potential Issues

• If possible, do not convert values of the Date type to string and back just to get the original value.

• Use a formatter - such as SimpleDateFormat - to control what specific format is used for formatting to String or when parsing from String.

  Implementation of toString() in the JDK classes is outside of Pricefx control, so when the default formatting is different in java.sql.Timestamp and java.util.Date, it can lead to different results.

• Check the data type of the field with date - e.g. using instanceof.

Sample of conversion of date to a string in a controlled way:

Importing Empty Strings to Data Source

Summary

In previous versions, empty strings were imported to Data Source as such, while in later 7.x releases and Godfather 8.0 empty strings are imported as null.

How to Avoid Potential Issues

There can be a problem with loading data to Data Sources when a key field contains an empty string – it will now be imported as null which is not allowed by the Data Source.

As a solution, add the datamart.loadData.importEmptyStrings property to the pricefx-config.xml file. If set to true, empty string will be imported as empty strings, if set to false, they will be imported as null.

This property is available since versions 8.4.11 and 9.1.1.

Upgrade to Version 8.1 and Higher

Changes in Datamart Refresh

Summary

Behavior of updated rows in the Analytics module during a Datamart Refresh has changed in Godfather 8.1: calculated fields are now cleared to NULL instead of being preserved.

Background

As a part of larger refactoring effort for the Godfather 8.0 and 8.1 releases, there were internal changes for the Data Load implementation which bring performance improvements to incremental Refresh Data Loads. Data Loads which in some cases took many hours, will now often complete in less than 1 hour due to the new approach.

On a functional level, when a Datamart defines loadable calculated fields, these need to be invalidated when the input values to the calculation may have changed, as is the case when existing rows are updated by a Data Load. New rows obviously also need to be calculated. For this reason, the Calculation job that performs this enrichment is to be run after the Refresh. In such setups, the state of the Datamart will be the same in 8.1 compared to previous versions.

However, when the Calculation job is not run after each Refresh, then values will be left empty for updated rows. This is not a desirable configuration, since new rows would also have empty values for the calculated fields. In this scenario, the state of the Datamart is different for updated rows in 8.1.

Technical Details

Changes in the Datamart refresh approach:

• Before 8.1, Datamart rows were loaded from its constituent Data Sources using UPSERT and UPDATE statements.

  • This leaves the calculated (enriched) field values in place, not by design but as a result of the details of the generated SQL.

• In 8.1, the source by source approach was replaced with a single load from a JOIN of all Data Sources, and existing rows are REPLACED rather than UPDATED.

  • The previously calculated fields are now set to NULL.

Datamart Refresh + Enrichment Use Case

When data loaded in Analytics is to be enriched in the Datamart, the configuration requires that:

• A Datamart Refresh is run whenever the Datamart needs to reflect new and updated rows from its Data Sources.

• The Analytics Calculation job runs, ideally immediately thereafter, to populate the calculated fields of the new and updated Datamart rows.
  This step cannot be skipped, as otherwise the new rows would not have the enriched values. And existing rows might have had incorrect enriched values, if relevant data was changed in the Datamart Refresh.

Both jobs are to run in sequence to ensure the integrity of the Datamart.