IntegrationManager 1.4.0 LTS

This document summarizes major improvements and fixes introduced in the Pricefx IntegrationManager release version.

VersionIntegrationManager 1.4.0 LTS (Long-Term Support)
Release dateMarch 9, 2021

In this section:

Migration Steps

Major Version Changes

  • From now on, IntegrationManager is released together with PlatformManager (approx. 1 release per month).
    For details see /wiki/spaces/INTG/pages/1492975782 (Pricefx only link).

Component Changes

  • pfx-csv:marshal property recordSeparator now requires the separator definition \n as LF and \r as CR. You may use various combination of separators.
    • Previously: <pfx-csv:marshal?recordSeparator=\r\n/>
    • Now:  <pfx-csv:marshal?recordSeparator=CRLF/>
  • Notable changes in parameters naming:
PreviouslyCurrently
DM.Refresh.dataMartName=xxxpfx-api:refresh?objectType=DM&targetName=xxx
ResetColumn.columnName=xxxpfx-api:resetColumn?columnName=xxx

Dependency Changes

  • camel-email was added as a default dependency – you do not need to provide the dependency anymore.

Property Changes

  • Default integration.route-error-handling.email.mailFrom property is now based on the integration name. E.g. if the integration.name property value is db-integration, integration.route-error-handling.email.mailFrom will be db-integration@pricefx.eu. You do not need to provide that property anymore.

Other

  • It is not recommended to use classes in the PFX namespace like Loaddata and Integrate which are used in XML definitions. Commands like Fetch, Save, Integrate, which were not part of the PFX namespace, but were accessible, are now removed. Please use the pfx-api Component instead. 
  • org.joda.time is no longer used within IntegrationManager. The only place where org.joda.time is still allowed is Groovy Sandbox. It will be removed in next LTS release.
  • Simple language can be used in criterions like: simple:exchangeProperty[MyProp]. The previous requirement was  $simple{exchangeProperty[MyProp]}.
  • MSDynamics service was added to IntegrationManager. Instead of using custom HTTP client, it is now defineable as an OAuth2 connection; see MS Dynamics 365 Configurations. Please, remove your MSDynamics service if it is used in integration, and use pfx-rest instead:
    • <to uri="pfx-rest:system?connection=qa_msdynamics_michalstepan_oauth&amp;systemName=msDynamics&amp;systemConfigurationItem=getRecord&amp;recordType=rebate&amp;recordId=2&amp;apiVersion=v9.1"/>

    • See pfx-rest Component – method 'system'.


Improvements

New Feature DescriptionID
MS Dynamics service has been added to System Configurations.PFIMCORE-413
Pricefx API usages have been moved to the pfx-api-client module. The pfx-api component now implements all calls to pfx-api. XML definition now uses PfxApi producers instead of commands.PFIMCORE-569
IM no longer uses org.joda.time. The only usage is in Groovy Sandbox where the Joda support will be dropped in next LTS release.PFIMCORE-585
Pricefx debug can now be set in runtime via REST API. IM HTTP client has been updated.PFIMCORE-586
Manual instances can now be started or stopped.PFIMCORE-605
IM can now handle general filter structure and filter preconfigurations via XML.PFIMCORE-608
The 'preferencesSource' field has been added into the Swagger definition manually.PFIMCORE-611
When the sandbox denies evaluation of a class, the script is listed in exceptions whenever possible.PFIMCORE-612
pfx-rest now supports streaming of content into a file. PFIMCORE-619
Converters documentation has been enhanced. PFIMCORE-620
PriceFxConnectionDto was created to allow instantiation of the connection from outside. Such connection is converted into PriceFxClient automatically. This is used in the dataload functionality as this connection cannot be created via connections API.PFIMCORE-622
There is a new property `integration.hostname` which overrides the default hostname resolver when filled in.PFIMCORE-623
Swagger client has been refreshed to have the latest core API.PFIMCORE-625
There is a new module 'System Configurations' which allows users to call external systems and retrieve data/metadata and perform standard mappings. Only JSON is supported.PFIMCORE-626
PFIMCORE-632
There is a new RoutesCheckerService job which checks for struct routes. The job is by default disabled. To enable it, set the property integration.schedulers.routes-checker.enabled=true. You must also specify the input type of the route ('from' definition) by setting the property integration.schedulers.routes-checker.input-types (by default sftp, ftp).PFIMCORE-633
The Filter component now supports more operators (to match Pricefx core 6.0).PFIMCORE-636

New classes were enabled in Sandbox:

  • java.time.ZoneOffset
  • java.time.temporal.ChronoUnit
  • java.time.DayOfWeek
  • org.apache.commons.lang.StringUtils
  • org.apache.camel.component.file.GenericFile

    and method: System.lineSeparator()
PFIMCORE-637

Properties loaded via REST API are now loaded as property resource. This allows them to be used in the context startup (beans) as the property load is performed in an earlier init phase. The whole logic is enabled only if integration.configuration.enabled=true. Note: You should restart IM after setting the property, so that the changes take effect.

Integration configuration directory structure is created depending on integration.name property. Although it works with spring.application.name property, it's recommended not to use it, since this property is internal and necessary for backward compatibility.

PFIMCORE-639
Maven modules 'integration-starters' and 'integration-platform' have been removed. Most of the small modules have been merged into 'pricefx-integration'.PFIMCORE-641
Dataload now sends progress information.PFIMCORE-642
AWS AutoConfiguration and SystemConfigurationsAutoConfiguration have been added. System Configurations can now be loaded either from AWS S3 or local resources. PFIMCORE-643
HTTP client now offers methods to get data from a connection.PFIMCORE-646
pfx-rest logging has been enhanced. Set loglevel to DEBUG for more information.PFIMCORE-648
pfx-rest can now handle file uploads.PFIMCORE-652
StringToDate now accepts string dates. It also allows you to set inputFormat to determine the format of the input string date.

Also, the parameter 'nullValue' has been added to further parametrize what is considered a null input after which 'defaultValue' is returned.
PFIMCORE-654
There is a new method pfx-api:internalCopy which requires a label.PFIMCORE-655
pfx-config now has the option toHeader which stores the result of the retrieval into a given header and keeps the exchange body intact. There is also a new option defaultValue which is returned if no value for the key is found.PFIMCORE-656
The [not] operator for filters has been implemented.PFIMCORE-657
Filter option [not] is now also supported in filters deployed via REST API.PFIMCORE-659
camel-email dependency has been added as a default dependency.PFIMCORE-661
The default mailFrom property is now based on the integration name. E.g. if the integration.name property value is db-integration, mailFrom will be db-integration@pricefx.eu.PFIMCORE-668
pfx-api:datamartImport has been implemented.PFIMCORE-671
pfx-api:datamartExport has been implemented.PFIMCORE-672
Web services via Spring-WS have been implemented, as well as the web service Delete. It is now possible to define a filter within the web service structure.PFIMCORE-673
Asynchronous delete has been implemented for all object types.PFIMCORE-684
Several not important log messages have been moved to the DEBUG level.PFIMCORE-685
IM HTTP client has been updated to transform deprecated structures for /camel/routes and /camelroutes to uniform API.PFIMCORE-686
Line separator in dataload templates is no longer hardcoded but it is now detected from the input data.PFIMCORE-687
There is now an option to set a business key without setting the length; that is done using the asterisk * character.PFIMCORE-688
pfx-xml:marshal has been implemented. Allowed body types are String and Map.PFIMCORE-689
Mappers defined via the Spring bean can now be used in pfx components in routes.PFIMCORE-690
System Configurations (pre-set configurations) now contain the Salesforce - Product mapper.PFIMCORE-692
The multilevel mapper now offers an aggregate and non-aggregate modes.PFIMCORE-694
FETCH for JLTV, JLTV2, MLTV, MLTV2, MLTV3, MLTV4, MLTV5 and MLTV6 now uses  lookuptablemanager.fetch instead of (general) fetch.PFIMCORE-695
General mapper has been enhanced with generic metadata definition via the transformer function. Configuration has been migrated to XML and aggregated with the FILTER configurations. DATA configurations have been renamed to SELECT configurations. Configuration extensions have been separated from the mapper/filter/connection API. HTTP client has been updated.PFIMCORE-696
Instance update event is now sent to the Kafka topic if a connection is added or removed.PFIMCORE-708
There is a new method pfx-rest:system which elevates the possibilities of system configurations into pfx-rest fetching capabilities.PFIMCORE-712
System configurations are now resolved in runtime with caching driven by the property integration.schedulers.system-configurations.cache.ms.PFIMCORE-713
Log masking:  properties with the name 'secret-key' are also masked now.PFIMCORE-717
System Configurations can now define converterExpression or Groovy in mapper rows.PFIMCORE-719

Fixed Issues

Bug DescriptionID
The web service Integrate throws a NullPointerException sometimes.PFIMCORE-509
IntegrationManager does not send cluster and server in a startup event for provisioned instances.PFIMCORE-539
IM HTTP client does not properly return Object for the method convert.PFIMCORE-621
pfx-api:fetch on the PX objectType causes an exception when SKU is missing in the filter condition.PFIMCORE-624
Retry mechanism does not handle Pricefx core exceptions.PFIMCORE-635
pfx-csv default separator is set to \n which is problematic at some platforms (macOS).PFIMCORE-653
Health check shows DOWN when one route is DOWN.PFIMCORE-658
Event is processed before it is fully saved into a file.PFIMCORE-660
Setting error-handling to false causes the application to fail on start.PFIMCORE-665
Web service filter definition cannot be properly deserialized.PFIMCORE-674
Filter deployment fails when the notNull operator is used.PFIMCORE-675
When loaddata fails, it logs an error about 'integrate' failure.PFIMCORE-676
No Camel context is loaded if IM is started without a profile.PFIMCORE-697
Simple language as value="simple:exchangeProperty[...]" defined in criteria fails on deploy.PFIMCORE-705
It is not possible to set recordSeparator for the pfx-csv:marshal command.PFIMCORE-714
Route used for invocation of a connection stays deployed when finished.PFIMCORE-718