IntegrationManager 1.2.0

This document summarizes major improvements and fixes introduced in the Pricefx IntegrationManager release version.

VersionIntegrationManager 1.2.0
Release dateSep 11, 2020

In this section:

Migration Steps

Dependency Changes

  • IM library dependency version 1.2.0
  • Spring boot version 2.2.7
  • Camel version 3.2.0

1. Camel Upgrade – Update Codebase

A new major version of Camel is used which resulted in major changes. 

Read Camel Migration Guides

Below are listed the official changes in Camel. Go through them and alter your code accordingly. 

Codebase Changelist

(warning) We strongly recommend to check your code against official guidelines and test changes properly before going live.

This changelist comes from our codebase changes:

ChangeOld ValueNew Value

Dependency xmllist was removed.

xmllist

camel-jaxb
camel-jackson

Components json-to-list and rest-to-list were removed.

json-to-list 
rest-to-list

Exchanges method getOut() is deprecated.

getOut()

getMessage()

Header name parameter renamed.<setHeader headerName="foo"><setHeader name="foo">
Property name parameter renamed.<setProperty propertyName="foo"><setProperty name="foo">
The http4 component has been renamed.http4 http
Simple language no longer supports constants.simple:Elasticityconstant:Elasticity
Simple language function property renamed to exchangeProperty.
Source: https://camel.apache.org/components/latest/languages/simple-language.html
simple:property.namesimple:exchangeProperty.name
Quartz2 is now renamed to Quartz.<quartz2...><quartz...>
Unmarshal ref tag now needs to be in a custom element.<unmarshal ref="xxx">...<unmarshal><custom ref="xxx"/>...

Other changes:

  • Source: The simple language uses ${body} placeholders for complex expressions or functions. (https://camel.apache.org/components/latest/languages/simple-language.html)
  • References to custom components must be prefixed.
    • All Bean methods (pfx:dsLoad, pfx:dsIntegrate, …) must be referenced as bean:xxx
    • All routes must be referenced as direct:xxx
    • Mocks are prefixed with mock:xxx
    • JdbcComponent are referrenced as jdbc:xxx
    • SqlComponent are referrenced as sql:xxx
    • Example of a Bean reference:

      Before
      <pfx:dsLoad id="loadProduct" objectType="P" businessKeys="sku" mapper="dynamicPartitionMapper" />
      
      <camelContext xmlns="http://camel.apache.org/schema/spring" id="camelContext">
      
      <route>
      	<from uri="direct://load-A"/>
      	<setHeader headerName="loadDataMapperPfx">
      		<groovy>exchange.context.registry.lookupByName("dynamicProductMapper-variantA")</groovy>
      	</setHeader>
      	<to uri="loadProduct"/>
      	<to uri="mock:response-load"/>
      </route>
      
      </camelContext>
      After
      <pfx:dsLoad id="test://loadProduct" objectType="P" businessKeys="sku" mapper="dynamicPartitionMapper" />
      
      <camelContext xmlns="http://camel.apache.org/schema/spring" id="camelContext">
      
      <route>
      	<from uri="direct://load-A"/>
      	<setHeader name="loadDataMapperPfx">
      		<groovy>exchange.context.registry.lookupByName("dynamicProductMapper-variantA")</groovy>
      	</setHeader>
      	<to uri="bean:loadProduct"/>
      	<to uri="mock:response-load"/>
      </route>
      
      </camelContext>

2. Use Maven Archetype for New Projects

There is a Maven archetype created. It is recommended to use this archetype for further development, as the archetype follows best practices, already has the basic configuration and will be regularly updated by Pricefx team.

(warning) You should also check your current code against the archetype.

Archetype documentation: Create Project from Maven Archetype

3. Connection Encryption

Connection passwords and clientSecrets will from now on be stored encrypted. Encryption password is managed by the integration.connections.encryption-password property. Encrypted fields are easily recognized by the {ENC} prefix. There is also an option to auto-encrypt current unencrypted properties by setting the integration.connections.auto-encrypt property to true.

All currently unencrypted connections will from now on appear in the logger with the WARN level.

(warning) If an encrypted connection cannot be read (e.g. changed encryption password), the connection cannot be used.

4. Component Changes

ComponentChangeOld valueNew value
pfx-eventThe pfx-event component needs now to have the method specified. <pfx-event?delay=5000/><pfx-event:fetch?delay=5000/>
pfx-c4cThe pfx-c4c component needs now to have the method specified as required param. <to uri="pfx-c4c?method=fetchCustomers&username=..."/><to uri="pfx-c4c:fetchCustomers?username=..."/>
pfx-api

The pfx-api component needs now to have the method specified. 

Implemented quote:savemassedit and fetch methods.

<to uri="pfx-api?method=loaddata&objectType=C&mapper=myMapper"/><to uri="pfx-api:loaddata?&objectType=C&mapper=myMapper"/>
pfx-restThere are several noteworthy changes on this component – see the changelog of the pfx-rest component.

pfx-dynamicsThis component was removed. 

pfx-filterThis component was removed. 

pfx-hybrisMarked as deprecated. Use pfx-rest.

pfx-google-shoppingMarked as deprecated. Use pfx-rest.

pfx-odata2Marked as deprecated. Use pfx-rest.

pfx-salesforceMarked as deprecated. Use pfx-rest.

5. IM Versioning

From now on, we switched to a more common versioning of IM. For more details see /wiki/spaces/INTG/pages/1492975782 (access for Pricefx only). 

6. Groovy Sandbox Changes

We were forced to narrow the scope of what Groovy sandbox can perform. If you used java.lang.System or java.lang.Class in your Groovy class, you are no longer able to do that. Please go through the updated documentation for more information how to handle it. You may also have to define whitelisted classes from PlatformManager.

7. Properties Changes

From now on, you should use only properties prefixed with integration.* when changing IM behavior. See the archetype documentation for required properties.

8. Secrets Stored Outside Application JAR

By default, the secrets (logstash and kafka connections, default passwords)  are now stored in Config Server. For more details read the /wiki/spaces/INTG/pages/2513666473 specification (Pricefx employees only) and Use ConfigServer for Passwords Management (Optional).

9. Properties checks

There is a property check running on the startup now. It will log all resolved properties so you might check possible duplicates, problems etc.

10. Route error handling  - enableSendingMail vs enable-sending-mail

There is now a property check that checks wrong setup of property enable-sending-mail, which syntax wasn't supported. Please, use enableSendingMail. Mailing configuration now support both formats but the enable-sending-mail is deprecated.

11. Logging File Change

There is a new default location for logs - logs/main.log (instead of main.log). Your options:

  • If possible, ask someone to use salt deploy_im_standard_im_1_2_and_newer.
  • If not possible and you want the same behavior (to store into the root folder):
    Put -Dlogging.file.name=main.log into JAVA_OPTS and integration.logging.file=main.log into application.properties.
  • If the new location suits you, create a folder logs in the root folder.

Improvements

New Feature DescriptionID
There is now a class which handles authentication and caching to Pricefx. The class is standalone and must be used as a bean.PFIMCORE-55
The Maven archetype was created. The steps for a basic project setup were updated.PFIMCORE-275
ITests were merged into Unit tests as external resources were mocked.PFIMCORE-353
Authentication for the pfx-rest component is now cached. Frequency of the cache renewal can be set via the integration.schedulers.auth.cache.ms property and is set to 10 minutes by default. See the documentation.PFIMCORE-431
There is now a possibility to update provisioned IM via PlatformManager.PFIMCORE-439
There is now a possibility to start/stop IM instance via PlatformManager.PFIMCORE-440
Routes can now be automatically removed along with the maper and connection via API.PFIMCORE-441

Connection passwords will from now on be stored encrypted. Password encryption is managed by the integration.connections.encryption-password property. Encrypted passwords are easily recognized by the {ENC} prefix. There is also an option to auto-encrypt current unencrypted passwords by setting the integration.connections.auto-encrypt property to true.

All unencrypted connections will from now on appear in the logger with the WARN level.

If the encrypted connection cannot be read (e.g. changed encryption password), the connection cannot be used.

PFIMCORE-442
There is a new component for sending custom messages to PlatformManager. See pfx-info ComponentPFIMCORE-444
There is now a masking mechanism which prevents sensitive information to be stored into log files. See Logging for more information.PFIMCORE-446
Camel was migrated from 2.25 to 3.2. Follow the steps described above.PFIMCORE-447
There is a new pfx-config componentPFIMCORE-448
Fetch is now implemented for the following object types:
DS
DM
LTV
PLI
PGI
CTLI (new)
DCR (new)
F (new)
MPL (new)
PBOME (new)
PCOMP (new)
PDESC (new)
PX (new)
PXREF (new)
QLI (new)
W (new)
PFIMCORE-450
pfx-odata2 now calls also GET on a resource.PFIMCORE-454
`org.apache.camel:apt` was removed because it was unused and caused problems.PFIMCORE-457
It is possible now to invalidate the PP cache.PFIMCORE-464
SAP Commerce migrated to 1905.PFIMCORE-467
pfx-rest now takes headers from exchange/connection. See the documentation for the pfx-rest component.PFIMCORE-468
pfx-rest now handles path and query parameters in more standardized manner.  See the documentation for the pfx-rest component.PFIMCORE-469
pfx-rest component shares context between component usages.PFIMCORE-470
Code of IM now resides in Gitlab.PFIMCORE-477
pfx-rest can set a list of acceptable HTTP responses.PFIMCORE-489
pfx-rest auth is refreshed upon token expiration.PFIMCORE-491
There are now more configurable options for the pfx-rest component.PFIMCORE-492
pfx-event now needs to specify the method (e.g. pfx-event:fetch?...) and it creates archive directories if required.PFIMCORE-502
Groovy sandbox now cannot be disabled.PFIMCORE-506
Groovy sandbox now runs on every processor.PFIMCORE-510
IM now provides API for creating typed connections.PFIMCORE-511
Defined partitions are now sent to Kafka via the startup event.PFIMCORE-519
There is now a banner in logs showing the IM, Spring and Camel versions and profile used.PFIMCORE-521
There is now an endpoint which shows all supported convertors.PFIMCORE-522
pfx-rest can now properly decode encoded response.PFIMCORE-527
Connections are now testable via API.PFIMCORE-531
The massedit method was added to the pfx-api component.PFIMCORE-534
The IM Java Client now uses the correct version when contacting APIs of different IM versions.PFIMCORE-538
pfx-rest component now fills CamelHttpResponseCode with the return code.PFIMCORE-543
Sensitive configuration data are now stored outside the application. See Use ConfigServer for Passwords Management (Optional) for details.PFIMCORE-544
pfx-api now takes the connection from the partitionPfxApi parameter.PFIMCORE-546
pfx-api can now call the save method of a quote.PFIMCORE-547
A connection 'pricefx' added during runtime has a higher priority than the default 'pricefx' connection.PFIMCORE-548
Java runner was added to the demo project.PFIMCORE-550
It is possible now to loaddata into a PP without knowing its type beforehand. You have to specify objectType=LT and IM determines the table type based on valueType of the response. Currently, all MLTVs and JLTVs are supported for this functionality.PFIMCORE-557
There are now methods:
- StringToNumberGER.convert(in)
- StringToNumberGER.convert(in, default)
- StringToNumberUS.convert(in)
- StringToNumberUS.convert(in, default)
available in Groovy expressions to convert a string number in a given locale.
PFIMCORE-563
Provisioned IM from version 1.2.0 will now run in docker containers.PFIMCORE-564
Application properties are now listed on startup to check the setup.PFIMCORE-572
Both forms 'enableSendingMail' and 'enable-sending-mail' are now supported in the configuration. There is also a check for the latter one which is deprecated.PFIMCORE-573

Fixed Issues

Bug DescriptionID
The data payload sent only with headers does not generate the report in the Records tab.PFIMCORE-342
The cluster name is sometimes not recognized based on the pfx.partition parameter URL.PFIMCORE-401
Default logback-spring.xml does not work with ELKINT.PFIMCORE-475
The HTTP status 409 is marked as NonRecoverable and is not retried. PFIMCORE-478
OData component fails when using basic authorization for metadata get.PFIMCORE-496
In IM data validations, the Quantity data type accepts just integers.PFIMCORE-517
The IM artefacts are missing from maven-ext.PFIMCORE-518
There is no possibility to retrieve metadata for the OBJECT and ARRAY types.PFIMCORE-533
The pfx-api default connection gets violated in one route.PFIMCORE-541
Kafka messages are masked and therefore IM instances are not monitorable.PFIMCORE-542

IntegrationManager version 5.0.0