IntegrationManager 1.2.0
In this section:
Migration Steps
- Dependency Changes
- 1. Camel Upgrade – Update Codebase
- 2. Use Maven Archetype for New Projects
- 3. Connection Encryption
- 4. Component Changes
- 5. IM Versioning
- 6. Groovy Sandbox Changes
- 7. Properties Changes
- 8. Secrets Stored Outside Application JAR
- 9. Properties checks
- 10. Route error handling - enableSendingMail vs enable-sending-mail
- 11. Logging File Change
Dependency Changes
- IM library dependency version 1.2.0
- Spring boot version 2.2.7
- Camel version 3.2.0
1. Camel Upgrade – Update Codebase
A new major version of Camel is used which resulted in major changes.
Read Camel Migration Guides
Below are listed the official changes in Camel. Go through them and alter your code accordingly.
- https://camel.apache.org/manual/latest/camel-3-migration-guide.html
- https://camel.apache.org/manual/latest/camel-3x-upgrade-guide.html
Codebase Changelist
We strongly recommend to check your code against official guidelines and test changes properly before going live.
This changelist comes from our codebase changes:
Change | Old Value | New Value |
---|---|---|
Dependency | xmllist |
|
Components | json-to-list rest-to-list | – |
Exchanges method | getOut() |
|
Header name parameter renamed. | <setHeader headerName="foo"> | <setHeader name="foo"> |
Property name parameter renamed. | <setProperty propertyName="foo"> | <setProperty name="foo"> |
The http4 component has been renamed. | http4 | http |
Simple language no longer supports constants. | simple:Elasticity | constant:Elasticity |
Simple language function property renamed to exchangeProperty .Source: https://camel.apache.org/components/latest/languages/simple-language.html | simple:property.name | simple:exchangeProperty.name |
Quartz2 is now renamed to Quartz. | <quartz2...> | <quartz...> |
Unmarshal ref tag now needs to be in a custom element. | <unmarshal ref="xxx">... | <unmarshal><custom ref="xxx"/>... |
Other changes:
- Source: The simple language uses
${body}
placeholders for complex expressions or functions. (https://camel.apache.org/components/latest/languages/simple-language.html) - References to custom components must be prefixed.
- All Bean methods (
pfx:dsLoad
,pfx:dsIntegrate
, …) must be referenced asbean:xxx
- All routes must be referenced as
direct:xxx
- Mocks are prefixed with
mock:xxx
- JdbcComponent are referrenced as
jdbc:xxx
- SqlComponent are referrenced as
sql:xxx
Example of a Bean reference:
Before<pfx:dsLoad id="loadProduct" objectType="P" businessKeys="sku" mapper="dynamicPartitionMapper" /> <camelContext xmlns="http://camel.apache.org/schema/spring" id="camelContext"> <route> <from uri="direct://load-A"/> <setHeader headerName="loadDataMapperPfx"> <groovy>exchange.context.registry.lookupByName("dynamicProductMapper-variantA")</groovy> </setHeader> <to uri="loadProduct"/> <to uri="mock:response-load"/> </route> </camelContext>
After<pfx:dsLoad id="test://loadProduct" objectType="P" businessKeys="sku" mapper="dynamicPartitionMapper" /> <camelContext xmlns="http://camel.apache.org/schema/spring" id="camelContext"> <route> <from uri="direct://load-A"/> <setHeader name="loadDataMapperPfx"> <groovy>exchange.context.registry.lookupByName("dynamicProductMapper-variantA")</groovy> </setHeader> <to uri="bean:loadProduct"/> <to uri="mock:response-load"/> </route> </camelContext>
- All Bean methods (
2. Use Maven Archetype for New Projects
There is a Maven archetype created. It is recommended to use this archetype for further development, as the archetype follows best practices, already has the basic configuration and will be regularly updated by Pricefx team.
You should also check your current code against the archetype.
Archetype documentation: Create Project from Maven Archetype
3. Connection Encryption
Connection passwords and clientSecrets will from now on be stored encrypted. Encryption password is managed by the integration.connections.encryption-password
property. Encrypted fields are easily recognized by the {ENC} prefix. There is also an option to auto-encrypt current unencrypted properties by setting the integration.connections.auto-encrypt
property to true.
All currently unencrypted connections will from now on appear in the logger with the WARN level.
If an encrypted connection cannot be read (e.g. changed encryption password), the connection cannot be used.
4. Component Changes
Component | Change | Old value | New value |
---|---|---|---|
pfx-event | The pfx-event component needs now to have the method specified. | <pfx-event?delay=5000/> | <pfx-event:fetch?delay=5000/> |
pfx-c4c | The pfx-c4c component needs now to have the method specified as required param. | <to uri="pfx-c4c?method=fetchCustomers&username=..."/> | <to uri="pfx-c4c:fetchCustomers?username=..."/> |
pfx-api | The Implemented | <to uri="pfx-api?method=loaddata&objectType=C&mapper=myMapper"/> | <to uri="pfx-api:loaddata?&objectType=C&mapper=myMapper"/> |
pfx-rest | There are several noteworthy changes on this component – see the changelog of the pfx-rest component. | ||
pfx-dynamics | This component was removed. | ||
pfx-filter | This component was removed. | ||
pfx-hybris | Marked as deprecated. Use pfx-rest . | ||
pfx-google-shopping | Marked as deprecated. Use pfx-rest. | ||
pfx-odata2 | Marked as deprecated. Use pfx-rest. | ||
pfx-salesforce | Marked as deprecated. Use pfx-rest. |
5. IM Versioning
From now on, we switched to a more common versioning of IM. For more details see /wiki/spaces/INTG/pages/1492975782 (access for Pricefx only).
6. Groovy Sandbox Changes
We were forced to narrow the scope of what Groovy sandbox can perform. If you used java.lang.System
or java.lang.Class
in your Groovy class, you are no longer able to do that. Please go through the updated documentation for more information how to handle it. You may also have to define whitelisted classes from PlatformManager.
7. Properties Changes
From now on, you should use only properties prefixed with integration.*
when changing IM behavior. See the archetype documentation for required properties.
8. Secrets Stored Outside Application JAR
By default, the secrets (logstash and kafka connections, default passwords) are now stored in Config Server. For more details read the /wiki/spaces/INTG/pages/2513666473 specification (Pricefx employees only) and Use ConfigServer for Passwords Management (Optional).
9. Properties checks
There is a property check running on the startup now. It will log all resolved properties so you might check possible duplicates, problems etc.
10. Route error handling - enableSendingMail vs enable-sending-mail
There is now a property check that checks wrong setup of property enable-sending-mail
, which syntax wasn't supported. Please, use enableSendingMail
. Mailing configuration now support both formats but the enable-sending-mail
is deprecated.
11. Logging File Change
There is a new default location for logs - logs/main.log
(instead of main.log
). Your options:
- If possible, ask someone to use salt
deploy_im_standard_im_1_2_and_newer
. - If not possible and you want the same behavior (to store into the
root
folder):
Put-Dlogging.file.name=main.log
intoJAVA_OPTS
and
into application.properties.integration.logging.file=main.log
- If the new location suits you, create a folder
logs
in theroot
folder.
Improvements
New Feature Description | ID |
---|---|
There is now a class which handles authentication and caching to Pricefx. The class is standalone and must be used as a bean. | PFIMCORE-55 |
The Maven archetype was created. The steps for a basic project setup were updated. | PFIMCORE-275 |
ITests were merged into Unit tests as external resources were mocked. | PFIMCORE-353 |
Authentication for the pfx-rest component is now cached. Frequency of the cache renewal can be set via the integration.schedulers.auth.cache.ms property and is set to 10 minutes by default. See the documentation. | PFIMCORE-431 |
There is now a possibility to update provisioned IM via PlatformManager. | PFIMCORE-439 |
There is now a possibility to start/stop IM instance via PlatformManager. | PFIMCORE-440 |
Routes can now be automatically removed along with the maper and connection via API. | PFIMCORE-441 |
Connection passwords will from now on be stored encrypted. Password encryption is managed by the integration.connections.encryption-password property. Encrypted passwords are easily recognized by the {ENC} prefix. There is also an option to auto-encrypt current unencrypted passwords by setting the integration.connections.auto-encrypt property to true. All unencrypted connections will from now on appear in the logger with the WARN level. If the encrypted connection cannot be read (e.g. changed encryption password), the connection cannot be used. | PFIMCORE-442 |
There is a new component for sending custom messages to PlatformManager. See pfx-info Component. | PFIMCORE-444 |
There is now a masking mechanism which prevents sensitive information to be stored into log files. See Logging for more information. | PFIMCORE-446 |
Camel was migrated from 2.25 to 3.2. Follow the steps described above. | PFIMCORE-447 |
There is a new pfx-config component. | PFIMCORE-448 |
Fetch is now implemented for the following object types: DS DM LTV PLI PGI CTLI (new) DCR (new) F (new) MPL (new) PBOME (new) PCOMP (new) PDESC (new) PX (new) PXREF (new) QLI (new) W (new) | PFIMCORE-450 |
pfx-odata2 now calls also GET on a resource. | PFIMCORE-454 |
`org.apache.camel:apt` was removed because it was unused and caused problems. | PFIMCORE-457 |
It is possible now to invalidate the PP cache. | PFIMCORE-464 |
SAP Commerce migrated to 1905. | PFIMCORE-467 |
pfx-rest now takes headers from exchange/connection. See the documentation for the pfx-rest component. | PFIMCORE-468 |
pfx-rest now handles path and query parameters in more standardized manner. See the documentation for the pfx-rest component. | PFIMCORE-469 |
pfx-rest component shares context between component usages. | PFIMCORE-470 |
Code of IM now resides in Gitlab. | PFIMCORE-477 |
pfx-rest can set a list of acceptable HTTP responses. | PFIMCORE-489 |
pfx-rest auth is refreshed upon token expiration. | PFIMCORE-491 |
There are now more configurable options for the pfx-rest component. | PFIMCORE-492 |
pfx-event now needs to specify the method (e.g. pfx-event:fetch?...) and it creates archive directories if required. | PFIMCORE-502 |
Groovy sandbox now cannot be disabled. | PFIMCORE-506 |
Groovy sandbox now runs on every processor. | PFIMCORE-510 |
IM now provides API for creating typed connections. | PFIMCORE-511 |
Defined partitions are now sent to Kafka via the startup event. | PFIMCORE-519 |
There is now a banner in logs showing the IM, Spring and Camel versions and profile used. | PFIMCORE-521 |
There is now an endpoint which shows all supported convertors. | PFIMCORE-522 |
pfx-rest can now properly decode encoded response. | PFIMCORE-527 |
Connections are now testable via API. | PFIMCORE-531 |
The massedit method was added to the pfx-api component. | PFIMCORE-534 |
The IM Java Client now uses the correct version when contacting APIs of different IM versions. | PFIMCORE-538 |
pfx-rest component now fills CamelHttpResponseCode with the return code. | PFIMCORE-543 |
Sensitive configuration data are now stored outside the application. See Use ConfigServer for Passwords Management (Optional) for details. | PFIMCORE-544 |
pfx-api now takes the connection from the partitionPfxApi parameter. | PFIMCORE-546 |
pfx-api can now call the save method of a quote. | PFIMCORE-547 |
A connection 'pricefx' added during runtime has a higher priority than the default 'pricefx' connection. | PFIMCORE-548 |
Java runner was added to the demo project. | PFIMCORE-550 |
It is possible now to loaddata into a PP without knowing its type beforehand. You have to specify objectType=LT and IM determines the table type based on valueType of the response. Currently, all MLTVs and JLTVs are supported for this functionality. | PFIMCORE-557 |
There are now methods: - StringToNumberGER.convert(in) - StringToNumberGER.convert(in, default) - StringToNumberUS.convert(in) - StringToNumberUS.convert(in, default) available in Groovy expressions to convert a string number in a given locale. | PFIMCORE-563 |
Provisioned IM from version 1.2.0 will now run in docker containers. | PFIMCORE-564 |
Application properties are now listed on startup to check the setup. | PFIMCORE-572 |
Both forms 'enableSendingMail' and 'enable-sending-mail' are now supported in the configuration. There is also a check for the latter one which is deprecated. | PFIMCORE-573 |
Fixed Issues
Bug Description | ID |
---|---|
The data payload sent only with headers does not generate the report in the Records tab. | PFIMCORE-342 |
The cluster name is sometimes not recognized based on the pfx.partition parameter URL. | PFIMCORE-401 |
Default logback-spring.xml does not work with ELKINT. | PFIMCORE-475 |
The HTTP status 409 is marked as NonRecoverable and is not retried. | PFIMCORE-478 |
OData component fails when using basic authorization for metadata get. | PFIMCORE-496 |
In IM data validations, the Quantity data type accepts just integers. | PFIMCORE-517 |
The IM artefacts are missing from maven-ext. | PFIMCORE-518 |
There is no possibility to retrieve metadata for the OBJECT and ARRAY types. | PFIMCORE-533 |
The pfx-api default connection gets violated in one route. | PFIMCORE-541 |
Kafka messages are masked and therefore IM instances are not monitorable. | PFIMCORE-542 |
IntegrationManager version 5.8.0