FAQ & Troubleshooting – Part 3

Owned by Ondřej Tesař
Last updated: Oct 16, 2024 by Ioana Kocurova
39 min read

Camel choice

Q: How I can get the following?

<route>             <from uri="direct:customerList" />             <when>                 <simple>${header.token} contains "ahoj"</simple>                 <log message="VALID TOKEN!" />             </when>             <otherwise>                 <log message="INVALID TOKEN!" />             </otherwise> ... </route>

LOG:

12:18:46.756 | INFO  | qtp1409864883-19 | route5 | ID-ni3mm4nd-K73SV-1523009915657-0-1 | route5 | VALID TOKEN! 12:18:46.759 | INFO  | qtp1409864883-19 | route5 | ID-ni3mm4nd-K73SV-1523009915657-0-1 | route5 | INVALID TOKEN!

How can I receive both? It does the same with "contains" or "==" or "=~", no matter what I choose. What am I doing wrong?

A: Choice is missing.

<choice> <when> <simple>$simple{headers["CamelSqlRowCount"]} > 0</simple> <log message="Received matmas material from DB $simple{body}" loggingLevel="INFO" logName="business.global.material" /> <split parallelProcessing="true"> <simple>$simple{body}</simple> <to uri="bean://productDataMapper" /> <to uri="disruptor://pxIntegrate?size=8192" /> </split> </when> <otherwise> <log message="No matmas material from db received" loggingLevel="WARN" logName="business.global.material" /> </otherwise> </choice>

Warning when calling calculation logic in PFX without sending any data to it

Q: I need to just call a calculation logic in PFX without sending any data to it. Any idea how to send an empty body? I tried <setBody> and empty ArrayList but I still got a warning.

A: It can be handled by catching the exception and continuing as if nothing happened. Or you just send some random dummy which then hopefully would be ignored.

A2: I think you need to set empty Map to body instead of ArrayList

Note about loadMapper 

If you already have values from a CSV header in the "sku;attribute1;..." format, just use:

You do not need to define a mapper if your CSV is 1:1.

Access body in criterion

Q: How do I access body[0] in a criterion like this?

And a property?

 

Q: It depends on what is in your body.

body[0][sku] means that your body is an array and you take the first element and it is the name sku.

If your body is just an object, it should be body.sku.

To learn about simple expressions, see Simple . 

Since Camel 2.15 it is correct to use:

Concatenation of several input values in mapper to final out field

Q: Is it possible to concatenate several input values in a mapper to final out field? Something like this:

A: You need to write a transformer.

Fetch QuoteLineItems

Q: How to fetch quote line items data?

A: Fetch the full quote using:

Cleaning up data feed

Q: How to clean up a data feed?

A: Call the Data Load Truncate after some FLUSH events. IM listens to the PADATALOAD_COMPLETED event and filters for DS_FLUSH | READY, then it will call DL TRUNCATE on the Data Feed.

PADATALOAD_COMPLETED event

Exporting price lists – mapping by label instead of name

Q: How can we export price list data correctly when its fields names are changeable?

A: Since IM 1.0.4.2, we can use net.pricefx.integration.processor.AddPLIMetadataBasedFieldsProcessor and the injectHeaderFromKeysToFirstLine attribute of the list-to-csv component to have headers from keys included and data mapped by its metadata/labels. For details see list-to-csv - Export Data to CSV (DEPRECATED). 

Price List export

Setting TAB character as CSV delimiter 

Q: How can I set the TAB character to delimiter a property of CsvDataFormat?

A: Use the hex code "&#x9;"

Saving or updating quote 

Q: How can I update a Quote from IM?

A: Use massEdit or quoteApi from pfxClient.

or 

Retrieving email list from user group 

Q: Is there an "official" way to retrieve an email list from a PFX user group?

A: use GeneralAPI and method fetch with object type U

fetch User Group by Id

Exporting to CSV file with duplicating header column name

Q: As you can notice, there is a duplicity in the column name. So my question is how can I achieve something like this?

A:  Write the header beforehand and then export only values.

Export header

Adding CFS calculation while processing

Q: Calling it to calculate when it is pending is ignored, but how is it when it's processing? Can we successfully add a calculate job while it's in processing state?

A:  While processing, the request will get into the queue.

Creating price list from IM

Q:  Is it possible to create a price list and upload the lines to it via a CSV header and data in a file. Is this possible from IM?

A:  Create JSON with a price list configuration. When you create a price list in PFX, it is the same process. You prepare a configuration and then you click Create PL and it is done.

Fetching data with BatchModed=true

Q: I've played with the new pfx-api:fetch component with BatchedMode=true. What should be the result of the fetch? I am getting List of BatchingInterval objects but with no data.

A: 

Web crawling - Metoda

Q: Did somebody hear about Methoda (web crawling partner)? Is it used somewhere?

A: The correct name is Metoda and its support is part of Pricefx.

Loading data to Data Source directly

Q:  I'm checking the new PFX APIs in IM 1.1.7 and I found that we can use pfx-api:loaddata to load data directly into a data source (ignoring steps of loading data into data feed + dmFlush call). Do you think it is possible?

A: Yes. Delete DF, then the data are loaded into DS. 

No events generated

Q: I have a listener for ITEM_UDPATE_PR and I wonder why no event is created if I do mass update on Price Records. What should be the right trigger or what are the rules? 

A: Mass edit/delete do not generate events. If you do an update, it generates an event. You can call integrate.

Starting LPG calculation 

Q: Is there any "official" API to start a LPG calculation from IM?

A: 

Preventing from being overridden by null value

Q: I have a requirement from the customer that they need to prevent null values from overriding existing values. Is there any solutions for that?

A:  No. You need to fetch data from Pricefx and compare it. Or if it is just only about one attribute, you can make a conditional integration request.

Working with IDOC format

Q: Do we have any project that works with SAP iDoc format (inbound data)?

A: iDocs are just XML files. You can search over Bitbucket and see how we dealt with them.

Working with JSON – jsonpath component

Q: Camel jsonpath is a good component for working with JSON. For example, I have this JSON: 

I would like to get the result 111 for allCalculationResults where resultName = price_authorized.

A:

Persistent message store in Camel

Q: Does anyone have some experience with persistent message store in Camel ? I tried to use krati but I had some issues with it.

A: We use MySQL and Oracle to store data on Bosch and MediaSaturn. 

Example with leveldb: https://svn.apache.org/repos/asf/camel/trunk/components/camel-leveldb/src/test/resources/org/apache/camel/component/leveldb/LevelDBSpringAggregateTest.xml (for aggregation only) 

Deleting approved contact

Q: Is it possible to delete an approved contract within PFX?

A: It is not possible. But Support can do that.

Using new components in IM 1.1.7

Q: I'm trying to use new components (in IM 1.1.7 and higher), but face error below:

A: Please add this section into your POM

Merging CSV columns into one PFX attribute

Q: How to merge two columns in CSV data into one column PFX table? (For instance, merging a “date” column and a “time” column into one column “datetime”)

A: You can do it in mapper:

Calling LPG recalculate in IM

Q: How to call LPG recalculate?

A: Please check MS project, custom processor "TriggerCompetitionRecalcProcessor".

SAP iDOC support in IM

Q: Does IM support working with SAP's iDOC flat text format?

A: No, IM currently only support iDOC as XML format.

Using dsLoad with key predefined in PFX

Q: When using dsLoad, how to force IM using keys defined in PFX instead of sending businessKey="sku,name"?

A: You can use detectJoinFields attribute (Supported from 1.1.9).

Using multiple PFX connections in single IM project

Q: Is it possible to use one inbound folder but data loaded into two partitions?

A: Yes, it's possible to have more than one pfx-connection in a IM project. These connections are identified by ID.

Connections are set up in camel-context.xml.

PRX Connections in camel-context.xml

To use a particalur connection you have to set the header partitionPfxApi before calling the PFX.

Route

Refer to these projects for more details: https://bitbucket.org/pricefx/toys-r-us-integration/src/master/src/main/resources/camel-context.xml

or https://bitbucket.org/pricefx/coesia-integration/src/master/ .

Working with big ZIP file larger than 4GB

Q: I am using Camel's ZipFileDataFormat to extract .ZIP files from customer. However, it showed error below when parsing files which have size bigger than 4GB.

A: That ZIP file must be corrupted. Possibly customer is using a non-supported ZIP64 application to compress the data file. In this case, the output file still can be extracted by some applications, but will be failed in some "standard" applications such as WinZip or Java JDK.

Samples of reading and writing data in Product Extension table

Q: Is there an example of read from and write to a Product Extension table?

A: Please put them here:
https://pricefx.atlassian.net/wiki/spaces/INTG/pages/537952266/Fetch+Data+from+Price+f+x
https://pricefx.atlassian.net/wiki/spaces/INTG/pages/537854028/Parse+CSV+and+Load+Data+to+General+Data+Source

A library of e-books

Q: Do/will we have a sharing library of e-books that we need in daily work?

A: I'm not aware of any shared ebook? We have an only a paper book: Camel in Action. One book is in Prague and the second is in Ostrava

Defining Table Name and Keys for PX or CX Master Data

Defining a name of a table for Product or Customer Extensions (CX, PX) is not all that intuitive.

Set the name of the table in a loadMapper first.

loadMapper

Do not forget to set detectJoinFields in the URI.

Key definition

Saving a flag into PFX Advanced configuration

Q: How to store a flag or a value to PFX Advanced configuration? I would like to store for example last updated time for a data feed.

A: There will be an API in XML. For now you have create a configurator bean and call its method. See the dana project.

Creating a REST endpoint to call your route

Q: I need to call a route that consumes a web service with special parameters,  e.g. for a retry or an initial data load.

A: Create a REST endpoint running on the localhost and implement a proxy route that consumes the endpoint and calls the target system. The implementation is used in the Cox project.

The sample code takes a JSON payload, sets headers and calls the target service.

Define the REST endpoint in the camel-context.xml file:

REST endpoint

Implement the proxy route:

The proxy route

The target route consumes the seda endpoint. Do not forget to override HTTP headers:

The target route

Changing the heap size for the IntegrationManager in a dedicated environment

You may find that a dedicated environment is very slow. You get very a slow response from the PFX UI too. Check the operational memory allocation. What you usually get is 32 GB of RAM. The PFX server alone has the heap size of 24 GB. Once you deploy more than one IntegrationManager on the server, you are in a potential trouble. IM has a default heap allocation of 4 GB. Once both IMs will reach their maximum heapsize, the operating system will start swapping and everything slows down. The heapsize of IM can be adjusted. You need to create a a Helpdesk ticket and specify the size. The size is controlled by the parameter -Xmx.

Checking the heapsize of IM, both have 2 GB:

Heap Size

Checking the amount of the RAM:

RAM

GUnzip processor for bigger files

The camel GZip unmarshal is in memory only. For larger files you end up with the out of memory exception. The following processor uses streams and creates .done file when unzipping is done.

GUnzipProcessor.java

How to use constants and reusable code snippets

I had a map value that was static but used it in a number of places.

  1. I created a class with a static method returning static members.

    Util Class

     

  2. I used in a Groovy code whenever I needed.

    Groovy snippets

Making a route to be a singleton

I need to update status of a row in a PPV table from multiple routes. First I fetch the row and then make changes. This unfortunately leads to a concurrency conflict. Data are overwritten as more than one instance of the update route exists. You can set <from> for the "direct:" to be blocking.

http://camel.apache.org/direct.html

Route

Formatting date when exporting a price list

When fetching a price list, the data come as String. Normally you would do a converter inside a mapper. That does not work and actually throws an exception. You can resolve it in Groovy:

Date Formatting

Getting output of a formula logic

Sometimes it is easier to implement a service inside Pricefx using a formula. The formula creates a map that is returned in a JSON form when the formula is called.

The logic element that emits the response has to have the Display Mode set to 'Everywhere'.
Request URI: https://irm-qa.pricefx.eu/pricefx/ironmtn-dev/formulamanager.executeformula/RadovanJsonTest
Parameters request payload:

Response body:

 

Logic parameter access:

Logic emitting the response:

Custom file processing strategy driven by a signal file

A customer uploads files on a cloud based SFTP. Once the upload is finished, it creates done_yyyymmdd.ctl file. After that we should start downloading files. As simple as it sounds it is not easily implemented in Camel.

I implemented a custom file processing strategy based on Class GenericFileRenameProcessStrategy<T> from the Camel sources. You define a bean and then put into it the file URI.

Source code:

https://bitbucket.org/pricefx/iron-mountain-integration/src/master/src/main/java/net/pricefx/integration/component/file/strategy/ironmountain/SftpDoneCtlFileProcessingStrategy.java

Server health check warning

How to disable this warning?

Set the following in the application.properties file (for core 1.1.11 or earlier): 

This will be by default disabled in IM 1.1.12.

How to fetch mapping of attributes to labels for Customer, Products and Extensions

Got tired of typing attribute numbers and their labels from PriceBuilder tables (Customers, Products, Customer Extensions, Product Extensions)?

You can easily fetch them over the API.

Sample endpoint for the CX mapping:

https://delphi-qa.pricefx.eu/pricefx/delphi-dev/fetch/CXAM

Filter definition for a particular extension:

Filter

Type Codes can be found here.

Sample response:

Response

This sample shows how to get just the relevant lines from the response using awk. It can be done with jq jq too.

There is a set of Groovy scripts to support this functionality.

https://bitbucket.org/pricefx/im-interface-generator/src/master/

How to set old IM <1.1.17 to work with openJDK 11 and newer

It may happen that new IM 1.1.17 is not released yet but some IMs should go to new servers with openJDK 11. Then you need to start old IM versions on openJDK 11 and newer.

  1. Add two dependencies into pom.xml and change the version of the spring-boot-maven-plugin.
    Example commit is here: https://bitbucket.org/pricefx/flint-nw-integration/commits/3b51ab77d1ea2878e7fd749bd34e78c55b9b26ee

  2. Create a support ticket specifying that you want to change your IM to work with openjdk11 and newer.

  3. The change to be done is from this:

    to this:

How to disable IM auto-registration into PlatformManager

Since IntegrationManager 1.1.17, IM instances are auto-registered in PlatformManager: IM sends the IntegrationManagerInstanceStartup event into Kafka every time IM starts up. This is enabled by default and could be disabled with the following configuration property:

How to process CUSTOM events in IM

Events in general are processed by eventType as in integration.events.event-to-route-mapping.ITEM_APPROVED_Q=direct:quoteApproved.

However, CUSTOM events have different "name" (eventType) in EventAdmin and eventType in the data of the event. IM has two routes to process events: the first one downloads the event by the "name" and the second one processes it (sends it to the route) using the eventType inside the event.

As these two eventTypes are different in CUSTOM events, you have to use this hack to process them (1st downloads all CUSTOM events, 2nd processes your custom event):

Using Header/Property in groovy exchange loadMapper

Understanding Imports and Includes in WSDL Files

  • include –  Only includes types from XSD to the current namespace.

  • import – Imports a namespace. There is a difference between xsd:import and wsdl_import.

It is well explained at https://www.ibm.com/developerworks/webservices/library/ws-tip-imports/ws-tip-imports-pdf.pdf .

Truncating Only Flushed Rows from Data Feed

Only flushed rows from a data feed can be truncated by adding dtoFilter to the truncate command. You have to filter rows where the column formulateResult equals "OK".

Filter and truncate

PX Columns Character Size Limits

For each column, there is a limit of 255 characters. For a product extension with 50 attributes, the limit is 70 characters for each column. If you change the size of the product extension from a lower number of attributes to 50 attributes and some of the attributes have more than 70 characters, a warning is displayed and the character string is truncated.

For details see Product Extensions. 

Event Types

For details see EventType Class. 

pfx-api:unmarshal does not propagate an exception outside of a split

When there is an error in the CSV format of the data like a quote inside quotes etc., the unmarshal component throws an exception but it does not stop processing the file. It merely skips the current batch and goes on. You have to add stopOnException to the split definition.

Split

Avoid Concurrent Updates on a Row

When I saved a status of processing in a PP table, I ran into an issue when the row-updating route was called from multiple places. During high load, it caused randomly the following exception:

net.pricefx.integration.api.NonRecoverableException: Error: There is probably a long running task which already updated or deleted data you are trying to manipulate. Please refresh your view. (Concurrent Data Modification)
at net.pricefx.integration.api.PriceFxExceptionTranslator.doRecoveryActions(PriceFxExceptionTranslator.java:119)
at net.pricefx.integration.api.client.LookuptableApi.integrate(LookuptableApi.java:501)

at net.pricefx.integration.command.ppv.Integrate.execute(Integrate.java:71)

I avoided the issue by letting the route process only one exchange at a time using ThrottlingInflightRoutePolicy.

Bean Definition

 

Route Definition

Log Snippet:
10:24:22.709 | INFO | Camel (camel-1) thread #13 - file:///home/customer/irm-emea-dev/filearea/inbound | dataSetFileStatusUpdate | ID-linux-wbx3-1601540648191-0-1 | o.a.c.i.ThrottlingInflightRoutePolicy | Throttling consumer: 2 > 1 inflight exchange by suspending consumer: Consumer[direct://dataSetFileStatusUpdate]
10:24:22.709 | INFO | Camel (camel-1) thread #13 - file:///home/customer/irm-emea-dev/filearea/inbound | dataSetFileStatusUpdate | ID-linux-wbx3-1601540648191-0-1 | o.a.c.i.ThrottlingInflightRoutePolicy | Throttling consumer: 1 <= 1 inflight exchange by resuming consumer: Consumer[direct://dataSetFileStatusUpdate]

onComplete behaviour when calling one route multiple times

OnComplete block is executed at the end of a parent route if used in a child route using the latest exchange. If the same route is called multiple times from a parent route, the onComplete block is not executed at the end of the child route but after all calls are finished. See the comments in the code sample:

How to Copy File(s) with SCP from One Server to Another via Proxy Jump

Since there is 2FA SSH in use, you no longer can easily copy a file from one server to another via SCP.

The way to do it: Copy from the remote server node1.customerA.pricefx.net (PROD) to the current server node1.customerA-qa.pricefx.net (QA).

Now you have the required file from the PROD server in the /tmp folder of the QA server.

How to Remove IM from Server (with Refresh of Monitoring Tool)

How to Send a Message to /dev/null

Sometimes you need to throw away a message to get a flow going. It can be achieved by logging to an empty channel. Camel will mark the message as consumed and go to another one.

Empty log

Correct setup for LOG and ELK in IM 1.3.x and newer

Many integrations might have a wrong setup. If you migrated to Camel 3.5 (IM 1.3.x and newer), you need to do these steps: 

  1. Remove the logback-spring.xml file from your IM project.
    IM has its own configuration file. The main.log file will be created in the new subfolder "logs" in the IM instance folder. Monitoring is available there. 

  2. Remove these properties from the property files. These properties are not needed. 

     

  3. Add these new properties into the property files. It is configuration for ELK if a config server is not used. 

How to set Constructor Parameter for Bean, e.g. Converter

Some values are not available as properties. For example pattern for DecimalToString can only be set as a constructor parameter.

How to access Spring properties in Simple blocks

Here is an example how to log a property set in a property file:

Manual Integration that uses event processing is filled with "perEventTypeEventRouteInputEventRoutePADATALOAD_COMPLETED" messages

Under Loggers, search for net.pricefx.integration.api.PriceFxExceptionTranslator and change the log level to WARN.

Provisioned instance fails on no spring.config.import property has been defined

If you had a provisioned AWS instance on version 3.7.0 and below, you might encounter this problem when you switch your project to Custom Image build. This is caused by missing dependency in pom.xml. Just add this dependency to pom.xml to resolve the issue:

Explanation: We have not shipped IM with bootstrap dependency/setup because it was not necessary. Now, when we added an option to build a custom image, it actually builds a project as a Maven project. So without bootstrap, it cannot connect to the config server.

IntegrationManager version 5.8.0