Page Comparison

Table of Contents

none

minLevel	1
maxLevel	6
outline	false
style		default
type	list
printable	true

Camel choice

Q: How I can get the following?

...

Code Block

<choice>
	<when>
		<simple>$simple{headers["CamelSqlRowCount"]} > 0</simple>
		<log message="Received matmas material from DB $simple{body}" loggingLevel="INFO" logName="business.global.material" />
		<split parallelProcessing="true">
			<simple>$simple{body}</simple>
			<to uri="bean://productDataMapper" />
			<to uri="disruptor://pxIntegrate?size=8192" />
		</split>
	</when>
	<otherwise>
		<log message="No matmas material from db received" loggingLevel="WARN" logName="business.global.material" />
	</otherwise>
</choice>

Warning when calling calculation logic in PFX without sending any data to it

Q: I need to just call a calculation logic in PFX without sending any data to it. Any idea how to send an empty body? I tried <setBody> and empty ArrayList but I still got a warning.

...

A2: I think you need to set empty Map to body instead of ArrayList

Note about loadMapper

If you already have values from a CSV header in the "sku;attribute1;..." format, just use:

...

You do not need to define a mapper if your CSV is 1:1.

Access body in criterion

Q: How do I access body[0] in a criterion like this?

...

Code Block
<pfx:criterion fieldName="attribute3" operator="greaterThan" value="simple: exchangeProperty[JMSUpdated]"/>

Concatenation of several input values in mapper to final out field

Q: Is it possible to concatenate several input values in a mapper to final out field? Something like this:

...

Code Block
<pfx:loadMapper> <pfx:groovy expression="body.DOCNUM + ':' + body.MAT13 + ':' + body.VKORG" out="IDOCKey"/> </pfx:loadMapper>

Fetch QuoteLineItems

Q: How to fetch quote line items data?

...

Code Block
priceFxClient.getQuoteApi().fetch(typedId);

Cleaning up data feed

Q: How to clean up a data feed?

A: Call the Data Load Truncate after some FLUSH events. IM listens to the PADATALOAD_COMPLETED event and filters for DS_FLUSH | READY, then it will call DL TRUNCATE on the Data Feed.

PADATALOAD_COMPLETED event

Code Block

#listen to PADATALOAD_COMPLETED event
# route to execute logic when DS_FLUSH completed
<route id="eventPADataLoadCompleted">
	<from uri="direct:eventPADataLoadCompleted"/>
	<log message="PADATALOAD_COMPLETED event received with info - type: ${body[data][0][type]} - targetName: ${body[data][0][targetName]} - status: ${body[data][0][status]}"/>
    <filter>
		<simple>${body[data][0][type]} == 'DS_FLUSH' &amp;&amp; ${body[data][0][status]} == 'READY'</simple>
		<to uri="direct:dataLoadCompleted"/>
	</filter>
</route>

Exporting price lists – mapping by label instead of name

Q: How can we export price list data correctly when its fields names are changeable?

A: Since IM 1.0.4.2, we can use net.pricefx.integration.processor.AddPLIMetadataBasedFieldsProcessor and the injectHeaderFromKeysToFirstLine attribute of the list-to-csv component to have headers from keys included and data mapped by its metadata/labels. For details see list-to-csv - Export Data to CSV (DEPRECATED).

Price List export

Code Block

<bean id="addPLIMetadataBasedFieldsProcessor" class="net.pricefx.integration.processor.AddPLIMetadataBasedFieldsProcessor">
	<property name="priceListId" value="simple:property.priceListId"/>
</bean>
<pfx:list-to-csv id="priceBookCSVTransform" outputUri="direct:priceBookToFile"
                     injectHeaderFromKeysToFirstLine="true" mapper="priceBookMapper" dataFormat="priceBookCsvFormat"/>
<route id="priceList-Fetch">
	<from uri="direct:priceListApproved"/>
	<setProperty propertyName="priceListId">
		<groovy>Double.valueOf(body.data.id[0]).longValue() + ""</groovy>
	</setProperty>
	<setProperty propertyName="priceListLabel">
		<simple>${body[data][0][label]}</simple>
	</setProperty>
	<setProperty propertyName="targetDate">
		<simple>${body[data][0][targetDate]}</simple>
	</setProperty>
    <setProperty propertyName="expiryDate">
		<simple>${body[data][0][expiryDate]}</simple>
	</setProperty>
	<log message="'ITEM_APPROVED_PL' notification received for price list '${property[priceListLabel]}' (id: ${property[priceListId]})" loggingLevel="INFO"/>
    <to uri="fetchPriceListItems"/>     
    <log message="Fetched ${body.size()} rows from PL '${property[priceListLabel]}' (id: ${property[priceListId]})" loggingLevel="INFO"/>
    <process ref="addPLIMetadataBasedFieldsProcessor"/>
    <to uri="direct://priceBookCSVTransform" />
</route>

Setting TAB character as CSV delimiter

Q: How can I set the TAB character to delimiter a property of CsvDataFormat?

...

Code Block
<property name="delimiter" value=" "/>

Saving or updating quote

Q: How can I update a Quote from IM?

...

Code Block

SaveQuoteRequest saveQuoteRequest = new SaveQuoteRequest();
PriceQuoteRequest priceQuoteRequest = new PriceQuoteRequest();
PriceQuoteRequestData priceQuoteRequestData = new PriceQuoteRequestData();
try {
	Response price = pricefxClient.getQuoteApi().price(priceQuoteRequest.data(priceQuoteRequestData.quote(quote)));
	...
} catch (ApiException e) {
	...
}

Retrieving email list from user group

Q: Is there an "official" way to retrieve an email list from a PFX user group?

A: use GeneralAPI and method fetch with object type U

fetch User Group by Id

Code Block
priceFxClient.getGeneralApi().fetchByObjectTypeAndObjectId("UG", objectId, requestBody)

Exporting to CSV file with duplicating header column name

Q: As you can notice, there is a duplicity in the column name. So my question is how can I achieve something like this?

A: Write the header beforehand and then export only values.

Export header

Code Block

<setBody>                			         <simple>sku;OutletID;CreateTime;UserName;ChangePriceType;DeletePreMaintainedPriceFlag;SalesPrice;Currency;CreateSystem;ScheduledTime;Status;ExecTime;ExecMessage;Subsidiary;Brand;CalculationUUID;PriceTypeCode;PriceTypeText;FinalPriceReason;PriceProposal;LastPrice;S-Flags;P-Flags\n</simple>
            </setBody>
            <setHeader headerName="CamelFileName">
                <simple>${property[CamelFileName]}</simple>
            </setHeader>
            <to uri="file://{{central-sales-price-export-work-directory}}${fileOutParameters}&fileExist=Append" />

Adding CFS calculation while processing

Q: Calling it to calculate when it is pending is ignored, but how is it when it's processing? Can we successfully add a calculate job while it's in processing state?

A: While processing, the request will get into the queue.

Creating price list from IM

Q: Is it possible to create a price list and upload the lines to it via a CSV header and data in a file. Is this possible from IM?

A: Create JSON with a price list configuration. When you create a price list in PFX, it is the same process. You prepare a configuration and then you click Create PL and it is done.

Fetching data with BatchModed=true

Q: I've played with the new pfx-api:fetch component with BatchedMode=true. What should be the result of the fetch? I am getting List of BatchingInterval objects but with no data.

...

Code Block

<route>
    <from uri="timer://fetchDataFromDatamartByQuery?repeatCount=1"/>
    <to uri="pfx-api:fetch?objectType=DM&dsUniqueName=1669.DMDS&batchedMode=true&batchSize=5000"/>
    <split>
        <simple>${body}</simple>
        <to uri="pfx-api:next"/>
        <log message="${body}"/>
    </split>
</route>

Web crawling - Metoda

Q: Did somebody hear about Methoda (web crawling partner)? Is it used somewhere?

A: The correct name is Metoda and its support is part of Pricefx.

Loading data to Data Source directly

Q: I'm checking the new PFX APIs in IM 1.1.7 and I found that we can use pfx-api:loaddata to load data directly into a data source (ignoring steps of loading data into data feed + dmFlush call). Do you think it is possible?

A: Yes. Delete DF, then the data are loaded into DS.

No events generated

Q: I have a listener for ITEM_UDPATE_PR and I wonder why no event is created if I do mass update on Price Records. What should be the right trigger or what are the rules?

A: Mass edit/delete do not generate events. If you do an update, it generates an event. You can call integrate.

Starting LPG calculation

Q: Is there any "official" API to start a LPG calculation from IM?

...

Code Block
<bean id="calculateFinalPriceListLPG" class="net.pricefx.integration.command.pg.Calculate"> <property name="priceGridId" value="simple:property[loadedPriceGridId]"/> </bean>

Preventing from being overridden by null value

Q: I have a requirement from the customer that they need to prevent null values from overriding existing values. Is there any solutions for that?

A: No. You need to fetch data from Pricefx and compare it. Or if it is just only about one attribute, you can make a conditional integration request.

Working with IDOC format

Q: Do we have any project that works with SAP iDoc format (inbound data)?

A: iDocs are just XML files. You can search over Bitbucket and see how we dealt with them.

Working with JSON – jsonpath component

Q: Camel jsonpath is a good component for working with JSON. For example, I have this JSON:

...

Code Block
<setHeader headerName="authorizedPrice"> <jsonpath>$.data[0].allCalculationResults[?(@.resultName == 'price_authorized')].result</jsonpath> </setHeader>

Persistent message store in Camel

Q: Does anyone have some experience with persistent message store in Camel ? I tried to use krati but I had some issues with it.

...

Example with leveldb: https://svn.apache.org/repos/asf/camel/trunk/components/camel-leveldb/src/test/resources/org/apache/camel/component/leveldb/LevelDBSpringAggregateTest.xml (for aggregation only)

Deleting approved contact

Q: Is it possible to delete an approved contract within PFX?

A: It is not possible. But Support can do that.

Using new components in IM 1.1.7

Q: I'm trying to use new components (in IM 1.1.7 and higher), but face error below:

...

Code Block
<dependency> <groupId>net.pricefx.integration</groupId> <artifactId>camel-pfx</artifactId> </dependency>

Merging CSV columns into one PFX attribute

Q: How to merge two columns in CSV data into one column PFX table? (For instance, merging a “date” column and a “time” column into one column “datetime”)

...

Code Block
<pfx:groovy expression=" body.Date + ' ' + body.Time" out="attribute1"/>

Calling LPG recalculate in IM

Q: How to call LPG recalculate?

A: Please check MS project, custom processor "TriggerCompetitionRecalcProcessor".

SAP iDOC support in IM

Q: Does IM support working with SAP's iDOC flat text format?

A: No, IM currently only support iDOC as XML format.

Using dsLoad with key predefined in PFX

Q: When using dsLoad, how to force IM using keys defined in PFX instead of sending businessKey="sku,name"?

A: You can use detectJoinFields attribute (Supported from 1.1.9).

Using multiple PFX connections in single IM project

Q: Is it possible to use one inbound folder but data loaded into two partitions?

...

Connections are set up in camel-context.xml.

PRX Connections in camel-context.xml

Code Block

    <pfx:connection id="coesia" uri="${pfx-coesia.url}" partition="${pfx-coesia.partition}" username="${pfx-coesia.username}" password="${pfx-coesia.password}" debug="${pfx-coesia.debug:false}"/>
    <pfx:connection id="gdm" uri="${pfx-coesia-gdm.url}" partition="${pfx-coesia-gdm.partition}" username="${pfx-coesia-gdm.username}" password="${pfx-coesia-gdm.password}" debug="${pfx-coesia-gdm.debug:false}"/>
    <pfx:connection id="raj" uri="${pfx-coesia-raj.url}" partition="${pfx-coesia-raj.partition}" username="${pfx-coesia-raj.username}" password="${pfx-coesia-raj.password}" debug="${pfx-coesia-raj.debug:false}"/>
    <pfx:connection id="norden" uri="${pfx-coesia-norden.url}" partition="${pfx-coesia-norden.partition}" username="${pfx-coesia-norden.username}" password="${pfx-coesia-norden.password}" debug="${pfx-coesia-norden.debug:false}"/>

To use a particalur connection you have to set the header partitionPfxApi before calling the PFX.

Route

Code Block
<route id="productMasterData"> <from uri="{{coesia-products-fromUri}}"/> <setHeader headerName="partitionPfxApi"> <constant>coesia</constant> </setHeader> <to uri="direct:ProductProcess" /> </route>

...

or https://bitbucket.org/pricefx/coesia-integration/src/master/ .

Working with big ZIP file larger than 4GB

Q: I am using Camel's ZipFileDataFormat to extract .ZIP files from customer. However, it showed error below when parsing files which have size bigger than 4GB.

...

A: That ZIP file must be corrupted. Possibly customer is using a non-supported ZIP64 application to compress the data file. In this case, the output file still can be extracted by some applications, but will be failed in some "standard" applications such as WinZip or Java JDK.

Samples of reading and writing data in Product Extension table

Q: Is there an example of read from and write to a Product Extension table?

A: Please put them here:
https://pricefx.atlassian.net/wiki/spaces/INTG/pages/537952266/Fetch+Data+from+Price+f+x
https://pricefx.atlassian.net/wiki/spaces/INTG/pages/537854028/Parse+CSV+and+Load+Data+to+General+Data+Source

A library of e-books

Q: Do/will we have a sharing library of e-books that we need in daily work?

A: I'm not aware of any shared ebook? We have an only a paper book: Camel in Action. One book is in Prague and the second is in Ostrava

Defining Table Name and Keys for PX or CX Master Data

Defining a name of a table for Product or Customer Extensions (CX, PX) is not all that intuitive.

Set the name of the table in a loadMapper first.

loadMapper

Code Block

    <!-- Product Extension Mapper -->
    <pfx:loadMapper convertEmptyStringToNull="true" id="ProductExtensionMasterDataMapper">
        <!-- set Product Extension table name -->
        <pfx:simple expression="ProductExtension1" out="name"/>
        <!-- attribute mapping -->
        <pfx:body in="DIM_ARTICLE_KEY"       out="sku"/>
        <pfx:body in="PALLET_SPEC_ID"        out="attribute1"/>
        <pfx:body in="ROLLS_PER_PACK"    ...

Do not forget to set detectJoinFields in the URI.

Key definition

Code Block
<to uri="pfx-api:loaddata?mapper=ProductExtensionMasterDataMapper&objectType=PX&detectJoinFields=true"/>

Saving a flag into PFX Advanced configuration

Q: How to store a flag or a value to PFX Advanced configuration? I would like to store for example last updated time for a data feed.

A: There will be an API in XML. For now you have create a configurator bean and call its method. See the dana project.

Creating a REST endpoint to call your route

Q: I need to call a route that consumes a web service with special parameters, e.g. for a retry or an initial data load.

...

Define the REST endpoint in the camel-context.xml file:

REST endpoint

Code Block

        <dataFormats>
            <json id="gson" useList="true" library="Gson"/>
            <json id="json" useList="true" library="Jackson"/>
        </dataFormats>
        <!-- REST service for initial load of Price2Spy data. Works as a proxy for the Price2Spy. -->
        <!-- example:
            curl -X POST -H "Content-Type:application/json" http://localhost:42080/Price2Spy/price -d '{"dateChangeFrom": "2018-01-01 00:00:00","dateChangeTo":"2018-12-01 00:00:00" }'
        -->
        <restConfiguration bindingMode="json" component="jetty" port="42080" host="localhost"/>
        <rest consumes="application/json" produces="text/plain">
            <post uri="/Price2Spy/price">
                <route id="adhocPrice2SpyREST" >
                    <to uri="direct:adhocPrice2SpyCompetitorData"/>
                </route>
            </post>
        </rest>

Implement the proxy route:

The proxy route

Code Block

        <!-- for loading data through REST bound to localhost:42080 -->
        <route id="adhocPrice2SpyCompetitorData">
            <from uri="direct:adhocPrice2SpyCompetitorData"/>
            <!--<log message="Got ${body}"/>-->
            <setHeader headerName="Price2SpyDateChangeTo">
                <jsonpath>$.dateChangeTo</jsonpath>
            </setHeader>
            <setHeader headerName="Price2SpyDateChangeFrom">
                <jsonpath>$.dateChangeFrom</jsonpath>
            </setHeader>
            <to uri="seda:updatePrice2SpyCompetitorData"/>
        </route>

The target route consumes the seda endpoint. Do not forget to override HTTP headers:

The target route

Code Block

        <route id="updatePrice2SpyCompetitorData">
            <!--<from uri="quartz2://price2spyDataUpdateTimer?cron={{price2spy-pricing-data-cron}}&trigger.timeZone=America/Chicago&stateful=true" />-->
            <from uri="direct:updatePrice2SpyCompetitorData"/>
            <from uri="seda:updatePrice2SpyCompetitorData"/>
            <!-- setup headers for the REST call -->
            <setHeader headerName="CamelHttpMethod">
                <constant>POST</constant>
            </setHeader>
            <setHeader headerName="Content-Type">
                <constant>application/json</constant>
            </setHeader>
            <setHeader headerName="Accept">
                <constant>application/json</constant>
            </setHeader>
            <setHeader headerName="Authorization">
                <simple>{{price2spy-pricing-data-authToken}}</simple>
            </setHeader>
            <setHeader headerName="CamelHttpUri">
                <simple>https4://your-service-provider.....</simple>
            </setHeader>
            <setBody>
                <simple>{
                    "dateChangeFrom": "${header.Price2SpyDateChangeFrom}",
                    "dateChangeTo": "${header.Price2SpyDateChangeTo}"
                    }</simple>
            </setBody>
            <log message="Fetching data from ${header.Price2SpyDateChangeFrom} to ${header.Price2SpyDateChangeTo}." />
            <!--<log message="Sending body:\n${body}"/>-->
            <!--<process ref="debugProcessor"/>-->
            <to uri="https4://your-service-provider....."/>
            <!-- convert json to map -->
            <unmarshal ref="gson"/>

Changing the heap size for the IntegrationManager in a dedicated environment

You may find that a dedicated environment is very slow. You get very a slow response from the PFX UI too. Check the operational memory allocation. What you usually get is 32 GB of RAM. The PFX server alone has the heap size of 24 GB. Once you deploy more than one IntegrationManager on the server, you are in a potential trouble. IM has a default heap allocation of 4 GB. Once both IMs will reach their maximum heapsize, the operating system will start swapping and everything slows down. The heapsize of IM can be adjusted. You need to create a a Helpdesk ticket and specify the size. The size is controlled by the parameter -Xmx.

Checking the heapsize of IM, both have 2 GB:

Heap Size

Code Block

root@node1.irm-qa.pricefx.net ~ # ps ax|grep im-
 2700 ?        Sl    13:12 /usr/bin/java -Dsun.misc.URLClassPath.disableJarChecking=true -Xmx2G -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/pricefx/runtime/im-iron-mountain-qa -Xloggc:gc.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=5M -Dfile.encoding=UTF-8 -Dspring.profiles.active=iron-mountain_qa -jar /var/pricefx/runtime/im-iron-mountain-qa/im-iron-mountain-qa.jar
 3282 ?        Sl     0:58 /usr/bin/java -Dsun.misc.URLClassPath.disableJarChecking=true -Xmx2G -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/pricefx/runtime/im-iron-mountain-dev -Xloggc:gc.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=5M -Dfile.encoding=UTF-8 -Dspring.profiles.active=iron-mountain_dev -jar /var/pricefx/runtime/im-iron-mountain-dev/im-iron-mountain-dev.jar

Checking the amount of the RAM:

RAM

Code Block
root@node1.irm-qa.pricefx.net ~ # free -h total used free shared buff/cache available Mem: 31G 30G 186M 10M 201M 93M Swap: 15G 1.9G 14G

GUnzip processor for bigger files

The camel GZip unmarshal is in memory only. For larger files you end up with the out of memory exception. The following processor uses streams and creates .done file when unzipping is done.

GUnzipProcessor.java

Code Block

package net.pricefx.integration.processor.your_project;
import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.*;
import java.util.zip.GZIPInputStream;
/**
 * Processor for unzipping gzipped files using streams.
 *
 * @author rbiciste
 */
public final class GUnZipProcessor implements Processor {
    private static final Logger LOGGER = LoggerFactory.getLogger(GUnZipProcessor.class);
    /**
     * Unzips a file using gzip as a streams. Implemented to avoid out of memory exception.
     * Camel gzip unmarshal is memory only.
     * After decompression it creates .done file to signal CSV routes.
     */
    @Override
    public void process(Exchange exchange) throws Exception {
        //Decompress the file to a temp folder
        File file = new File(exchange.getIn().getHeader("CamelFilePath", String.class));
        GZIPInputStream in = null;
        OutputStream out = null;
        File target = null;
        File doneFile = null;
        try {
            //Open the compressed file
            in = new GZIPInputStream(new FileInputStream(file));
            String targetFileName = file.getName().substring(0, file.getName().lastIndexOf('.'));
            LOGGER.debug("Unzipping to " + targetFileName);
            //Open the output file
            target = new File(file.getParent(), targetFileName);
            target.createNewFile();
            out = new FileOutputStream(target);
            //Transfer bytes from the compressed file to the output file
            //Buffer of 1 mb.
            byte[] buf = new byte[1048576];
            int len;
            while ((len = in.read(buf)) > 0) {
                out.write(buf, 0, len);
            }
            // close streams
            in.close();
            out.close();
            LOGGER.debug("Unzipping is done. Streams closed.");
            // create .done file for CSV routes
            doneFile = new File(file.getParent(), targetFileName + ".done");
            doneFile.createNewFile();
            LOGGER.debug(".done file created.");
        } catch (FileNotFoundException e) {
            LOGGER.error("File " + file.getName() + " not found");
            throw e;
        } catch (IOException e) {
            LOGGER.error("Gunzipping the file " + file.getName() + " failed." + e.getMessage());
            throw e;
        } finally {
            // close in
            if(in != null) {
                in.close();
            }
            // close out
            if(out != null) {
                out.close();
            }
        }
    }
}

How to use constants and reusable code snippets

I had a map value that was static but used it in a number of places.

I created a class with a static method returning static members.

Util Class

Code Block

package net.pricefx.integration.util.ironmountain;
import java.util.HashMap;
import java.util.Map;
public class DataLoadOrchestrationUtils {
    /** returns Country Code from a file name like irm_bi_ca_rm_contract_details_20190217.csv
     *
      * @param fileName
     * @return
     */
    public static String getCountryCode(String fileName) {
        String[] parts = fileName.split("_");
        // country from 'irm_bi_ca_rm_contract_details_20190217.csv'
        return parts[2].toUpperCase();
    }
    /** returns Data Set from a file name like irm_bi_ca_rm_contract_details_20190217.csv
     *
     * @param fileName
     * @return
     */
    public static String getDataSet(String fileName) {
        String[] parts = fileName.split("_");
        return parts[parts.length-1].replace(".csv","");
    }
    /** returns Map of entities needed to run Calculations
     *
     * @return
     */
    static Map<String, String> entityMap;
    static {
        entityMap = new HashMap<>();
        entityMap.put("InvoiceRevenue", "attribute2");
        entityMap.put("ContractDetails","attribute3");
        entityMap.put("RateTable",      "attribute4");
        entityMap.put("BillCode",       "attribute5");
    }
    public static Map getEntityMap() {
        return entityMap;
    }
}

I used in a Groovy code whenever I needed.

Groovy snippets

Code Block

...
            <setHeader headerName="country">
                <groovy>
                    return net.pricefx.integration.util.ironmountain.DataLoadOrchestrationUtils.getCountryCode(headers.CamelFileNameOnly)
                </groovy>
            </setHeader>
            <log message="Setting country to ${header[country]}" loggingLevel="INFO"/>
            <!-- Data Set Date -->
            <setHeader headerName="dataSet">
                <groovy>
                    return net.pricefx.integration.util.ironmountain.DataLoadOrchestrationUtils.getDataSet(headers.CamelFileNameOnly)
                </groovy>
            </setHeader>
...
                    <transform>
                        <groovy>
                            // map entities to attributes in PPV
                            def entities = net.pricefx.integration.util.ironmountain.DataLoadOrchestrationUtils.getEntityMap();
                            // if list is empty throw the exception
                            if (body.size()==0) {
                                throw new RuntimeException("Received the flush event without data in Running state.")
                            } else { // update existing row
                                body[0].put(entities.get(headers['entity']),'Done');
                            }
                            return body;
                        </groovy>
                    </transform>

Making a route to be a singleton

I need to update status of a row in a PPV table from multiple routes. First I fetch the row and then make changes. This unfortunately leads to a concurrency conflict. Data are overwritten as more than one instance of the update route exists. You can set <from> for the "direct:" to be blocking.

http://camel.apache.org/direct.html

Route

Code Block

        <!-- sets the entity status into the PPV row -->
        <route id="dataSetFileStatus">
            <!-- blocks for only one consumer, multiple calls raise concurrent update issues. -->
            <from uri="direct:dataSetFileStatus?block=true"/>

Formatting date when exporting a price list

When fetching a price list, the data come as String. Normally you would do a converter inside a mapper. That does not work and actually throws an exception. You can resolve it in Groovy:

Date Formatting

Code Block
<pfx:groovy expression="if (body.ServiceReviewDate) { Date.parse('yyyy-mm-dd', body.ServiceReviewDate).format('MM/dd/yyyy')}" out="ServiceReviewDate" />

Getting output of a formula logic

Sometimes it is easier to implement a service inside Pricefx using a formula. The formula creates a map that is returned in a JSON form when the formula is called.

...

Code Block

api.local.response = [:]
def response = [:]
response.params = [] as List
api.local.estimatedVolume = api.stringUserEntry("estimatedVolume")
api.local.industry = api.stringUserEntry("industry")
api.local.numberOfMarkets = api.stringUserEntry("numberOfMarkets")
def params = [:]
params["estimatedVolume"] = api.local.estimatedVolume
params["industry"] = api.local.industry
params["numberOfMarkets"] = api.local.numberOfMarkets
response.params.add(params)
api.local.response["data"] = response
api.logInfo(api.jsonEncode(api.local.response))
return api.local.response

Custom file processing strategy driven by a signal file

A customer uploads files on a cloud based SFTP. Once the upload is finished, it creates done_yyyymmdd.ctl file. After that we should start downloading files. As simple as it sounds it is not easily implemented in Camel.

...

https://bitbucket.org/pricefx/iron-mountain-integration/src/master/src/main/java/net/pricefx/integration/component/file/strategy/ironmountain/SftpDoneCtlFileProcessingStrategy.java

Server health check warning

How to disable this warning?

...

This will be by default disabled in IM 1.1.12.

How to fetch mapping of attributes to labels for Customer, Products and Extensions

Got tired of typing attribute numbers and their labels from PriceBuilder tables (Customers, Products, Customer Extensions, Product Extensions)?

...

Filter definition for a particular extension:

Filter

Code Block
{ "data": { "_constructor": "AdvancedCriteria", "criteria": [ { "operator": "equals", "fieldName": "name", "value": "CustomerAddtionalAttr" } ] } }

Type Codes can be found here.

Sample response:

Response

Code Block

{
    "response": {
        "status": 0,
        "startRow": 0,
        "node": "node1",
        "csrfToken": "1t8kgyrh14ou7l8y9p3kqpa2q",
        "data": [
            {
                "version": 0,
                "typedId": "200.CXAM",
                "fieldName": "attribute1",
                "label": "INCOTERMS",
                "fieldType": 2,
                "requiredField": false,
                "readOnly": false,
                "name": "CustomerAddAttr",
                "createDate": "2019-08-19T22:05:05",
                "createdBy": 10,
                "lastUpdateDate": "2019-08-19T22:05:05",
                "lastUpdateBy": 10
            },

...

https://bitbucket.org/pricefx/im-interface-generator/src/master/

How to set old IM <1.1.17 to work with openJDK 11 and newer

It may happen that new IM 1.1.17 is not released yet but some IMs should go to new servers with openJDK 11. Then you need to start old IM versions on openJDK 11 and newer.

Add two dependencies into pom.xml and change the version of the spring-boot-maven-plugin.
Example commit is here: https://bitbucket.org/pricefx/flint-nw-integration/commits/3b51ab77d1ea2878e7fd749bd34e78c55b9b26ee
Create a support ticket specifying that you want to change your IM to work with openjdk11 and newer.

The change to be done is from this:

Code Block
if [ ${3} == "flintnw_prod" ]; then TARGET=int4.eu.pricefx.net deploy_im_standard ${1} ${2} ${3} ${4} exit 0 fi

to this:

Code Block
if [ ${3} == "flintnw_prod" ]; then TARGET=int4.eu.pricefx.net deploy_im_standard_openjdk11andnewer ${1} ${2} ${3} ${4} exit 0 fi

How to disable IM auto-registration into PlatformManager

Since IntegrationManager 1.1.17, IM instances are auto-registered in PlatformManager: IM sends the IntegrationManagerInstanceStartup event into Kafka every time IM starts up. This is enabled by default and could be disabled with the following configuration property:

Code Block
# disable kafka (it is recommended for DEV environment) integration.event-driven.enabled=false # disable autoregistration integration.event-driven.auto-registration.enabled=false

How to process CUSTOM events in IM

Events in general are processed by eventType as in integration.events.event-to-route-mapping.ITEM_APPROVED_Q=direct:quoteApproved.

...

Code Block
integration.events.event-to-route-mapping.CUSTOM=hackForCUSTOMevents integration.events.event-to-route-mapping.CUSTOM_OutBound_PP_ADD=direct:exportPriceList

Using Header/Property in groovy exchange loadMapper

Code Block
<pfx:groovy expression="request.headers.nationalCode + body.sku" out="prod_id"/> <pfx:groovy expression="exchange.properties.nationalCode + body.sku" out="prod_id"/>

Understanding Imports and Includes in WSDL Files

include – Only includes types from XSD to the current namespace.
import – Imports a namespace. There is a difference between xsd:import and wsdl_import.

It is well explained at https://www.ibm.com/developerworks/webservices/library/ws-tip-imports/ws-tip-imports-pdf.pdf .

Truncating Only Flushed Rows from Data Feed

Only flushed rows from a data feed can be truncated by adding dtoFilter to the truncate command. You have to filter rows where the column formulateResult equals "OK".

Filter and truncate

Code Block

<pfx:filter id="truncateFlushedFilter" resultFields="name">
    <pfx:and>
        <pfx:criterion fieldName="formulaResult" operator="equals" value="OK"/>
    </pfx:and>
</pfx:filter>
....
<to uri="pfx-api:truncate?targetName=DMF.SalesTransactions&dtoFilter=truncateFlushedFilter"/>

PX Columns Character Size Limits

For each column, there is a limit of 255 characters. For a product extension with 50 attributes, the limit is 70 characters for each column. If you change the size of the product extension from a lower number of attributes to 50 attributes and some of the attributes have more than 70 characters, a warning is displayed and the character string is truncated.

For details see /wiki/spaces/UDEV/pages/1680834622.

Event Types

For details see EventType Class.

pfx-api:unmarshal does not propagate an exception outside of a split

When there is an error in the CSV format of the data like a quote inside quotes etc., the unmarshal component throws an exception but it does not stop processing the file. It merely skips the current batch and goes on. You have to add stopOnException to the split definition.

Split

Code Block
<!-- batching, add total number of rows in the header --> <split streaming="true" strategyRef="recordsCountAggregation" stopOnException="true"> <tokenize token="\n" group="5000"/>

Avoid Concurrent Updates on a Row

When I saved a status of processing in a PP table, I ran into an issue when the row-updating route was called from multiple places. During high load, it caused randomly the following exception:

...

I avoided the issue by letting the route process only one exchange at a time using ThrottlingInflightRoutePolicy.

Bean Definition

Code Block

    <!-- Process only one exchange at a time. Used for updating statuses -->
    <bean id="oneExchangeThrottlePolicy" class="org.apache.camel.impl.ThrottlingInflightRoutePolicy">
        <property name="maxInflightExchanges" value="1"/>
        <property name="scope" value="Route"/>
    </bean>

Route Definition

Code Block
<route id="dataSetFileStatusUpdate" routePolicyRef="oneExchangeThrottlePolicy">

^{Log Snippet:}
^{10:24:22.709 | INFO | Camel (camel-1) thread #13 -}^{file:///home/customer/irm-emea-dev/filearea/inbound}^{| dataSetFileStatusUpdate | ID-linux-wbx3-1601540648191-0-1 | o.a.c.i.ThrottlingInflightRoutePolicy | Throttling consumer: 2 > 1 inflight exchange by suspending consumer: Consumer[direct://dataSetFileStatusUpdate]}
^{10:24:22.709 | INFO | Camel (camel-1) thread #13 -}^{file:///home/customer/irm-emea-dev/filearea/inbound}^{| dataSetFileStatusUpdate | ID-linux-wbx3-1601540648191-0-1 | o.a.c.i.ThrottlingInflightRoutePolicy | Throttling consumer: 1 <= 1 inflight exchange by resuming consumer: Consumer[direct://dataSetFileStatusUpdate]}

onComplete behaviour when calling one route multiple times

OnComplete block is executed at the end of a parent route if used in a child route using the latest exchange. If the same route is called multiple times from a parent route, the onComplete block is not executed at the end of the child route but after all calls are finished. See the comments in the code sample:

Code Block

        <!-- Price Condition 006 -->
        <route id="exportPriceConditionA006Regions">
            <from uri="direct:exportPriceConditionsA006Regions"/>
                <!-- EMEA -->
                    <log message="Exporting EMEA Region"/>
                    <!-- header for a dynamic over Sales Orgs -->
                    <setHeader headerName="Country">
                        <constant>EMEA</constant>
                    </setHeader>
                    <setHeader headerName="sftpOutputFolder">
                        <constant>INF0070</constant>
                    </setHeader>
                    <to uri="direct-vm:fetchSalesOrg"/>
                    <to uri="direct:exportPriceConditionsA006"/>
                <!-- Russia -->
                    <log message="Exporting Russia Region"/>
                    <!-- header for a dynamic over Sales Orgs -->
                    <setHeader headerName="Country">
                        <constant>RU</constant>
                    </setHeader>
                    <setHeader headerName="sftpOutputFolder">
                        <constant>INF0068-1</constant>
                    </setHeader>
                    <to uri="direct-vm:fetchSalesOrg"/>
                    <to uri="direct:exportPriceConditionsA006"/>
                <!-- North America -->
                    <log message="Exporting North America Region"/>
                    <!-- header for a dynamic over Sales Orgs -->
                    <setHeader headerName="Country">
                        <constant>US</constant>
                    </setHeader>
                    <setHeader headerName="sftpOutputFolder">
                        <constant>INF0068</constant>
                    </setHeader>
                    <to uri="direct-vm:fetchSalesOrg"/>
                    <to uri="direct:exportPriceConditionsA006"/>
                <!-- all three onComplete will run now with headers etc. setup by the last call -->
        </route>
        <route id="exportPriceConditionA006">
            <!--      <from uri="timer://exportPriceConditionA004?repeatCount=1"/>-->
            <from uri="direct:exportPriceConditionsA006"/>
            <log loggingLevel="INFO" message="Sales Orgs: ${header.salesOrgsList}"/>
            <!-- set the export file name -->
            <setHeader headerName="CamelFileNameOnly">
                <simple>PC_A006_DT_${date:now:yyyyMMdd_HHmmssSSS}.csv</simple>
            </setHeader>
            <setHeader headerName="CamelFileName">
                <simple>{{sap-price-condition-export.folder}}/${header.CamelFileNameOnly}</simple>
            </setHeader>
            <log loggingLevel="INFO" message="Starting export of the price condition to ${header.CamelFileName}"/>
            <!-- fetches the data set row by its status -->
            <to uri="pfx-api:fetch?objectType=LTV&pricingParameterName=A006_PricingConditions&filter=priceConditionsA006Filter"/>
            <!-- save number of rows for journal -->
            <setHeader headerName="exportedRows">
                <simple>${header.totalRows}</simple>
            </setHeader>
            <setHeader headerName="csvHeader">
                <constant>Condition type,Sales Org.,Distr. Channel,Price List,Document Currency,Material,Amount,Unit,Condition Pricing Unit,UoM,Valid From,Valid To,Scale Quantity,Scale Amount</constant>
            </setHeader>
            <!-- group by column for Scale Quantity Processor -->
            <setProperty propertyName="scaleQuantityKeyName">
                <constant>Material</constant>
            </setProperty>
            <filter>
                <simple>${header.exportedRows} > 0</simple>
                <multicast parallelProcessing="false" stopOnException="true">
                    <to uri="direct:exportPriceConditionA006ToFile"/>
                    <to uri="direct:exportPriceConditionA006ToDS"/>
                    <to uri="direct:exportPriceConditionA006MarkExported"/>
                </multicast>
            </filter>
            <log loggingLevel="INFO" message="Processing completed. Exported ${header.exportedRows} rows."/>
            <onCompletion onCompleteOnly="true">
                <!-- only if we found rows to export -->
                <choice>
                    <when>
                        <simple>${header.exportedRows} > 0</simple>
                        <!-- journal the export -->
                        <wireTap uri="direct-vm:journalExportedFile"/>
                        <!-- transfer the file to the sftp -->
                        <to uri="direct:exportPriceConditionsToSFTP"/>
                    </when>
                    <otherwise>
                    </otherwise>
                </choice>
            </onCompletion>
        </route>

How to Copy File(s) with SCP from One Server to Another via Proxy Jump

Since there is 2FA SSH in use, you no longer can easily copy a file from one server to another via SCP.

...

Now you have the required file from the PROD server in the /tmp folder of the QA server.

How to Remove IM from Server (with Refresh of Monitoring Tool)

Code Block

root@server-a.pricefx.net ~ # salt-call grains.get im
local:
    ----------
    installed_ims:
        - im-customer1-prod
        - im-customer1-qa
        - im-customer2-qa
        - im-customer1-dev
        - im-customr2-prod
        - im-customer3-prod
    installed_pies:
    running_ims:
        - im-customer1-prod
        - im-customer1-qa
        - im-customer2-qa
        - im-customer1-dev
        - im-customr2-prod
        - im-customer3-prod
    running_pies:
root@server-a.pricefx.net ~ # systemctl stop im-customer3-prod
root@int1.us-vh.pricefx.net ~ # systemctl disable im-customer3-prod
im-customer3-prod.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable im-customer3-prod
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_ADDRESS = "cs_CZ.UTF-8",
LC_NAME = "cs_CZ.UTF-8",
LC_MONETARY = "cs_CZ.UTF-8",
LC_PAPER = "cs_CZ.UTF-8",
LC_IDENTIFICATION = "cs_CZ.UTF-8",
LC_TELEPHONE = "cs_CZ.UTF-8",
LC_MEASUREMENT = "cs_CZ.UTF-8",
LC_TIME = "cs_CZ.UTF-8",
LC_NUMERIC = "cs_CZ.UTF-8",
LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_ADDRESS = "cs_CZ.UTF-8",
LC_NAME = "cs_CZ.UTF-8",
LC_MONETARY = "cs_CZ.UTF-8",
LC_PAPER = "cs_CZ.UTF-8",
LC_IDENTIFICATION = "cs_CZ.UTF-8",
LC_TELEPHONE = "cs_CZ.UTF-8",
LC_MEASUREMENT = "cs_CZ.UTF-8",
LC_TIME = "cs_CZ.UTF-8",
LC_NUMERIC = "cs_CZ.UTF-8",
LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
root@server-a.pricefx.net ~ # unlink /etc/init.d/im-customer3-prod
root@server-a.pricefx.net ~ # systemctl daemon-reload
root@server-a.pricefx.net ~ # systemctl reset-failed
root@server-a.pricefx.net ~ # salt-call saltutil.refresh_grains
local:
    True
root@server-a.pricefx.net ~ # salt-call grains.get im
local:
    ----------
    installed_ims:
        - im-customer1-prod
        - im-customer1-qa
        - im-customer2-qa
        - im-customer1-dev
        - im-customr2-prod
    installed_pies:
    running_ims:
        - im-customer1-prod
        - im-customer1-qa
        - im-customer2-qa
        - im-customer1-dev
        - im-customr2-prod
    running_pies:
root@server-a.pricefx.net ~ #
Removed im-customer3-prod from configuration.py -> It will be replaced by command execution "salt-call state.apply im.monitoring" with next relase of monitoring tool.

How to Send a Message to /dev/null

Sometimes you need to throw away a message to get a flow going. It can be achieved by logging to an empty channel. Camel will mark the message as consumed and go to another one.

Empty log

Code Block

                <choice>
                <when>
                    <simple>${header.exportedRows} > 0</simple>
                    <wireTap uri="direct-vm:journalExportedFile"/>
                    <!-- transfer the file to the sftp -->
                    <to uri="direct:exportCreditMemoToSFTP"/>
                </when>
                    <otherwise>
                        <to uri="log:dev.null?level=OFF"/>
                    </otherwise>
                </choice>

Correct setup for LOG and ELK in IM 1.3.x and newer

Many integrations might have a wrong setup. If you migrated to Camel 3.5 (IM 1.3.x and newer), you need to do these steps:

Remove the logback-spring.xml file from your IM project.
IM has its own configuration file. The main.log file will be created in the new subfolder "logs" in the IM instance folder. Monitoring is available there.
Remove these properties from the property files. These properties are not needed.
Code Block
logging.file.name=main.log integration.logging.file=main.log
Add these new properties into the property files. It is configuration for ELK if a config server is not used.
Code Block
integration.logstash.enabled=true integration.logstash.address=elkint.pricefx.eu:4560

How to set Constructor Parameter for Bean, e.g. Converter

Some values are not available as properties. For example pattern for DecimalToString can only be set as a constructor parameter.

Code Block
<bean id="decimalToString" class="net.pricefx.integration.mapper.converter.DecimalToString"> <constructor-arg name="pattern" value="#.###############################################"/> </bean>

How to access Spring properties in Simple blocks

Here is an example how to log a property set in a property file:

Code Block

Property file:
bosch-rexroth.initial-load-master-data-batch-size=50000
Route:
<log message="Running batch number# ${exchangeProperty.CamelSplitIndex}, batch size ${properties:bosch-rexroth.initial-load-master-data-batch-size} for file ${header.CamelFileNameOnly}"/>

Manual Integration that uses event processing is filled with "perEventTypeEventRouteInputEventRoutePADATALOAD_COMPLETED" messages

Under Loggers, search for net.pricefx.integration.api.PriceFxExceptionTranslator and change the log level to WARN.

Provisioned instance fails on no spring.config.import property has been defined

If you had a provisioned AWS instance on version 3.7.0 and below, you might encounter this problem when you switch your project to Custom Image build. This is caused by missing dependency in pom.xml. Just add this dependency to pom.xml to resolve the issue:

...

Page Comparison

Versions Compared

Old Version 2

New Version Current

Key

Camel choice

Warning when calling calculation logic in PFX without sending any data to it

Note about loadMapper

Access body in criterion

Concatenation of several input values in mapper to final out field

Fetch QuoteLineItems

Cleaning up data feed

PADATALOAD_COMPLETED event

Exporting price lists – mapping by label instead of name

Price List export

Setting TAB character as CSV delimiter

Saving or updating quote

Retrieving email list from user group

fetch User Group by Id

Exporting to CSV file with duplicating header column name

Export header

Adding CFS calculation while processing

Creating price list from IM

Fetching data with BatchModed=true

Web crawling - Metoda

Loading data to Data Source directly

No events generated

Starting LPG calculation

Preventing from being overridden by null value

Working with IDOC format

Working with JSON – jsonpath component

Persistent message store in Camel

Deleting approved contact

Using new components in IM 1.1.7

Merging CSV columns into one PFX attribute

Calling LPG recalculate in IM

SAP iDOC support in IM

Using dsLoad with key predefined in PFX

Using multiple PFX connections in single IM project

PRX Connections in camel-context.xml

Route

Working with big ZIP file larger than 4GB

Samples of reading and writing data in Product Extension table

A library of e-books

Defining Table Name and Keys for PX or CX Master Data

loadMapper

Key definition

Saving a flag into PFX Advanced configuration

Creating a REST endpoint to call your route

REST endpoint

The proxy route

The target route

Changing the heap size for the IntegrationManager in a dedicated environment

Heap Size

RAM

GUnzip processor for bigger files

GUnzipProcessor.java

How to use constants and reusable code snippets

Making a route to be a singleton

Route

Formatting date when exporting a price list

Date Formatting

Getting output of a formula logic

Custom file processing strategy driven by a signal file

Server health check warning

How to fetch mapping of attributes to labels for Customer, Products and Extensions

Filter

Response

How to set old IM <1.1.17 to work with openJDK 11 and newer

How to disable IM auto-registration into PlatformManager

How to process CUSTOM events in IM

Using Header/Property in groovy exchange loadMapper

Understanding Imports and Includes in WSDL Files

Truncating Only Flushed Rows from Data Feed

Filter and truncate

PX Columns Character Size Limits

Event Types

pfx-api:unmarshal does not propagate an exception outside of a split

Split

Avoid Concurrent Updates on a Row