/
Split - Performance

Split - Performance

Owned by Ondřej Tesař
Jan 29, 2025
1 min read

Split is a powerful IM (Camel) feature which can be used by multiple parameters which can significantly affect speed and also memory consumption. It if necessary to be fully aware of all possible risks.

Official documentation: Split

IM 4.10.12 split usage comparison

This is a route used for our tests. We will only change parameters like streaming and parallelProcessing.

The test involves fetching 1 million records in batches of 50,000 from the PX table. The memory consumption results for this test are tracked based on different combinations of split parameters, which impact how efficiently memory is used during the data fetching process. The table mentioned reflects these memory consumption metrics and helps identify the optimal combination of parameters for performance tuning.

<route id="fetch-route" streamCache="true" xmlns="http://camel.apache.org/schema/spring"> <from uri="timer://fetchTest?repeatCount=1"/> <to uri="pfx-api:fetch?sql=select * where name='test'&amp;objectType=PX&amp;batchedMode=true&amp;batchSize=50000"/> <split> <simple>${body}</simple> <toD uri="pfx-api:fetchIterator"/> <log message="Processing"/> </split> <log message="Done"/> </route>

 

 

 

 

 

......
<split streaming="true">
<simple>${body}</simple>
<to uri="pfx-api:fetchIterator"/>
<log message="Processing"/>
</split>
<log message="Done"/>
.......

Used memory: 137 MB

<split streaming="true">
<simple>${body}</simple>
<to uri="pfx-api:fetchIterator"/>
<log message="Processing"/>
<setBody>
<constant></constant>
</setBody>
</split>

Used memory: 286 MB

......
<split>
<simple>${body}</simple>
<to uri="pfx-api:fetchIterator"/>
<log message="Processing"/>
<setBody>
<constant></constant>
</setBody>
</split>
<log message="Done"/>
.......

Used memory: 383 MB

<split streaming="true" parallelProcessing="true">
<simple>${body}</simple>
<to uri="pfx-api:fetchIterator"/>
<log message="Processing"/>
<setBody>
<constant></constant>
</setBody>
</split>

Used memory: 883 MB

......
<split streaming="true" parallelProcessing="true">
<simple>${body}</simple>
<to uri="pfx-api:fetchIterator"/>
<log message="Processing"/>
</split>
<log message="Done"/>
.......

Used memory: 978 MB

......
<split>
<simple>${body}</simple>
<to uri="pfx-api:fetchIterator"/>
<log message="Processing"/>
</split>
<log message="Done"/>
.......

Used memory: 1,2 GB

 

IntegrationManager version 6.0.0