Fetch with Pages

To fetch larger data set, a loop is required to fetch all records if the number of records is larger than the page size. You can either increment the page number or add a criteria “business key is larger than business key of the last record”.

In the above example, a data process shape with a custom Groovy script is used to update the ‘customerId’ dynamic process property which is used by the filter request message in the next shape.

import java.util.Properties;
import java.io.InputStream;
import com.boomi.execution.ExecutionUtil;

InputStream inputStream = dataContext.getStream(0);
def customerIds = inputStream.getText("UTF8").split("''")
def customerId = customerIds[customerIds.length-1].replace("'","")
ExecutionUtil.setDynamicProcessProperty("customerId", customerId, true);

for( int i = 0; i < dataContext.getDataCount(); i++ ) {
    InputStream is = dataContext.getStream(i);
    Properties props = dataContext.getProperties(i);

    dataContext.storeStream(is, props);
}

The filter request message shape constructs the fetch request message with one more criteria “customerId > last customer Id” (parameter {1} calculated from the previous data process shape). This is to make sure the fetched records are different than on the previous page.

'{
  "data": {
   
  "criteria" : [
    {
        "fieldName" : "customerId",
        "operator" : "notNull"
      },
      {
        "fieldName" : "customerId",
        "operator" : "greaterThan",
        "value" : "'{1}'"
      }
  ],
  "operator" : "and"
  },
"sortBy":["customerId"]
}'

The decision shape compares the number of records returned with the page size. If the number of records returned is smaller than the page size, that means no further fetch request is required as it is already the last page.