Second Iteration – Structure and Performance

The goal of this iteration is to give the logic a better structure, so it follows a best-practice pattern and is easier to read by others. We will also add some performance improvements.

Best-practice Logic Structure

This is the general structure of a logic which follows the best practices. The rules are not strict but dividing the logic into multiple logical parts helps the readability.

Element Name/Prefix

Description

Example

Global

Element with the api.retailGlobal statement and possible api.global and api.local field initialization.

-

InputXXX

Input elements starting with the Input prefix.

InputCountry, InputDiscount, etc.

AbortIfInputGeneration

Special element with the input generation statement. The code after this element is not executed in the input generation mode. See also /wiki/spaces/KB/pages/784957624.

-

GetXXX

Elements with the Get prefix are meant to fetch and pre-process data.

GetCostData, GetCountryData, GetMargin

-

All other elements go here.

ProductCost, BasePrice, FinalPrice

Errors

It is a good practice to provide the users with some kind of indication if something went wrong (e.g., in a neatly formatted ResultMatrix).

-

We will update the structure of the logic according to this best practice in the following way:

Element Name

Description

Global

This stays the first element of the logic.

GetCountryData

Here we fetch and cache the information about countries.

Note: Contrary to the best-practice structure we place this element before the InputCountry element, but that is because we need the country data to populate the drop-down with countries. Therefore it is okay.

InputCountry

Generates the user input.

AbortIfInputGeneration

Aborts the calculation in the input generation mode.

GetCostData

Fetches and processes the data related to costs.

ProductCost

Calculates the product cost.

ProductGroupAverageCost

Displays the data calculated in GetCostData.

AverageCostDiffPct

Calculates the average cost difference.

MarginAdjustment

Fetches the margin adjustment for the product group.

BasePrice

Calculates the base price.

CountryAdjustedBasePrice

Adjusts the base price with the country factor.

Errors

Displays the errors.

Global Cache Pattern, Caveats of the Find Method

Country Element

Let’s divide the Country element into the data-gathering part GetCountryData and the input part InputCountry as described above. And also let’s add some global cache.

Before

Country
// fetch the country factors
def countries = api.findLookupTableValues("CountryFactor")

// store the factors in the api.local space for further re-use
api.local.countries = countries

// transform the countries in a map so we can use it as labels for the drop-down
// and sort the results by the label alphabetically
def labels = countries.collectEntries {
  [it.key1, it.attribute2]
}.sort {
  it.getValue()
}

options = labels.keySet().toList()

// create the user entry
def selectedCountry = api.option("Country", options, labels)

// add input validation and set a default value if the input is not specified
if (!selectedCountry) {
  api.yellowAlert("Country not selected, defaulting to AU")
  api.local.errors << "Country not selected, defaulting to AU"
  selectedCountry = "AU"
}

return selectedCountry

After

GetCountryData
// do this only once during the PL generation
if (api.global.countries == null) {

  // fetch the country factors
  def countries = api.findLookupTableValues("CountryFactor")

  // transform the countries in a map so we can use it as labels for the drop-down
  // and sort the results by the label alphabetically
  def labels = countries.collectEntries {
    [it.key1, it.attribute2]
  }.sort {
    it.getValue()
  }

  def options = labels.keySet().toList()

  // cache the values
  api.global.countries = countries
  api.global.labels = labels
  api.global.options = options
}

return null
InputCountry
// create the user entry
def selectedCountry = api.option("Country", api.global.options, api.global.labels)

// set default value if entry not specified
if (!selectedCountry) {
  api.yellowAlert("Country not selected, defaulting to AU")
  selectedCountry = "AU"
}

return selectedCountry

Step-by-step Analysis

if (api.global.countries == null) {
	...
	api.global.countries = countries
}

This is a caching pattern which ensures the block of code within the if-statement is run only once per the price list generation. For the line items other than the first one api.global.countries will be non-null and therefore the block will be skipped.

You need to set api.retainGlobal = true in the first element of the logic as shown earlier.


api.global.countries = countries
api.global.labels = labels
api.global.options = options

Here we cache the list of countries, the labels and options in the api.global space. This is done only once (when the first line item is processed) and we can re-use them later in the new InputCountry element.

def selectedCountry = api.option("Country", api.global.options, api.global.labels)

Here we re-use the previously cached options and labels for the country drop-down. Note this code is run for every line item.

ProductCost Element

Let’s give this element a small performance boost by filtering on the Valid From date directly in the database.

Before

ProductCost
def sku = api.product("sku")

// get the current or target date
def targetDate = api.calendar().format("yyyy-MM-dd")

// fetch the costs
// we could also do the filter on targetDate here but I couldn't demonstate the usage of a closure later
def data = api.productExtension("Product_Costs")

// find only records where the validity date is lower than the target date
def validRecords = data.findAll{ it.attribute4 <= targetDate }

if (!validRecords) {
  api.redAlert("Cost not found for $sku")
  api.local.errors << "Cost not found for $sku"
  return null
}

// sort the costs by date in descending order and pick the most recent one
validRecords = validRecords.sort { a, b ->
  return b.attribute4 <=> a.attribute4;
}

return validRecords[0].attribute1.toBigDecimal()

After

ProductCost
def sku = api.product("sku")

// get the current or target date
def targetDate = api.calendar().format("yyyy-MM-dd")

// set up the filters
def filters = [
    Filter.equal("name", "Product_Costs"), // PX name
    Filter.equal("sku", sku), // Sku
    Filter.lessOrEqual("attribute4", targetDate) // Cost Date
]

// fetch the latest cost from the PX
// use the expansion operator to expand a list into a varargs array
def data = api.find("PX", 0, 1, "-attribute4", ["attribute1"], *filters)
if (!data) {
  api.redAlert("Cost not found for $sku")
  api.local.errors << "Cost not found for $sku"
  return 0.0
}

return data[0].attribute1?.toBigDecimal()

Step-by-step Analysis

// set up the filters
def filters = [
	Filter.equal("name", "Product_Costs"), // PX name
	Filter.equal("sku", sku), // Sku
	Filter.lessOrEqual("attribute4", targetDate) // Cost Date
]

// fetch the latest cost from the PX
// use the expansion operator to expand a list into a varargs array
def data = api.find("PX", 0, 1, "-attribute4", ["attribute1"], *filters)

Instead of fetching all product costs related to the product and then filtering the ones valid on the target date, we use api.find with a sortBy parameter. Because we filter only records which are less or equal to the target date and we sort by the Valid From date (attribute4) in the descending order (that is why there is a minus), we will get just what we want.

  • When using api.find, only use the sortBy parameter if you have a good reason for it. It has a negative impact on performance since it implies an extra sorting operation.
  • Always specify the list of fields returned by the api.find method. Only in very rare cases you will need all the fields. But in majority of the cases you will save a lot of bandwidth and therefore time when you only specify the fields you need.
  • When you do not specify the limits of api.find because you expect to load a potentially big number of records, use api.stream instead. It has some overhead cost but for large records it is quite fast.

Data Aggregation on the Database Level

ProductGroupAverageCost Element

We will divide this element into two parts. The first will gather the data and calculate the result, the other will just display the result.

We will therefore move everything from the ProductGroupAverageCost element into the new GetCostData element and make it more performance effective by adding the global cache pattern. Then we display the result in the original element.

Before

ProductGroupAverageCost
// let's create a reference to the library so the code is better readable
// especially useful when there are multiple libraries and/or long names
def Math = libs.DemoLib.MathUtil

def pg = api.product("Product Group")

// fetch the costs from the PX
// notice using the Product Master attribute to filter the PX
def cit = api.stream("PX", null,
    Filter.equal("name", "Product_Costs"),  // PX Name
    Filter.equal("ProductMaster__attribute2", pg)) // Product Group
def costs = cit.collect{ it }
cit.close()

if (!costs) {
  api.local.errors << "Missing costs for group $pg"
  return null
}

def avgCost = costs.sum{ it.attribute1.toBigDecimal() } / costs.size()

return Math.round(avgCost, 4)

After

GetCostData
// let's create a reference to the library so the code is better readable
// especially useful when there are multiple libraries and/or long names
def Math = libs.DemoLib.MathUtil

// calculate the average costs only once and cache the result
if (api.global.productGroupAverageCosts == null) {

  api.global.productGroupAverageCosts = [:]

  // fetch the distinct product groups (there can be potentially a lot of them)
  def productGroupsIt = api.stream("P", "sku", ["attribute2"], true)
  def productGroups = productGroupsIt.collect { it.attribute2 }
  productGroupsIt.close()

  // remove empty product group
  productGroups -= null

  // for each product group except for empty, calculate the average cost
  for (pg in productGroups) {

    // fetch the costs from the PX
    // notice using the Product Master attribute to filter the PX
    def avgCosts = api.find("PX", 0, 1, "sku", ["attribute1": "AVG"], false,
        Filter.equal("name", "Product_Costs"), // PX Name
        Filter.equal("ProductMaster__attribute2", pg)) // Product Group

    // extract the average cost
    def avgCost = avgCosts ? avgCosts[0].attribute1.toBigDecimal() : null
    if (avgCost == null) {
      api.local.errors << "Missing costs for group $pg"
      continue
    }

    // store the result in the Global cache
    api.global.productGroupAverageCosts[pg] = Math.round(avgCost, 4)

  }
}
ProductGroupAverageCost
// use the cached result on subsequent runs
def productGroup = api.product("Product Group")
return api.global.productGroupAverageCosts[productGroup] ?: null

Step-by-step Analysis

if (api.global.productGroupAverageCosts == null) {

api.global.productGroupAverageCosts = [:]

... rest of the code
}

Ideally we want to run the average group cost calculation only once for every group and then re-use it. This caching pattern ensures the inner block of code is run only once.

def productGroupsIt = api.stream("P", "sku", ["attribute2"], true)
def productGroups = productGroupsIt.collect { it.attribute2 }
productGroupsIt.close()

This code opens a stream to the Product Master and fetches only distinct product groups thanks to the last parameter set to true. We sort the results by sku because we do not really care. Sorting by attribute2 would be perfectly fine, but sku is ensured to be indexed, therefore it might be faster.

Thanks to Groovy we can call the collect closure directly on the stream iterator and collect the distinct group names. Don’t forget to close it after you’re done.

productGroups -= null

Because there might be products without a product group, the previous call could return also null as one of the product groups. Here we remove it from the list. This syntax is equivalent to calling productGroups.remove(null).

def avgCosts = api.find("PX", 0, 1, "sku", ["attribute1": "AVG"], false,
	Filter.equal("name", "Product_Costs"), // PX Name
	Filter.equal("ProductMaster__attribute2", pg)) // Product Group

This overload of api.find allows us to specify an aggregation on the database level. We want to get an average cost of all products of a group pg. We do that by providing a map of fields instead of the list, and an aggregation function ["attribute1": "AVG"]. Since we only aggregate one group, we limit the result to only one line.

There is currently a limitation that there is no equivalent to the group-by clause we are used to in SQL. The method aggregates all the data. Also, we cannot combine normal and aggregated fields. Everything is aggregated.

Aggregation works also with api.stream.


def avgCost = avgCosts ? avgCosts[0].attribute1.toBigDecimal() : null
if (avgCost == null) {
	api.local.errors << "Missing costs for group $pg"
	continue
}

// store the result in the Global cache
api.global.productGroupAverageCosts[pg] = Math.round(avgCost, 4)

Here we extract the cost from the first (and only) item returned by the previous api.find. If no cost is found, we display an error in the Errors element. Otherwise we store the average cost in the api.global.productGroupAverageCosts map with the product group pg as the key.

def productGroup = api.product("Product Group")
return api.global.productGroupAverageCosts[productGroup] ?: null

In the ProductGroupAverageCost element we only access the previously cached value from the api.global.productGroupAverageCosts field.

MarginAdjustment Element

You might also think about improving the performance of the MarginAdjustment element by caching the margins for each product group. Let’s do that.

Before

MarginAdjustment
def pg = api.product("Product Group")

def marginAdj = api.vLookup("MarginAdj", "Margin Adj", pg)
if (marginAdj == null) {
  marginAdj = 0.0
}

return marginAdj

After

MarginAdjustment
if (api.global.margins == null) {
  // if the key is not found, the map returns 0.0
  api.global.margins = [:].withDefault{ 0.0 }

  def margins = api.findLookupTableValues("MarginAdj")
  margins.inject(api.global.margins, { map, it ->
    map[it.name] = it.attribute1
    return map
  })
}

def pg = api.product("Product Group")
return api.global.margins[pg]

Step-by-step Analysis

if (api.global.margins == null) {
	// if the key is not found, the map returns 0.0
	api.global.margins = [:].withDefault{ 0.0 }
}

def pg = api.product("Product Group")
return api.global.margins[pg]

If this is the first execution of the logic on the first line item, we initialize api.global.margins with a map. Notice withDefault{ 0.0 }. If a key is not found in the map, the value 0.0 is returned instead of null.

If this is the next execution of the logic, we just retrieve the margin from the global map.

def margins = api.findLookupTableValues("MarginAdj")
margins.inject(api.global.margins, { map, it ->
	map[it.name] = it.attribute1
	return map
})

We fetch the complete contents of the MarginAdj PP. Then we use the fancy inject closure which helps us populate the previously created map with the entries. The map key is the product group name and the value is the margin percentage.


Depending on the data, this optimization is not that necessary if there are just a couple of product groups. The vLookup function is optimized and automatically uses a cache of its own. Therefore a call to the function with a given key accesses the database only once and repeated calls with the same key retrieve the value from a cache. Given we have a price list of 100,000 products but just 10 product groups, there will be just 10 accesses to the DB.


It is not necessary to cache the call to the the vLookup function because it is cached automatically on the server. Every value is accessed only once for any given key combination.

Found an issue in documentation? Write to us.