Second Iteration – Structure and Performance
The goal of this iteration is to give the logic a better structure, so it follows a best-practice pattern and is easier to read by others. We will also add some performance improvements.
Best-practice Logic Structure
This is the general structure of a logic which follows the best practices. The rules are not strict but dividing the logic into multiple logical parts helps the readability.
Element Name/Prefix | Description | Example |
---|---|---|
Global | Element with the | - |
InputXXX | Input elements starting with the Input prefix. | InputCountry, InputDiscount, etc. |
AbortIfInputGeneration | Special element with the input generation statement. The code after this element is not executed in the input generation mode. See also /wiki/spaces/KB/pages/784957624. | - |
GetXXX | Elements with the Get prefix are meant to fetch and pre-process data. | GetCostData, GetCountryData, GetMargin |
- | All other elements go here. | ProductCost, BasePrice, FinalPrice |
Errors | It is a good practice to provide the users with some kind of indication if something went wrong (e.g., in a neatly formatted ResultMatrix). | - |
We will update the structure of the logic according to this best practice in the following way:
Element Name | Description |
---|---|
Global | This stays the first element of the logic. |
GetCountryData | Here we fetch and cache the information about countries. Note: Contrary to the best-practice structure we place this element before the InputCountry element, but that is because we need the country data to populate the drop-down with countries. Therefore it is okay. |
InputCountry | Generates the user input. |
AbortIfInputGeneration | Aborts the calculation in the input generation mode. |
GetCostData | Fetches and processes the data related to costs. |
ProductCost | Calculates the product cost. |
ProductGroupAverageCost | Displays the data calculated in GetCostData. |
AverageCostDiffPct | Calculates the average cost difference. |
MarginAdjustment | Fetches the margin adjustment for the product group. |
BasePrice | Calculates the base price. |
CountryAdjustedBasePrice | Adjusts the base price with the country factor. |
Errors | Displays the errors. |
Global Cache Pattern, Caveats of the Find Method
Country Element
Let’s divide the Country element into the data-gathering part GetCountryData and the input part InputCountry as described above. And also let’s add some global cache.
Before
// fetch the country factors def countries = api.findLookupTableValues("CountryFactor") // store the factors in the api.local space for further re-use api.local.countries = countries // transform the countries in a map so we can use it as labels for the drop-down // and sort the results by the label alphabetically def labels = countries.collectEntries { [it.key1, it.attribute2] }.sort { it.getValue() } options = labels.keySet().toList() // create the user entry def selectedCountry = api.option("Country", options, labels) // add input validation and set a default value if the input is not specified if (!selectedCountry) { api.yellowAlert("Country not selected, defaulting to AU") api.local.errors << "Country not selected, defaulting to AU" selectedCountry = "AU" } return selectedCountry
After
// do this only once during the PL generation if (api.global.countries == null) { // fetch the country factors def countries = api.findLookupTableValues("CountryFactor") // transform the countries in a map so we can use it as labels for the drop-down // and sort the results by the label alphabetically def labels = countries.collectEntries { [it.key1, it.attribute2] }.sort { it.getValue() } def options = labels.keySet().toList() // cache the values api.global.countries = countries api.global.labels = labels api.global.options = options } return null
// create the user entry def selectedCountry = api.option("Country", api.global.options, api.global.labels) // set default value if entry not specified if (!selectedCountry) { api.yellowAlert("Country not selected, defaulting to AU") selectedCountry = "AU" } return selectedCountry
Step-by-step Analysis
if (api.global.countries == null) { ... api.global.countries = countries }
This is a caching pattern which ensures the block of code within the if-statement is run only once per the price list generation. For the line items other than the first one api.global.countries
will be non-null and therefore the block will be skipped.
You need to set api.retainGlobal = true
in the first element of the logic as shown earlier.
api.global.countries = countries api.global.labels = labels api.global.options = options
Here we cache the list of countries, the labels and options in the api.global
space. This is done only once (when the first line item is processed) and we can re-use them later in the new InputCountry element.
def selectedCountry = api.option("Country", api.global.options, api.global.labels)
Here we re-use the previously cached options and labels for the country drop-down. Note this code is run for every line item.
ProductCost Element
Let’s give this element a small performance boost by filtering on the Valid From date directly in the database.
Before
def sku = api.product("sku") // get the current or target date def targetDate = api.calendar().format("yyyy-MM-dd") // fetch the costs // we could also do the filter on targetDate here but I couldn't demonstate the usage of a closure later def data = api.productExtension("Product_Costs") // find only records where the validity date is lower than the target date def validRecords = data.findAll{ it.attribute4 <= targetDate } if (!validRecords) { api.redAlert("Cost not found for $sku") api.local.errors << "Cost not found for $sku" return null } // sort the costs by date in descending order and pick the most recent one validRecords = validRecords.sort { a, b -> return b.attribute4 <=> a.attribute4; } return validRecords[0].attribute1.toBigDecimal()
After
def sku = api.product("sku") // get the current or target date def targetDate = api.calendar().format("yyyy-MM-dd") // set up the filters def filters = [ Filter.equal("name", "Product_Costs"), // PX name Filter.equal("sku", sku), // Sku Filter.lessOrEqual("attribute4", targetDate) // Cost Date ] // fetch the latest cost from the PX // use the expansion operator to expand a list into a varargs array def data = api.find("PX", 0, 1, "-attribute4", ["attribute1"], *filters) if (!data) { api.redAlert("Cost not found for $sku") api.local.errors << "Cost not found for $sku" return 0.0 } return data[0].attribute1?.toBigDecimal()
Step-by-step Analysis
// set up the filters def filters = [ Filter.equal("name", "Product_Costs"), // PX name Filter.equal("sku", sku), // Sku Filter.lessOrEqual("attribute4", targetDate) // Cost Date ] // fetch the latest cost from the PX // use the expansion operator to expand a list into a varargs array def data = api.find("PX", 0, 1, "-attribute4", ["attribute1"], *filters)
Instead of fetching all product costs related to the product and then filtering the ones valid on the target date, we use api.find
with a sortBy
parameter. Because we filter only records which are less or equal to the target date and we sort by the Valid From date (attribute4) in the descending order (that is why there is a minus), we will get just what we want.
- When using
api.find
, only use thesortBy
parameter if you have a good reason for it. It has a negative impact on performance since it implies an extra sorting operation. - Always specify the list of fields returned by the
api.find
method. Only in very rare cases you will need all the fields. But in majority of the cases you will save a lot of bandwidth and therefore time when you only specify the fields you need. - When you do not specify the limits of
api.find
because you expect to load a potentially big number of records, useapi.stream
instead. It has some overhead cost but for large records it is quite fast.
Data Aggregation on the Database Level
ProductGroupAverageCost Element
We will divide this element into two parts. The first will gather the data and calculate the result, the other will just display the result.
We will therefore move everything from the ProductGroupAverageCost element into the new GetCostData element and make it more performance effective by adding the global cache pattern. Then we display the result in the original element.
Before
// let's create a reference to the library so the code is better readable // especially useful when there are multiple libraries and/or long names def Math = libs.DemoLib.MathUtil def pg = api.product("Product Group") // fetch the costs from the PX // notice using the Product Master attribute to filter the PX def cit = api.stream("PX", null, Filter.equal("name", "Product_Costs"), // PX Name Filter.equal("ProductMaster__attribute2", pg)) // Product Group def costs = cit.collect{ it } cit.close() if (!costs) { api.local.errors << "Missing costs for group $pg" return null } def avgCost = costs.sum{ it.attribute1.toBigDecimal() } / costs.size() return Math.round(avgCost, 4)
After
// let's create a reference to the library so the code is better readable // especially useful when there are multiple libraries and/or long names def Math = libs.DemoLib.MathUtil // calculate the average costs only once and cache the result if (api.global.productGroupAverageCosts == null) { api.global.productGroupAverageCosts = [:] // fetch the distinct product groups (there can be potentially a lot of them) def productGroupsIt = api.stream("P", "sku", ["attribute2"], true) def productGroups = productGroupsIt.collect { it.attribute2 } productGroupsIt.close() // remove empty product group productGroups -= null // for each product group except for empty, calculate the average cost for (pg in productGroups) { // fetch the costs from the PX // notice using the Product Master attribute to filter the PX def avgCosts = api.find("PX", 0, 1, "sku", ["attribute1": "AVG"], false, Filter.equal("name", "Product_Costs"), // PX Name Filter.equal("ProductMaster__attribute2", pg)) // Product Group // extract the average cost def avgCost = avgCosts ? avgCosts[0].attribute1.toBigDecimal() : null if (avgCost == null) { api.local.errors << "Missing costs for group $pg" continue } // store the result in the Global cache api.global.productGroupAverageCosts[pg] = Math.round(avgCost, 4) } }
// use the cached result on subsequent runs def productGroup = api.product("Product Group") return api.global.productGroupAverageCosts[productGroup] ?: null
Step-by-step Analysis
if (api.global.productGroupAverageCosts == null) { api.global.productGroupAverageCosts = [:] ... rest of the code }
Ideally we want to run the average group cost calculation only once for every group and then re-use it. This caching pattern ensures the inner block of code is run only once.
def productGroupsIt = api.stream("P", "sku", ["attribute2"], true) def productGroups = productGroupsIt.collect { it.attribute2 } productGroupsIt.close()
This code opens a stream to the Product Master and fetches only distinct product groups thanks to the last parameter set to true
. We sort the results by sku
because we do not really care. Sorting by attribute2
would be perfectly fine, but sku
is ensured to be indexed, therefore it might be faster.
Thanks to Groovy we can call the collect
closure directly on the stream iterator and collect the distinct group names. Don’t forget to close it after you’re done.
productGroups -= null
Because there might be products without a product group, the previous call could return also null as one of the product groups. Here we remove it from the list. This syntax is equivalent to calling productGroups.remove(null)
.
def avgCosts = api.find("PX", 0, 1, "sku", ["attribute1": "AVG"], false, Filter.equal("name", "Product_Costs"), // PX Name Filter.equal("ProductMaster__attribute2", pg)) // Product Group
This overload of api.find
allows us to specify an aggregation on the database level. We want to get an average cost of all products of a group pg
. We do that by providing a map of fields instead of the list, and an aggregation function ["attribute1": "AVG"]
. Since we only aggregate one group, we limit the result to only one line.
There is currently a limitation that there is no equivalent to the group-by clause we are used to in SQL. The method aggregates all the data. Also, we cannot combine normal and aggregated fields. Everything is aggregated.
Aggregation works also with api.stream
.
def avgCost = avgCosts ? avgCosts[0].attribute1.toBigDecimal() : null if (avgCost == null) { api.local.errors << "Missing costs for group $pg" continue } // store the result in the Global cache api.global.productGroupAverageCosts[pg] = Math.round(avgCost, 4)
Here we extract the cost from the first (and only) item returned by the previous api.find
. If no cost is found, we display an error in the Errors element. Otherwise we store the average cost in the api.global.productGroupAverageCosts
map with the product group pg
as the key.
def productGroup = api.product("Product Group") return api.global.productGroupAverageCosts[productGroup] ?: null
In the ProductGroupAverageCost element we only access the previously cached value from the api.global.productGroupAverageCosts
field.
MarginAdjustment Element
You might also think about improving the performance of the MarginAdjustment element by caching the margins for each product group. Let’s do that.
Before
def pg = api.product("Product Group") def marginAdj = api.vLookup("MarginAdj", "Margin Adj", pg) if (marginAdj == null) { marginAdj = 0.0 } return marginAdj
After
if (api.global.margins == null) { // if the key is not found, the map returns 0.0 api.global.margins = [:].withDefault{ 0.0 } def margins = api.findLookupTableValues("MarginAdj") margins.inject(api.global.margins, { map, it -> map[it.name] = it.attribute1 return map }) } def pg = api.product("Product Group") return api.global.margins[pg]
Step-by-step Analysis
if (api.global.margins == null) { // if the key is not found, the map returns 0.0 api.global.margins = [:].withDefault{ 0.0 } } def pg = api.product("Product Group") return api.global.margins[pg]
If this is the first execution of the logic on the first line item, we initialize api.global.margins
with a map. Notice withDefault{ 0.0 }
. If a key is not found in the map, the value 0.0 is returned instead of null.
If this is the next execution of the logic, we just retrieve the margin from the global map.
def margins = api.findLookupTableValues("MarginAdj") margins.inject(api.global.margins, { map, it -> map[it.name] = it.attribute1 return map })
We fetch the complete contents of the MarginAdj PP. Then we use the fancy inject
closure which helps us populate the previously created map with the entries. The map key is the product group name and the value is the margin percentage.
Depending on the data, this optimization is not that necessary if there are just a couple of product groups. The vLookup
function is optimized and automatically uses a cache of its own. Therefore a call to the function with a given key accesses the database only once and repeated calls with the same key retrieve the value from a cache. Given we have a price list of 100,000 products but just 10 product groups, there will be just 10 accesses to the DB.
It is not necessary to cache the call to the the vLookup
function because it is cached automatically on the server. Every value is accessed only once for any given key combination.
Found an issue in documentation? Write to us.