Automated Deployment Using Pipelines
- Michal Smíšek
- Bruno Mendes
- Jana Volencová (Deactivated)
Continuous Integration or Continuous Delivery (CI/CD) are common and widely used practices in software development.
Why to Use It
Git should be the source of truth for any code deployed to QA, UAT, and PROD partitions.
When deploying a configuration and doing a QA release (from DEV) or a PROD release (from QA), it is a strongly recommended best practice to use the GitLab pipelines (or alternatively any other CI/CD tool) prior to manual deployment from Pricefx Studio. This will reduce potential sources of errors significantly.
By enabling the automated deployment it also will be ensured that the repository matches the partition 1:1. This is especially useful to avoid situations where a hotfix gets manually deployed to PROD and is never streamlined back to DEV, causing code loss and regression issues after new rollouts.
The pipelines can also be used to run unit tests (TDD4C) and code quality tools (Sonar).
How it Works
When a commit is made to QA or PROD branch (or potentially to any additional environement), the pipeline is triggered. The pipeline executes a pfxpackage command line tool that reads all the Pricefx objects in the repository and deploys them to the target partition. If the object does not exist in the target partition, it gets created and if it already exists, it gets updated. A special account (typically named “pfx.deploy”) should be created on the partition with the access of a Configuration Engineer.
The automated deployment to the DEV environment typically does not make sense because each commit would most probably overwrite the configuration work in progress.
Packaging Tool deploys the objects in a smart way, that means it compares the object with the one from the partition and deploys it only if there is a change. The objects that were not changed and therefore are out the scope are listed in the log as “skipping”.
What is Part of Scope
Deployment to QA and PROD partitions (potentially additional ones).
Creation of logics, tables, and other objects that you see in Studio can be deployed using pipelines.
What is Out of Scope (To Be Deployed Manually)
Deployment to DEV partitions. The DEV partitions are deployed by configuration engineers while developing the solutions. Setting up auto-deployment would ruin their development in progress.
.*.csvfiles: Data Files should not be kept in git. The recommended approach is to create system-required rows manually during the release or by writing a migration script using Calculated Field Sets or https://pricefx.atlassian.net/wiki/spaces/UNITY/pages/5202020991.Deletion of the objects that were removed from GIT is still to be done manually after the deployment; either directly on the partition or in the Fetch tool window in Studio.
CF_.*+Deployed.jsonfiles: Calculation Flow definitions in Deployed status should not be kept in git. The issue is that right after those objects get deployed, the calculation flow job will be triggered because the startingDate parameter will always be in the past.
How to setup
Create a user account for auto-deployment
The user account for the automated deployment with the relevant access rights needs to be created in the partition. The recommendation is to create an account named “pfx.deploy” and assign the “Configuration Engineer” role to it, the steps to do it are described below.
The configuration engineer role is part of the “Initial Security Setup” reusable component that can be downloaded directly from the PfxComponents tool window in Studio. It contains other frequently used business roles too and it is recommended to use them as a starting point for your project.
If the Configuration Engineer role is not deployed to your partition yet, then follow these steps in Studio:
Navigate to the PfxComponents tool window
Select “Initial Security Setup” component
Hit “Import” to download the component to the folder
.PfxComponents/initial-security-setup
In Studio:
Navigate to Security > Business roles in Deploy tool window
Select the “initial-security-setup” in the module/component picker.
Select the target partition
Select the “Configuration Engineer” role
Click Deploy
In the target partition:
Open to Administration > Access Admin > User Admin.
Add a new user named “pfx.deploy” and set some strong password.
Assign the “Configuration Engineer” business role to this user.
Now you can use this user account in your pipeline setup.
Set up the pipelines as per your GIT service
In Studio:
Right click on the project root > Pricefx > CI Deployment and select the desired GIT service:
Create ‘.gitlab-ci.yml’ for Gitlab, or
Create ‘.github-ci.yml’ for Github, or
Create ‘.bitbucket-pipelines.yml’ for BitBucket, or
Create ‘.azure-pipelines.yml’ for Microsoft Azure Devops
In the dialog, select the branch for each environment and partition which you want to automaticaly deploy.
Click Create 'xxx' file
Once you commit the created file to GIT, it will be picked up.
Important Information and Limitations
Primary Key Fields Changes
If there is a change in the key field structure during the release, the data gets truncated. So you should potentially back up the data first, adjust and then import. Or alternatively modify the table, but create another one using a different name.
User Groups after GO LIVE
During development, the initial configuration of user groups is designed by a Configuration Engineer. After GO LIVE the security setup is owned and maintained by the customer. So it has to be ensured that the security adjusted by the customer in the partition is not overwritten by the old user groups from Git. There is a switch “keepUserGroups” in Packaging Tool that does that.
Shifting Attributes
When shifting an attribute, e.g. “Country” from e.g. attribute5 to attribute1 (P, PP, PX, C, CX, S, SX), the deployment will fail because there is a unique check on the Label and in certain moment there would be two attribute fields attribute5 and attribute1 with the same Label (“Country”) and that is not allowed. The solution for CP tables is to inactivate the previous version. Now for P,PX,C,CX tables you can delete all attributes' meta-data and deploy from Studio to first generate a clean state and then run the pipeline.
Running Dataload Locks DS or DM Deployment
When a Data Source (DS) or Datamart (DM) schema change is being deployed and at the same time some dataload locked the DS or DM, the deployment of the schema will fail with an unknown error.
Schema Change Between Pricefx Releases
If there is a new Pricefx major release, there is a big probability that there was a field added in some object. Such a field existing on the partition is recognized as a change by Packaging Tool and therefore is deployed every time. This also pollutes the pipeline log with a lot of warning messages. So it is recommended to fetch from the target partition to Git to have the new fields in Git as well. This will help run the pipeline faster and make the log human readable again.
Troubleshooting
General Error
General error means that backend has crashed. Check the server log to find out why.
Not Authorized for Command
In case you get an error message “Not authorized for command”, make sure your deploy user account has the required user roles to manage/administer the given object.
Follow the “Create a user account for auto-deployment” section above to use the Configuration engineer business role.
Here is the list of permissions required for every entity.
Note that General Admin contains various permissions like Configuration Wizards and Message Templates.
Entity | User Role |
|---|---|
Action Items | Administer Actions |
Advanced Configuration | General Admin (without User Management) |
Agreement & Promotion Attr Agreement & Promotion Types Agreement & Promotion Type Attr Condition Types Condition Type Attr | Administer A&P module |
Business Roles | Administer Users |
Calculated Fieldsets | Manage Calculated Field Sets |
Calculation Flows | General Admin (without User Management) |
Calculation Logics | Manage Calculation Logics Manage Analytics Calculation Logics Manage Custom Form Calculation Logics Manage A&P Calculation Logics Manage Rebate Calculation Logics Manage Actions Calculation Logics Manage Sales Compensation Calculation Logics |
Claim Type | Manage Claims |
Company Parameters | Administer Company Parameters Manage Company Parameters Tree |
Compen Accrual Record Compen Record Attr | Administer Compensation Records |
Compen Attr Compen Calculation Compen Cond Type Attr Compen Cond Type Compen Header Type Attr Compen Header Type | Administer Compensation Plans |
Configuration Wizards | General Admin (without User Management) |
Customer Attr Customer Extension Attr | Administer Customers |
Custom Form Custom Form Type Custom Form Type Attr | Administer Custom Form |
Custom Form Module Category | Manage Model in Module Category |
Dashboards | Export Dashboards |
Data Change Req Types | Manage Data Change Requests |
Data Loads | Manage Data Manager |
Datamarts Data Sources | Administer Schemas |
Event Orchestration | Partition Events/Jobs - in Platform Manager! |
Live Price Grid | Administer LPG |
Manual Pricelists | Administer Price Lists |
Message Templates | General Admin (without User Management) |
Model Classes | Administer Model Classes/Objects |
Module Categories | Manage Model in Module Category |
Preferences | Edit Global Preferences |
Price List/Grid Type | Manage Price List and LPG Types |
Price Record Attr | Administer Price Records |
Product Attr Product Competition Product Extension Attr | Administer Products |
Publishing Templates | General Admin (without User Management) |
Quote Attr Quote Type Quote Type Attr | Administer Quoting |
Rebate Agr Attr Rebate Templ Rebate Templ Attr Rebate Agr Types Rebate Agr Type Attr Rebate Cond Types Rebate Cond Type Attr | Administer Rebate Agreements |
Rebate Calculation | |
Rebate Record Attr Payout Record Attr | Administer Rebate Records |
Rollups | Manage Published Rollups |
Saved Charts | Manage Saved Charts |
Seller Seller Attr Seller Extension | Administer Sellers |
Users User Groups | Administer Users |
Workflow Logics | Manage Workflow Logics |
Timeout
If the deployment pipeline performed by pfxpackage tool is taking too long, it might be worth extending the timeout threshold - 120s is set in the pipeline templates as default. This may happen momentarily if the latency between the worker server running the pfx-package tool and the pricefx cluster has increased, or if your project has too many artifacts to be deployed at once, especially during rollouts of new phases.
Objects locked
When a calculation job is running, the table is locked. So you may need to re-run the pipeline.
Further reading:
See also: Git Version Control in Pricefx
Found an issue in documentation? Write to us.