As Florent Dotto mentioned in his post, we use the Agile approach, we have several teams and QAs are respected members of the teams. I highlighted the word intentionally because I often hear other QAs saying that developers do not respect them, or developers saying that QA role is useless because they can check what they developed by themselves, and QA is just not needed. We don't have this approach in Pricefx at all, testing is valuable part of the process and I've never heard from anyone here something like "hey, QA is not needed". It's perfect, and it's one of the reasons why I like to work here. 

Now let me describe the release process and the related infrastructure from QA perspective. We have two major releases per year, then we have a minor release every month. And we have four (or better five) environments that can be used for testing and development: develop, staging, master, oldstable. Master is the version that is used on production, the most recent stable version. Oldstable is older, also stable version (used by some customers). Staging and develop are versions used for the development. What we do on staging, it's meant to be released in the minor release, so usually bug fixes or some very important features that some customer is waiting for. Develop is meant for the big new features that will be released in the major release. Now where is the fifth environment I mentioned? Well, this is part of the QA paradise – the fifth environment is called templates, and it's used as (simply said) a storage of our data and configurations. We don't do any testing or development there but when we want to store something permanently, we save it there and every night everything from templates is propagated to all the above mentioned environments. It means we can do whatever testing, playing, breaking and we know it's safe because the next day it will be "fixed" again to the default state. And of course, we can run the restoring job manually, if needed.

I do manual testing; it means I test mostly front-end and mostly the way user would use the application. Although manual testing can be considered as lengthy process, it's needed here because Pricefx is highly configurable software and it usually takes time to understand the context of the bug/feature and the configuration needed, to be able to test it. But we also do automation testing, we have a whole team dedicated to automation. We use http://cypress.io and when it makes sense (e.g., commonly used configuration) we try to cover the use case by an automated test. When I am test some issue and I think it should be covered by automation, I just use a label in JIRA and guys from the automation team will find the issue and can cover it.

Now the negative part – as I already mentioned, Pricefx is highly configurable software. In practice, it means we have an endless number of combinations that could (and ideally should) be tested. Everybody knows there is nothing like 100% test coverage. On the other hand, we have customers who are saying "hey, we have this specific configuration with these parameters, and it doesn't work!". How to find a balance? How not to go crazy and still satisfy the customers? Especially before the major release, when we do the regression testing and with many combinations, it can take a month of testing (when we don't do anything else than going through the application and checking every single feature, clicking on every single button etc.) – that sounds terrible, I know!

Right now, we are trying to cover as many regular/common scenarios as possible by automation so manual testers can concentrate on the configurations and don't have to spend time with "easy" use cases. It will take time but we are getting there. I think one of the most valuable qualities of a good manual tester is intuition – we just somehow know (or feel, if you want) where is a critical place that should be examined. Of course, it's based on our experience, we know what areas have been touched by the developers, so we know where to pay our attention.

The good thing is that we aware of what is not ideal and we work on it. I am with Pricefx for almost three years, and we already improved the process a lot. Florent started his article with F-Words, I would like to end with them – when we see something doesn't work or it works but it's not perfect, we just change it so it's better. It's a continuous process, but that's it – being Fast, Flexible and Friendly at the same time.

• It required time, as any manual process. The person doing the release needed to have an environment ready to use the Maven commands, and then follow the documented steps one by one.

• It was not a 'pleasant' job to do. Because the process was very mechanical in nature, it was not something that had room for creativity. On top of that, to ensure that everybody was familiar with the process, a different person was chosen every time to do the release.

• It was error prone, as every manual job is as well.

This release process was a particularly suitable candidate to be automated. The goal was to have a reliable 'push button' that anyone could use when needed, without any other intervention.

What Kind of Releases Are Done

We have a structure of Git branches similar to this:

Releases are done only from the 'stable' branch once QA gives the greenlight.

We have two types of releases:

• Hotfix releases. It's a small release that contains only bugfixes that are considered important for a customer. They are done as needed.

• Minor releases. It's a bigger release, that contains both bugfixes and new backward compatible features. They are done once a month.

Occasionally we also do ‘major’ releases, but that’s out of the scope here.

Starting to Automate the Release Process

Before the automation, and to have an effective automated release process, one change to our internal versioning system needed to be made. 

Semantic Versioning

Initially versions consisted of two digits. When tags were created, they actually contained three digits. When creating such tags, the person who was doing the release needed to check what was the latest hotfix version released, and use that information accordingly.

So one change that was done was to switch to Semantic Versioning. This way, the hotfix version could be stored in the project, and could be used automatically. 

Maven Release vs. Manual Tagging

Our software is built with Apache Maven, which already provides a release plugin that could do some of the automation we want. However, this plugin was designed to do a series of fixed steps which are not customizable. The plugin documentation states the following:

To be clear, this is all the Release plugin does, it isn't a substitute for organizational process, it wasn't designed to be customized to meet your specific process.

The release plugin also violates the Single Responsibility Principle – one command changes versions, creates tags, deploys artifacts and site documentation.

In our case, we wanted to have more control over the process, so we went for a script that would run only the commands we needed. Something similar to what we initially had in our releases documentation. 

Working on a Release Script

The script that was prepared is simple, about 130 lines. The operation of the script is roughly described in the following diagram:

This diagram applies to minor releases. The merge request is still a manual step, so it can be reviewed if needed before the automate steps take control. For hotfix releases, there is no merge request involved, but the staging branch is still kept up-to-date at the end.

Also, the following points were considered during the design of the script.

Fail Fast

The script needs some preconditions to work, for example a GitLab token needs to be set as an environment variable. All the preconditions are checked at the very beginning, and if any of these preconditions fail, the script fails immediately, without doing any actual changes in the repository.

Make Changes Atomic

Perhaps one of the most important conditions of the script is to never leave the branches in an inconsistent state. Because of this, no changes are pushed until the very end. Git offers the --atomic parameter, which ensures that either all the changes in all the branches changed will be pushed, or none will. This approach has also another good side, and that is that in case the script fails, whatever error happened can be corrected, and the operation can be just retried again without any other side effects.

Propagating Changes

As any other branching model, commits in 'lower' branches must be present in 'higher' branches as well. Once a release is completed in the 'stable' branch, all the commits should be propagated also to 'staging', the immediate superior. This is as simple as merging the 'stable' branch into the 'staging' branch. 

Dealing with Conflicts

However, conflicts are almost always sure to happen. As in every hotfix release the version in 'stable' is changed after it was branched from 'staging', one possible merge conflict will be the version.

On the other hand, if we knew that the only possible conflict was going to be the version, then an 'empty' merge commit could be done, and this way, any manual intervention could be avoided. Git offers a merge strategy named 'ours', and that's what the script uses.

But it's very important to make sure that the version is indeed the only possible conflict. Otherwise, other changes would be lost in this empty merge commit. Because of this, one of the preconditions of the script is to check if the 'stable' and 'staging' branches are in sync. This is, if there are commits in 'stable' which are not in 'staging'. If that's the case, the script fails just as it would do with any other precondition, and it's up to someone to correct the situation first.

How It Is Used

As described in the previous diagram, the process is expected to start after the staging branch is merged into the stable branch, in the case of minor releases. Right now, after the changes that the script brings, to start the release process, we have a Gitlab job that anyone can trigger without any further training or documentation to read. And for hotfix releases, the process is even a bit simpler as no merge request is involved.

This is essentially a one-click button to do releases, and exactly what we were aiming for. Of course, there are still some aspects that can be improved even more, and that is something that we will be looking into in the future.

Future Work

Everything can be improved. Automation is not a goal, but more of a journey. And during that journey, new ideas can come up. For example: should integration tests be run in the pipeline build for the tag? Usually, the only thing one is interested in in such builds are the artifacts. But on top of that, it should not be possible to do a release if there are broken tests in the branch. Right now, the person who does the release can see whether the branch is in good shape or not before pushing the release button, but this can also be enforced with GitLab rules.

Our team organization is built collectively. It is a process rather than a fixed setup.

Like many other teams in Pricefx, we have our own specifics. They can be very stimulating but can also create troubles or add burden: we work fully remotely, our team is truly multi-cultural, with a large skillset (data scientists, front- & back-end developers, QA, AI developers), we operate in up to 3 time zones. Our other specifics help us mitigate these issues: we have technically skilled senior people, they develop highly sophisticated, yet usable software and their mindset is highly cooperative.

You won’t be surprised to learn that we use the Agile approach. We run two-week sprints with five basic rituals:

• Daily Stand Up Meeting, set to a time zone allowing us to onboard junior/new team members. Considering our fully remote mode, this short synchronous meeting helps us stay connected on a daily basis, have small talks and tune in to more technical stuff with other teammates during the day.

• Sprint Planning, Grooming Session, Sprint Review and Sprint Retrospective are set alternatively to cover each time zone and allow all our members to regularly participate.

To communicate our team uses a dedicated channel in Microsoft Teams. Our policy is to share all interactions among team members. For example, each internal peer-programming session or work meeting are launched in this chat to allow anybody on the team to participate. This approach is particularly important to mitigate the time zone effect (by sharing information asynchronously) and the fully-remote constraint (by exposing each peer-to-peer interaction to the whole team). The purpose is not to track people's activities but to be sure each teammate has a clear understanding of what is happening "live" in the team, as if we all worked in an open space.

No matter how well we are organized and what handy tools we use, as a fully-remote team we also need real-life events to reinforce the links among us and to break our solitary routines. We meet at least twice a year for two days (if covid permits). These two days are dedicated to team retrospectives and working sessions and – just as importantly – we want to have time to eat & talk together (many French people in the team...). As the goal of these events is to provide an enjoyable experience for everyone, they must be well prepared – and in this the whole team participates. Each time we come back recharged and refreshed by the trust in our relationships, ready to overcome any little misunderstanding we will face in the next six months of working remotely.

That’s our organization as of now and we are very curious to see what it will be like at the end of 2022!