NEWGeneral truths and myths about testing

Well, think about this situation - software testers will find a bug, it is reported and assigned to the developer. However, since the deployment of the software to the customer will be next week, it is decided by the stakeholder to postpone bug fixing and deploy the software with the reported bug. So the bug was founded, but wasn’t fixed yet - obviously the quality of the software is still the same as it was before the bug was found, right? So, NO, software testing will not improve the quality of the software. Purpose of software testing is to find important information about the software (bugs are one of them), so important decisions can be done by stakeholders as soon as possible.

I can imagine the idea seems to be great, but we should consider the bigger picture and ask some questions first:

• do we have a big data set we need to test and is the application stable? If yes, then we should really think about test automatization, because it will help us test regression testing faster and with more test data than we can do manually. However, we still should think about somebody is needed to evaluate reports from automated tests, report bugs and modify automated scripts.

• who will create those automated tests? Not every tester is able to create them because they usually require special technical skills. Do we have a suitable resource?

• is our tested application stable? If it is not - automated testing is usually very expensive and much cheaper is manual testing, because automated tests must be modified and upgraded over and over again

• do we have the right tool for automatization and are we ready to pay for licences? Test automatization needs a special tool, some of them are free or very cheap, but some of them are very expensive. First, we should try the tool and check if it is suitable for our software. Then we must evaluate and decide if we can afford to buy it.

Obviously, test automatization is quite a complex topic. It can help, but it can also be very expensive, much more than continue with manual testing. So, think about your project situation.

Regarding this topic, there is a beautiful saying: “If you want to be a good tester, learn to think like one, not look like one”. So when we mean being a good tester, then NO. Being a good tester requires a set of skills and a mindset not everybody has. It requires analytical skills to understand the software, technical skills to understand details of the solution, thoroughness, critical thinking but seeing the big picture at the same time, often also to understand business of the customer and on top of that being a good communicator, understand the project development process, … . So not everybody can be a good tester, but everybody can look like a tester .

If you hear from anybody he/she expect zero bugs in the software, be very careful because this person either doesn’t understand the complexity of the software or have no experience with software development. Having zero bugs in the software is NEVER goal of software testing and there are basically two main reasons for that:

1. Time is limited: the software today is complex, it has a lot of inputs, outputs and it provides a lot of functionality and behaviour, so to check every possible combination, functionality and behaviour took us ages. We don’t have time for that on the project, the customer is never willing to pay us for a long time of testing so we have to be reasonable and effective. We are using techniques for test analysis to be effective, we prioritize to find the most important bugs first and we use tools to check even things not visible from user interface to check everything is really working. When testing the software we’re doing our best to be sure the software is fulfilling requirements, is working without bugs and to satisfy our customer but we can never be absolutely sure there is no bug in the software.

2. We don’t control everything: Have you noticed how many programs are running on your computer? Starting from OS (like Windows or MacOS), then there is an internet browser, CRM ERP and a lot of other systems completely out of our control. Also, their updates are out of our control! We often even can’t choose if we want them to install! We assume they will not affect the functionality of our software, but we can’t be sure. Basically, it’s the same like point 1: we’re doing our best, but it can happen there is a bug outside of our software which causes our software to stop working.

This idea is usually coming when there is a big pressure on lowering the project price. At first sight it can reasonable, right? In some cases it can really work - especially when the customer has experience with testing on IT projects and is ready to test everything by himself. Typically, however, it is not working well. Why? There are several differences between developers and testers:

1. Testing coverage: developer is typically testing just the requirement he developed and typically very simple scenarios. Testers are trying also more complex scenarios, testing on edges and regression testing of more requirements together. So obviously when testers are not on the project, customer can expect he will find more bugs

2. Test management: Testers are typically documenting what they are doing at least in a simple way. They are typically using test management tool (like X-Ray) to create test cases and documenting results so they can be easily checked later or used again in case it is needed. Created test cases can be also used by the customer as a baseline for his testing. In case testing is done just by developers, test management is typically completely missing on the project.

3. Help with UAT: As customers are typically not experienced software testers, they often need help with UAT phase. When UAT phase is coming, a lot of questions arise, like:

   1. How UAT test cases look like?

   2. What we should cover during UAT?

   3. Is there a tool we can use?

   4. How UAT phase should be organized?

While software testers can usually help the customer with all those questions and make the UAT phase more smoother, developers can’t typically help here and the customer must deal with UAT by himself. As UAT phase is so important to the project, it can significantly increase the risks on the project.

Every sw is upgraded from time to time. There are different reasons for this - new features, bug fixes, security patches, UI upgrades or for other reasons. So first important thing here is to accept that sw upgrades are normal. And is testing after upgrades necessary? In some cases not - in case of low priority sw, but with our sw we are talking about sw important for business with a lot of business data and complex calculations, so it is definitely reasonable to check that main business cases are working correctly. We are not recommending to test the whole sw from scratch, but business critical use cases should be definitely selected and retested so the customer is sure that everything is working correctly after upgrade. In case test cases were created for UAT phase, some of those test cases can be definitely used, based on selected use cases. In long term perspective test automation can help a lot with testing after upgrades, as business cases are usually not changing much.

Well, to be honest, it can be true. Especially when we are looking just at project team costs. When there are QA guys or software testers in the project team, somebody must pay for them, right? But this way of thinking can lead to questioning other roles in the project team (like PM and SA). Isn’t the best solution to have a super experienced developer who will lead the project, communicate with the customer, propose the solution, develop the solution, test the solution and make a demo for the customer. Kind of superman, right? Of course not. Every role in the project team has its own purpose and requires different skills. If we remove any role from the project team, we increase the risk our customer will find bugs during UAT, and not finishing the project in time. Is this risk really what we want and are we ready for consequences?