Progression vs Regression Testing (Big Difference?)
If you have ever heard of progression testing, then the chances are, you may have heard of regression testing. But are they different? similar, or something completely different? Let me explain…
What’s the difference between progression and regression testing? Progression testing focusses on new functionality and proving that it works as per the requirements. Whereas regression testing focuses on proving that existing functions of the application are not broken from the addition of new code.
During my years of testing, regression, and progression testing have been critical. Especially when you are dealing with systems that have serious impacts on customers. So, let me explain in further detail the major differences. In addition to this, I will give some examples to make it more understandable.
What is progression testing?
Is reality its another word for saying functional testing. Some people also refer to this as incremental testing (Click here to see what the incremental model really is), depending on what type of testing model you’re using.
Essentially, it’s testing new functionality in a methodical approach. To make sure that the new functionality matches the desired requirements.
I say sometimes referred to as the “incremental model” because it depends on what model of testing is being used. For example, you could be using the V-model.
However, whatever your model of choice is, progression testing still comes down to testing new functionality against your requirements.
What exactly is functional testing?
Functional testing is a method of verifying the outcome of the application in comparison to the agreed requirements.
It is often referred to as a form of “black box testing”. This is commonly referred to as black box because we are not focused on the underlying code for this. Testing code modules is known as unit testing (Click here to see the difference between unit and integration testing) or “white box testing”.
So, ultimately with functional testing, you are looking at the functions of the system and making sure that expected outputs are seen, are you with me?
So, what is regression testing?
Regression testing is different from progression testing. Ultimately, with regression testing, we are making sure that we have not broken anything when we introduce new code. Simple as that.
Let me give you a working example here…
Example: Calculator App
Let’s assume that you’re testing a calculator application. Currently, it is very basic and can only add and subtract, nothing else.
Now requirements have been drawn up to add a multiplication feature to this app. We can also assume that the application is fully functional and in production at present.
To regression test, the calculator application after the new multiplication feature is tested. You need do some additional tests to:
- verify that the addition feature is still working.
- verify that the subtraction feature is also still working.
Why do we need regression testing?
Regression testing is needed because mistakes or errors in code can be expensive to clean up post-go-live.
Typically you’d need to use regression techniques for the following situations:
- If you have a new defect fix.
- New functionality (Such as the calculator example above).
- Performance improvement.
- Routine maintenance release.
All of these potential changes can provide a risk of damaging existing functionality which can have adverse cost effects to the business if they are not tested.
How do you do regression test?
Once you have decided that regression testing is important and worth the time investment. You then need to think to yourself, how are you going to tackle it? Let me explain…
Essentially, there are three main ways to approach regression testing:
- Test everything
- Test a selection of test cases
- Prioritize testing (Risk-based)
Testing everything is one of the most thorough ways of doing regression testing. Meaning you test the entire suite of tests after every new change is implemented.
You’ll often find that most companies will not want to invest in this. Simply because there is a massive cost impact for doing this.
To be perfectly honest, from my experience of testing, it just isn’t feasible to be able to do this every time a change is implemented into the system. So, it’s unlikely that this will ever be used.
A selection of tests
A selection of test cases can be chosen instead. This is usually the approach businesses choose. But there a decision made on what to test. In some cases, a standard regression suite can be selected for this purpose. Depending on the business needs.
Prioritizing test cases (Risk-based approach)
Prioritizing is a clever way to select which test cases should be tested first. Depending on time restraints its likely that only the high priority regression tests will be executed.
Essentially, you and your project team make a risk assessment on which test cases need to be focused first. This could be based on the risk of the system being impacted or it could be based on how often a particular function is used.
There can be a number of different reasons behind these decisions. But, essentially it comes down to testing only a subset of test cases based on their perceived priority or risk Factor.
Q: How do you use regression testing in agile?
In agile testing, you mainly use short sprints of functionality (click here to see what an Agile sprint is) to test. Therefore there are two main ways that you can tackle regression testing in agile:
At the Sprint level
At this sprint-level, you will effectively test the functionality which is relevant to that particular sprint. It is more of a scaled-down modular based approach.
The end-to-end approach Focuses on the entire application. With each sprint module talking to each other. To regression test this, a selection of test cases are run to ensure that there are no problems (defects).
In modern agile testing, there is a growing appetite to automate the majority of these tests to speed up the delivery to market.
For example, in my experience of testing, I was working for a well-known mobile telecommunication company. They had a Christmas promotion one year targeting millions of homes.
Effectively it was offering their customer base as a special offer for Christmas. This was new functionality added to an existing codebase. Therefore, a large amount of repeatable regression testing was required.
Due to the number of man-hours required to test this manually a decision was made to use selenium test automation suite.
We implemented the selenium test automation suite to go through many laborious iterations of dropdown forms on a web application. This is an example of how test automation could be used to speed up regression testing
Q: What are the test automation regression tools are there?
As mentioned above Selenium is one good example. However, there are quite a few others such as QTP, winrunner, etc.
Q: What are the advantages and disadvantages of regression testing?
The advantages of regression testing is it gives your application higher quality delivery. It’s one of the best ways to secure the quality of your application.
Also, you have the advantage of being able to automate some of these laborious tests to speed up the delivery.
Disadvantages are the additional cost that comes along with regression testing to bolt onto a project. Unfortunately, some companies have a tight budget and cut corners. This can be a disadvantage.
Also if automation is not an option for your company due to lack of budget for the tools, you are left with quite a large burden of manual tests.
Either way, in my opinion, regression is critical. And, although it may seem expensive when you look at the costs up front, long term it can save your company some serious time and money downstream. The costs after going live can be exponentially more than if it was picked up before then.