0

What is Non-Functional Testing? (2 examples)

If you are preparing for your ISTQB or just curious, you may be wondering what non-functional testing (NFT) is and how it fits into the testing life-cycle. Let me explain…

What is Non-functional Testing? It tests how well a software system behaves, based on an agreed specification. In particular, it focusses on certain aspects of the system, for example, performance, usability or security.

Now that you know what it is, let me explain a bit more detail Such as when is it used, how it can be measured if special skills are required, the types of non-functional testing and more. Keep reading…

When is non-functional testing used?

Understanding what Non-functional testing is, is one thing. But, understanding when to use it is another. For that reason, in this section, I will explain when it is a good idea to use this. The answer may shock you because it’s not what most people assume or do.

Even though most software testers will assume that non-functional testing should be scheduled after system testing. The reality is, it should be done at all test levels. In fact, the earlier the better.

Why is this? Simply because in the event that you find a serious “show-stopping” non-functional defect. It is better to detect it sooner rather than later. The cost of finding it later is nearly always significantly more.

How can Black-box testing techniques be used for this?

While planning non-functional tests you need a method to work out the test casts. For that reason, in this section, I am going to explain how you can use some known Black-box testing techniques to draw up these cases.

One of the known Black-box techniques, called boundary value analysis (BVA) can actually be used to derive non-functional test cases. IN BVA, the agreed boundaries of fields and inputs are tested to make sure that they meet the agreed specification.

How can the Non-functional tests be measured?

Without the ability to report and quantify testing would effectively make Non-functional testing worthless. But, you may be wondering, with that being said, how can it actually be measured, right?

The coverage of non-functional tests relies on clear requirements/specifications. For each non-functional element specified, it is possible to measure the coverage based on the number of tests run per element.

This can then be used to help to find any gaps in the testing. Such gaps may be deemed as acceptable based on the risk it poses to the system, assuming a risk-based analysis is used.

Are special skills required?

Non-functional tests cover a large set of possible tests. But, you may be interested to know if any resource can run or design these tests, or does it need a special level of expertise?

In most cases, non-functional tests, for example, performance tests, typically need a specialized resource with a great level of experience. This experience could be deep knowledge of a tool, e.g. win-runner. Or specialised security-related skills to make sure the system is tested correctly.

Who does Non-functional Testing?

In the last section, we touched on what type of skill set is required to do non-functional testing. However, I did not explicitly state who would be the typical resource member to do it.

Non-functional is quite a broad categorization, so there will be different roles for every different particular testing activity. For example, for performance testing, you would typically have a dedicated performance test analyst. If the testing required is quite significant, this could be a team of testers. With one of these resources being a Test lead or senior tester.

Is security testing functional or non-functional?

If you have security testing in the scope of your project, or you are just curious, you may be wondering what this is classified as. In this section, I will explain this.

Security testing is classed as non-functional. It is required to verify that the system in question is secure. Such as resilience to external infiltration.

What is the difference between functional and non-functional testing?

To make sure that it is clear, I want to draw a distinction between functional & non-functional testing. This will help you to understand what NFT is.

Functional testing (Click here to see why Functional Testing is Required) is concerned with the functional elements of the system. Typically, this is defined in a functional spec as “features”. At this stage, we are not concerned with code-level testing. It is mainly black-box tests that focus on the inputs and expected outputs of each function. And, verifying that they meet the agreed specification.

Non-functional testing is different because it focuses on aspects of the system that are not usually addressed during functional testing. Such as how many users can use the system simultaneously (max load) before it stops responding, are you with me?

Then, to follow on with this example. Once we establish the max load, how does this compare to the documented non-functional specification? Does it support as many users as we expected, or not?

My experience with Non-functional testing

Reading about NFT is easy, but understanding it is another. Sometimes a real life example will help to put things into perspective. For that reason, in this section, I will give you an example of my first-hand experience using this method.

In one of my contracts in the past, the client was responsible for providing vehicle finance. Therefore, they have a large amount of critical data that needs to be secure and resilient. In particular, in the event of a disaster, e.g. office fire, damage to servers, etc.

I was tasked with leading an annual task they had. Which was to verify that they can recover all of their core systems following a disaster. To do this we emulated the situation by creating a “disaster day”

Disaster Day (Disaster Recovery – Non-Funtional testing)

This testing is known as Disaster recovery. And, is a known example of non-functional testing. The objective was to prove that we can prove that each core system can be used following the disaster.

To prove this, we had systems located on another site. Their sole purpose was to provide a different backup environment that we could switch to following the disaster, are you with me?

The recovery Site

On the recovery site we had two goals:

• Prove that the core systems can be used (core functionality)
• Verify the data was restored correctly.

Proving that the core systems can be used

To prove this, we had a selection of Subject Matter Experts (SME’s). Their role was to test the recovered systems and verify that the core functionality was operational, following an agreed test plan. The time was limited (one day on site for all tests), so we had to be efficient.

Verify the data was restored correctly

The second goal, and arguably one of the most important, was to prove that the data was recovered successfully. This was done by taking a snapshot of the data before the disaster day. And then, comparing that snapshot on the recovered system.

To clarify, although this sounds complicated, it was done as simply as possible (due to time constraints. The data snapshots were as simple as taking some screenshots before the disaster and then comparing them after the data was restored on the backup systems, are you with me?

Why are non-functional requirements important?

You may be wondering what is so important about these non-funtional requirements. And, you may even consider cutting corners and leaving them as a “nice to have”. But, in this section I will explain why they are required.

Outline Performance Boundaries

One of the main reasons why it is important is the performance boundaries. Once you understand these boundaries, e.g. how many users can use the system simultaneously. It can help your organization understand what they need to do to scale up. It can give you an idea of the cost of this and the impacts it may have on other projects, resources, etc.

Also, it can give an insight to see how long the system is useable in its current form. This can help with planning for future quarters, etc.