Sunday, May 8, 2011

Economic principles of testing - why test, what and how long

This week I received an email in which I was asked following questions:
  1. How to determine if functionality is critical or not? Is it on the basis of business turnover or something else?
  2. Consider this situation: around 40 systems, approx. 800 interfaces, and thousands of functionalities. How to effectively address the importance (in terms of being business critical) of individual functionalities and their relationships? Are there any dependency algorithms for this?
  3. Is it possible to define end of testing based on criteria that if the bug count is below some threshold, you can go into production? Or is it necessary to go to production only when the bug count is zero in all components?
As all of these questions revolve around the economic understanding of testing, I realized that I did not address the explanation of basic principles yet and decided to remedy this situation.
So here are the basic economic principles of testing.

We test because it pays off and as long as it pays off

In business every step that we do we do with the prospect of future profits. If we offer employees various benefits, we're doing it because we want to entice quality employees and reduce costs caused by their volatility. If we invest in quality products, we do it because better quality brings us more money. Whether due to the fact that we do not have to spend large amounts of money on the correction of defects discovered by customers, or because it brings us more satisfied customers.

This is related to the fact that we test as long as it pays off. If the cost of discovering and fixing defects at some stage is bigger than the estimated costs resulting from leaving them there, it makes no sense to continue with testing.

Sometimes the manager can get into a situation where the application is very buggy, testing is effective and it would be good to have two or three months to continue with testing and fixing bugs, but for some reasons, for example legal issues, it is still more cost-effective to deploy the application.
It is customer call to know the overall situation and decide when to deploy the application. This decision should be in hands of manager of that part of company, who will use it or will sell it on mass market. Test manager only provides him with information for making this decision, especially to warn him about the quality risks.
If only 20% of an application is fully and reliably functional, 10% has too many defects to use it and the rest has not been tested yet, what it could mean if application is deployed?

Thus tester is not the one to define the acceptance criteria or making the decision when to release the application into production. This is customers call and he decides how many defects of what severity to tolerate or if he makes the decision not based on quality but based on the situation on market.

We monitor the effectiveness of testing in terms of number of newly discovered defects 

As mentioned above, we should test as long as it is financially worthwhile. What does this mean? It means that we need to stop testing in the point when the future cost of finding and correcting defects would be bigger than cost of not to continue to test and risk that the defects will be discovered by customers. But determination of such a moment is not trivial. Apart from the need to work with a probability, it is also necessary to have sufficiently detailed information on the defect and testing costs. This is information that most companies do not monitor at such level is not monitored and so it is not available. Less accurate but simple and reasonable alternative is to track the number of newly found defects. The following picture shows a typical example of how the number of newly found defects fluctuates during testing.

Initially, the test team founds a lot of new defects. Over time, the number is decreasing and it appears that further testing could not bring any surprises. At this point, an inexperienced test manager terminates the testing thinking that the product is ready. Experienced test manager continues, knowing that the testing normally grabs a second wind, and the number of discovered bugs will begin to grow again. Moreover the test manager will support the growth in discovered defects by variation of tests. However, even without the change of testing approach, the increase is expected. End of testing is advisable after the second or third recess. Of course this is only one of the methods how to determine when to stop, there are others that are based on data gathering and monitoring.

Testing is an investment. Money should be put where it will have the greatest effect

The reason for determining priority of the functionalities is to focus the efforts appropriately. What is important for business is determined by the customer again. Most often in complex systems, the criticality of applications and business processes is well known and to find it, it is sufficient to contact the risk manager or look into contingency plans. Determination of the criticality is based on what the impact would be of the application unavailability or malfunction.
For example, the losses of one day unavailability are estimated, and the ranges are matched with importance labels.

Regarding the importance of individual tests, here you can draw mainly from the priority of use cases. The customer identifies their priorities in the phase of gathering requirements and specifications. To determine priorities he usually think about how many people use the function, or if it comes to contact with critical resources (money, personal data, etc.) or how essential the business processes are, which function handles. However often the customer, especially in smaller projects, identifies the importance based on feelings and according to contractor’s instruction determined 20% of the most important functionalities. Popular and well understandable is assigning three levels of priority:

Critical to quality - Without functions with this label the product cannot be use and it would fail to fulfill even the most basic needs of the customer.

Good enough quality – This label marks needed but not the most critical functions.

Nice to have – This label marks features that are the proverbial icing on the cake. They can make improve the experience or make a work a little bit easier or be just another thing that someone could use in the future.

So back to the start, the answer to all three questions is: "ask the customer what he wants". Because he is the one who invests, and he is the one who will consider the return of the investment.

Sunday, March 13, 2011

Importance of Communication in Testing or How to Communicate Your Ideas

This week I attended a CzechTest conference and it was a wonderful experience, as there were many foreign experts that had 10 to 20 years experience in Software Testing. We have a serious lack of those people in Czech Republic, don't we? But I was most impressed by Lloyd Roden, consultant from United Kingdom, who has been also speaker at conferences such as STARWest, STAREast, AsiaSTAR and EuroSTAR.

What was so different about this man was not his ideas or content of his presentation because I have met many intelligent and experienced people like him, but his communication and presentation skills. After his presentation you not only think about the topic, but you also feel energized and want to transform these ideas into your everyday life. And that is probably one of things that make him such a good Test Manager, because in testing (especially test management) it is all about communication.

Rik Marselis, one of other speakers at this conference asked us if we think testing is technical discipline or people discipline. I think it is both.

There are a lot of good testers who understands more computers than people and that is fine as long as they are teamed with testers who are good in communicating with and understanding to stakeholders.
Communication is the most important skill to all managers and people who wants to be heard. And I have not met a tester yet who does not want to be heard and promote good quality in his organization.

Tester is a programer’s best friend.

I teach my students that tester is the programmer’s best friend.
But how could someone who is constantly finding faults in the men work be his best friend?
Only by being tactful, helpful and by providing good information that this man needs.
But hey! That is exactly what it means to be a good tester. Tester provides information? Isn’t this great?

Programmer is just one other person we are helping by providing necessary information. If we care for programmers and help them find and fix the problem quickly by providing relevant information, we really are their good friends.

The same goes for other stakeholders as well: managers, users, business analytics, and so on.

Tester provides information. Good tester provides the right information.

This is the skill we need to hone: how to provide the right information and how to communicate it properly.
And if you want to go now and improve your skills, here is some tips how to start.


Testing survey

During the last year I was very busy with teaching in University of Economics in Prague (I am teaching graduate student there) and working as test analyst, test methodic or test coordinator. But it was a fruitful time as I am happy to present the results of a testing and quality management survey which I spent several month of gathering data for.

Original study and results are published at, but only in Czech language. But you can see Google automatic translation of this page here.

More than 300 Czech IT companies were asked to fill out this questionnaire and in the end we got 84 answering sheets. The results show average knowledge among software testers working in Czech Republic. If you speak with more experienced testers or testers who actively seek knowledge, you will see their opinions and answers would be much different. I present these results as it is without any interpretation or analysis as I will present processed results in journals or at conferences.

It would be good to compare the results to diffrent country. So if you have an opportunity to do this survey in your country, please, DO IT.  Do it and send me the results.

 If you have any questions please do not hasitate and email me.