I discovered that I do not have enough time and energy to continue with this blog. I also abandoned my previous research about state of testing in diffrent countries.
I apologise for that.
I am still an active participant in Czech testing comunity and will try to continue at least with Czech version of this blog.
Tuesday, February 25, 2014
Thursday, January 19, 2012
Testing research
Hello,
I am doing research of current state of testing and quality management to compare knowledge and state of testing in different countries and I would like to ask you if you could fill out a simple questionnaire.
The research is anonymous. You do not fill any personal information about you or your employer’s company.
Reasearch is done for my PhD. studies at University of Economics, Prague, Faculty of Informatics and Statistics and the aggregated results will be presented at conferencies and journals.
I will also provide the aggregated results after 1st of March at the same webpage as you can find now the questionnare.
You can find the questionnaire here: http://studiekvality.vse.cz/studie_en/
Thank you for each completion of the questionnare.
I am doing research of current state of testing and quality management to compare knowledge and state of testing in different countries and I would like to ask you if you could fill out a simple questionnaire.
The research is anonymous. You do not fill any personal information about you or your employer’s company.
Reasearch is done for my PhD. studies at University of Economics, Prague, Faculty of Informatics and Statistics and the aggregated results will be presented at conferencies and journals.
I will also provide the aggregated results after 1st of March at the same webpage as you can find now the questionnare.
You can find the questionnaire here: http://studiekvality.vse.cz/studie_en/
Thank you for each completion of the questionnare.
Sunday, May 8, 2011
Economic principles of testing - why test, what and how long
This week I received an email in which I was asked following questions:
So here are the basic economic principles of testing.
We test because it pays off and as long as it pays off
In business every step that we do we do with the prospect of future profits. If we offer employees various benefits, we're doing it because we want to entice quality employees and reduce costs caused by their volatility. If we invest in quality products, we do it because better quality brings us more money. Whether due to the fact that we do not have to spend large amounts of money on the correction of defects discovered by customers, or because it brings us more satisfied customers.
This is related to the fact that we test as long as it pays off. If the cost of discovering and fixing defects at some stage is bigger than the estimated costs resulting from leaving them there, it makes no sense to continue with testing.
Sometimes the manager can get into a situation where the application is very buggy, testing is effective and it would be good to have two or three months to continue with testing and fixing bugs, but for some reasons, for example legal issues, it is still more cost-effective to deploy the application.
It is customer call to know the overall situation and decide when to deploy the application. This decision should be in hands of manager of that part of company, who will use it or will sell it on mass market. Test manager only provides him with information for making this decision, especially to warn him about the quality risks.
If only 20% of an application is fully and reliably functional, 10% has too many defects to use it and the rest has not been tested yet, what it could mean if application is deployed?
Thus tester is not the one to define the acceptance criteria or making the decision when to release the application into production. This is customers call and he decides how many defects of what severity to tolerate or if he makes the decision not based on quality but based on the situation on market.
We monitor the effectiveness of testing in terms of number of newly discovered defects
As mentioned above, we should test as long as it is financially worthwhile. What does this mean? It means that we need to stop testing in the point when the future cost of finding and correcting defects would be bigger than cost of not to continue to test and risk that the defects will be discovered by customers. But determination of such a moment is not trivial. Apart from the need to work with a probability, it is also necessary to have sufficiently detailed information on the defect and testing costs. This is information that most companies do not monitor at such level is not monitored and so it is not available. Less accurate but simple and reasonable alternative is to track the number of newly found defects. The following picture shows a typical example of how the number of newly found defects fluctuates during testing.
Initially, the test team founds a lot of new defects. Over time, the number is decreasing and it appears that further testing could not bring any surprises. At this point, an inexperienced test manager terminates the testing thinking that the product is ready. Experienced test manager continues, knowing that the testing normally grabs a second wind, and the number of discovered bugs will begin to grow again. Moreover the test manager will support the growth in discovered defects by variation of tests. However, even without the change of testing approach, the increase is expected. End of testing is advisable after the second or third recess. Of course this is only one of the methods how to determine when to stop, there are others that are based on data gathering and monitoring.
Testing is an investment. Money should be put where it will have the greatest effect
The reason for determining priority of the functionalities is to focus the efforts appropriately. What is important for business is determined by the customer again. Most often in complex systems, the criticality of applications and business processes is well known and to find it, it is sufficient to contact the risk manager or look into contingency plans. Determination of the criticality is based on what the impact would be of the application unavailability or malfunction.
For example, the losses of one day unavailability are estimated, and the ranges are matched with importance labels.
Regarding the importance of individual tests, here you can draw mainly from the priority of use cases. The customer identifies their priorities in the phase of gathering requirements and specifications. To determine priorities he usually think about how many people use the function, or if it comes to contact with critical resources (money, personal data, etc.) or how essential the business processes are, which function handles. However often the customer, especially in smaller projects, identifies the importance based on feelings and according to contractor’s instruction determined 20% of the most important functionalities. Popular and well understandable is assigning three levels of priority:
Critical to quality - Without functions with this label the product cannot be use and it would fail to fulfill even the most basic needs of the customer.
Good enough quality – This label marks needed but not the most critical functions.
Nice to have – This label marks features that are the proverbial icing on the cake. They can make improve the experience or make a work a little bit easier or be just another thing that someone could use in the future.
So back to the start, the answer to all three questions is: "ask the customer what he wants". Because he is the one who invests, and he is the one who will consider the return of the investment.
- How to determine if functionality is critical or not? Is it on the basis of business turnover or something else?
- Consider this situation: around 40 systems, approx. 800 interfaces, and thousands of functionalities. How to effectively address the importance (in terms of being business critical) of individual functionalities and their relationships? Are there any dependency algorithms for this?
- Is it possible to define end of testing based on criteria that if the bug count is below some threshold, you can go into production? Or is it necessary to go to production only when the bug count is zero in all components?
So here are the basic economic principles of testing.
We test because it pays off and as long as it pays off
In business every step that we do we do with the prospect of future profits. If we offer employees various benefits, we're doing it because we want to entice quality employees and reduce costs caused by their volatility. If we invest in quality products, we do it because better quality brings us more money. Whether due to the fact that we do not have to spend large amounts of money on the correction of defects discovered by customers, or because it brings us more satisfied customers.
This is related to the fact that we test as long as it pays off. If the cost of discovering and fixing defects at some stage is bigger than the estimated costs resulting from leaving them there, it makes no sense to continue with testing.
Sometimes the manager can get into a situation where the application is very buggy, testing is effective and it would be good to have two or three months to continue with testing and fixing bugs, but for some reasons, for example legal issues, it is still more cost-effective to deploy the application.
It is customer call to know the overall situation and decide when to deploy the application. This decision should be in hands of manager of that part of company, who will use it or will sell it on mass market. Test manager only provides him with information for making this decision, especially to warn him about the quality risks.
If only 20% of an application is fully and reliably functional, 10% has too many defects to use it and the rest has not been tested yet, what it could mean if application is deployed?
Thus tester is not the one to define the acceptance criteria or making the decision when to release the application into production. This is customers call and he decides how many defects of what severity to tolerate or if he makes the decision not based on quality but based on the situation on market.
We monitor the effectiveness of testing in terms of number of newly discovered defects
As mentioned above, we should test as long as it is financially worthwhile. What does this mean? It means that we need to stop testing in the point when the future cost of finding and correcting defects would be bigger than cost of not to continue to test and risk that the defects will be discovered by customers. But determination of such a moment is not trivial. Apart from the need to work with a probability, it is also necessary to have sufficiently detailed information on the defect and testing costs. This is information that most companies do not monitor at such level is not monitored and so it is not available. Less accurate but simple and reasonable alternative is to track the number of newly found defects. The following picture shows a typical example of how the number of newly found defects fluctuates during testing.
Initially, the test team founds a lot of new defects. Over time, the number is decreasing and it appears that further testing could not bring any surprises. At this point, an inexperienced test manager terminates the testing thinking that the product is ready. Experienced test manager continues, knowing that the testing normally grabs a second wind, and the number of discovered bugs will begin to grow again. Moreover the test manager will support the growth in discovered defects by variation of tests. However, even without the change of testing approach, the increase is expected. End of testing is advisable after the second or third recess. Of course this is only one of the methods how to determine when to stop, there are others that are based on data gathering and monitoring.
Testing is an investment. Money should be put where it will have the greatest effect
The reason for determining priority of the functionalities is to focus the efforts appropriately. What is important for business is determined by the customer again. Most often in complex systems, the criticality of applications and business processes is well known and to find it, it is sufficient to contact the risk manager or look into contingency plans. Determination of the criticality is based on what the impact would be of the application unavailability or malfunction.
For example, the losses of one day unavailability are estimated, and the ranges are matched with importance labels.
Regarding the importance of individual tests, here you can draw mainly from the priority of use cases. The customer identifies their priorities in the phase of gathering requirements and specifications. To determine priorities he usually think about how many people use the function, or if it comes to contact with critical resources (money, personal data, etc.) or how essential the business processes are, which function handles. However often the customer, especially in smaller projects, identifies the importance based on feelings and according to contractor’s instruction determined 20% of the most important functionalities. Popular and well understandable is assigning three levels of priority:
Critical to quality - Without functions with this label the product cannot be use and it would fail to fulfill even the most basic needs of the customer.
Good enough quality – This label marks needed but not the most critical functions.
Nice to have – This label marks features that are the proverbial icing on the cake. They can make improve the experience or make a work a little bit easier or be just another thing that someone could use in the future.
So back to the start, the answer to all three questions is: "ask the customer what he wants". Because he is the one who invests, and he is the one who will consider the return of the investment.
Sunday, March 13, 2011
Importance of Communication in Testing or How to Communicate Your Ideas
This week I attended a CzechTest conference and it was a wonderful experience, as there were many foreign experts that had 10 to 20 years experience in Software Testing. We have a serious lack of those people in Czech Republic, don't we? But I was most impressed by Lloyd Roden, consultant from United Kingdom, who has been also speaker at conferences such as STARWest, STAREast, AsiaSTAR and EuroSTAR.
What was so different about this man was not his ideas or content of his presentation because I have met many intelligent and experienced people like him, but his communication and presentation skills. After his presentation you not only think about the topic, but you also feel energized and want to transform these ideas into your everyday life. And that is probably one of things that make him such a good Test Manager, because in testing (especially test management) it is all about communication.
Rik Marselis, one of other speakers at this conference asked us if we think testing is technical discipline or people discipline. I think it is both.
There are a lot of good testers who understands more computers than people and that is fine as long as they are teamed with testers who are good in communicating with and understanding to stakeholders.
Communication is the most important skill to all managers and people who wants to be heard. And I have not met a tester yet who does not want to be heard and promote good quality in his organization.
Tester is a programer’s best friend.
I teach my students that tester is the programmer’s best friend.
But how could someone who is constantly finding faults in the men work be his best friend?
Only by being tactful, helpful and by providing good information that this man needs.
But hey! That is exactly what it means to be a good tester. Tester provides information? Isn’t this great?
Programmer is just one other person we are helping by providing necessary information. If we care for programmers and help them find and fix the problem quickly by providing relevant information, we really are their good friends.
The same goes for other stakeholders as well: managers, users, business analytics, and so on.
This is the skill we need to hone: how to provide the right information and how to communicate it properly.
And if you want to go now and improve your skills, here is some tips how to start.
What was so different about this man was not his ideas or content of his presentation because I have met many intelligent and experienced people like him, but his communication and presentation skills. After his presentation you not only think about the topic, but you also feel energized and want to transform these ideas into your everyday life. And that is probably one of things that make him such a good Test Manager, because in testing (especially test management) it is all about communication.
Rik Marselis, one of other speakers at this conference asked us if we think testing is technical discipline or people discipline. I think it is both.
There are a lot of good testers who understands more computers than people and that is fine as long as they are teamed with testers who are good in communicating with and understanding to stakeholders.
Communication is the most important skill to all managers and people who wants to be heard. And I have not met a tester yet who does not want to be heard and promote good quality in his organization.
Tester is a programer’s best friend.
I teach my students that tester is the programmer’s best friend.
But how could someone who is constantly finding faults in the men work be his best friend?
Only by being tactful, helpful and by providing good information that this man needs.
But hey! That is exactly what it means to be a good tester. Tester provides information? Isn’t this great?
Programmer is just one other person we are helping by providing necessary information. If we care for programmers and help them find and fix the problem quickly by providing relevant information, we really are their good friends.
The same goes for other stakeholders as well: managers, users, business analytics, and so on.
Tester provides information. Good tester provides the right information.
And if you want to go now and improve your skills, here is some tips how to start.
Tips:
- Improve your bug reports: crash course on reporting bugs
- Improve your influential writing and speaking: coaching testera how to email and influance stakeholders
- Improve your presentation skills for when you speak with managers: Take lessons in presentation skills if you can, or if not read some tips (Presentation Magazine) and watch some video tutorials (Effective Presentation Skills)
- Exercise your skills whenever you can: uTest bug battle
Testing survey
During the last year I was very busy with teaching in University of Economics in Prague (I am teaching graduate student there) and working as test analyst, test methodic or test coordinator. But it was a fruitful time as I am happy to present the results of a testing and quality management survey which I spent several month of gathering data for.
Original study and results are published at studiekvality.vse.cz, but only in Czech language. But you can see Google automatic translation of this page here.
More than 300 Czech IT companies were asked to fill out this questionnaire and in the end we got 84 answering sheets. The results show average knowledge among software testers working in Czech Republic. If you speak with more experienced testers or testers who actively seek knowledge, you will see their opinions and answers would be much different. I present these results as it is without any interpretation or analysis as I will present processed results in journals or at conferences.
It would be good to compare the results to diffrent country. So if you have an opportunity to do this survey in your country, please, DO IT. Do it and send me the results.
If you have any questions please do not hasitate and email me.
Original study and results are published at studiekvality.vse.cz, but only in Czech language. But you can see Google automatic translation of this page here.
More than 300 Czech IT companies were asked to fill out this questionnaire and in the end we got 84 answering sheets. The results show average knowledge among software testers working in Czech Republic. If you speak with more experienced testers or testers who actively seek knowledge, you will see their opinions and answers would be much different. I present these results as it is without any interpretation or analysis as I will present processed results in journals or at conferences.
It would be good to compare the results to diffrent country. So if you have an opportunity to do this survey in your country, please, DO IT. Do it and send me the results.
If you have any questions please do not hasitate and email me.
Saturday, December 5, 2009
Why should we test our software?
This article is not only for testers, but also managers interested in what software testing could bring for them and perhaps does not bring now. Therefore I start with definitions of two terms, which I often use here:
Quality Assurance - planned and systematic processes designed to ensure the suitability of the product for its intended purpose.
Testing - the process of collecting and sorting information obtained through the examination of the product, a part of more general quality assurance.
If we feel that the testing and quality assurance is not given enough attention, and this side of the software development is constantly underestimated, perhaps we are not able to convince others of the importance of testing and its expert management.
So what are the benefits of software testing and quality assurance? Which items should we mention in the presentation of its benefits?
Quality assurance process affects three managerially important areas:
- Marketing
- Risk management
- Reducing costs
The fact that a good product delivered to a customer has a positive impact on reputation of company, while highly unreliable software destroys its reputation, is a simple logical conclusion. It is harder to predict or just realize a level of this influence. Poor product influences reputation in waves, some of them are immediate and fast disappearing, some has negative impact on business for several years.
The first wave is a response of direct customer, who makes decisions about subsequent cooperation based on his contentment.
The second wave is the reaction of end customers. They can cause a number of inconveniences in the case of bad reception of a product. Their constant complaints and bug reporting can cause project to go unexpectedly overprice. So that initially profitable project can become unprofitable project. Furthermore, their negative reception of a product may cause a change in the view of management on the supplier or get among the general public through newspaper and television news.
The third wave is a reaction of current employees and others in the field of software development. Reputations of software companies spread further due to constant staff turnover and persist for many years. Competent employees leave firms that are not able to deliver software in required quality because they do not want to be associated with next failure, or because they are simply not motivated to improve there. Where is no objective criteria of quality there is always a lack of efforts to improve. For the same reasons, good potential employees avoid the firm. They heard negative things about it from former or current employees.
The fourth wave, which affects a software company with the largest delay, is the reluctance of other customers or other software firms to cooperate with it. It is not possible to conceal a lack of efforts to ensure quality and poor processes from employees. If these employees are convinced that they themselves would never give their firm a contract for software development, then they would be less willing to give this company a contract even after several years when they will make such decisions from a position of managers and directors. The same goes for anyone who learned about poorly designed processes from their colleagues and friends.
In these four waves, each project influences according to its quality and impact in a positive or negative direction the company's business.
Testing is a tool to obtain objective information on the status of developed software. This information is the most important input for risk management of software development. Testing is intertwined through the entire process of development. From the very beginning, testing controls compliance of the development with customer's needs, clarity and logic of outputs. It prevents misunderstandings and unnecessary waste of resources by timely informing about the shortcomings and errors. Quality assurance moreover define procedures and monitors developments in terms of providing simplicity, clarity, accuracy, precision, speed and other quality standards. As a result there is both a significant reduction in the likelihood of quality problems during and after development, and a limitation of an impact when a problem occurs.
Testing significantly prevents problems with malicious bugs. A single malicious bug in financial, medical or other critical sector can impact a loss that exceeds the entire budget for the development of this software.
Without testing, there are only two unpleasant ways to reduce the risk associated with software development:
- To find the subcontractor that takes a responsibility to some extent
- If it is possible, to insure against certain types of problems
Although one of the most common management mistakes is sacrificing the quality during cost reduction, it is just effective quality assurance that brings the greatest savings. From the economic point of view, testing should last as long as the estimated average cost of finding and correcting a bug discovered in the next test cycle is less than the average cost of a bug discovered by the customer multiplied by the probability of its discovery. Simply, testing should last as long as it is financially more profitable than not testing. Quality assurance is an investment, so it is useful to monitor its return.
The problem is that it is very difficult to manage quality management with maximum efficiency and minimum cost. Such a task requires experience, feeling and excellent knowledge of a professional. Therefore, when you assure quality, you need to do well.
Set effective testing process is not a one time thing, but it is created by constant tuning based on different reasonably chosen metrics which warn unless everything is alright and where you can monitor the deterioration or improve efficiency.
Understanding the key role of quality management in software development, setting standards, quality control process and having excellent professionals with experience directly in testing is a good basis for any quality assurance department.
Quality Assurance - planned and systematic processes designed to ensure the suitability of the product for its intended purpose.
Testing - the process of collecting and sorting information obtained through the examination of the product, a part of more general quality assurance.
If we feel that the testing and quality assurance is not given enough attention, and this side of the software development is constantly underestimated, perhaps we are not able to convince others of the importance of testing and its expert management.
So what are the benefits of software testing and quality assurance? Which items should we mention in the presentation of its benefits?
Quality assurance process affects three managerially important areas:
- Marketing
- Risk management
- Reducing costs
Marketing
The fact that a good product delivered to a customer has a positive impact on reputation of company, while highly unreliable software destroys its reputation, is a simple logical conclusion. It is harder to predict or just realize a level of this influence. Poor product influences reputation in waves, some of them are immediate and fast disappearing, some has negative impact on business for several years.
The first wave is a response of direct customer, who makes decisions about subsequent cooperation based on his contentment.
The second wave is the reaction of end customers. They can cause a number of inconveniences in the case of bad reception of a product. Their constant complaints and bug reporting can cause project to go unexpectedly overprice. So that initially profitable project can become unprofitable project. Furthermore, their negative reception of a product may cause a change in the view of management on the supplier or get among the general public through newspaper and television news.
The third wave is a reaction of current employees and others in the field of software development. Reputations of software companies spread further due to constant staff turnover and persist for many years. Competent employees leave firms that are not able to deliver software in required quality because they do not want to be associated with next failure, or because they are simply not motivated to improve there. Where is no objective criteria of quality there is always a lack of efforts to improve. For the same reasons, good potential employees avoid the firm. They heard negative things about it from former or current employees.
The fourth wave, which affects a software company with the largest delay, is the reluctance of other customers or other software firms to cooperate with it. It is not possible to conceal a lack of efforts to ensure quality and poor processes from employees. If these employees are convinced that they themselves would never give their firm a contract for software development, then they would be less willing to give this company a contract even after several years when they will make such decisions from a position of managers and directors. The same goes for anyone who learned about poorly designed processes from their colleagues and friends.
In these four waves, each project influences according to its quality and impact in a positive or negative direction the company's business.
Risk management
Testing is a tool to obtain objective information on the status of developed software. This information is the most important input for risk management of software development. Testing is intertwined through the entire process of development. From the very beginning, testing controls compliance of the development with customer's needs, clarity and logic of outputs. It prevents misunderstandings and unnecessary waste of resources by timely informing about the shortcomings and errors. Quality assurance moreover define procedures and monitors developments in terms of providing simplicity, clarity, accuracy, precision, speed and other quality standards. As a result there is both a significant reduction in the likelihood of quality problems during and after development, and a limitation of an impact when a problem occurs.
Testing significantly prevents problems with malicious bugs. A single malicious bug in financial, medical or other critical sector can impact a loss that exceeds the entire budget for the development of this software.
Without testing, there are only two unpleasant ways to reduce the risk associated with software development:
- To find the subcontractor that takes a responsibility to some extent
- If it is possible, to insure against certain types of problems
Reducing costs
Although one of the most common management mistakes is sacrificing the quality during cost reduction, it is just effective quality assurance that brings the greatest savings. From the economic point of view, testing should last as long as the estimated average cost of finding and correcting a bug discovered in the next test cycle is less than the average cost of a bug discovered by the customer multiplied by the probability of its discovery. Simply, testing should last as long as it is financially more profitable than not testing. Quality assurance is an investment, so it is useful to monitor its return.
The problem is that it is very difficult to manage quality management with maximum efficiency and minimum cost. Such a task requires experience, feeling and excellent knowledge of a professional. Therefore, when you assure quality, you need to do well.
How to start?
Set effective testing process is not a one time thing, but it is created by constant tuning based on different reasonably chosen metrics which warn unless everything is alright and where you can monitor the deterioration or improve efficiency.
Understanding the key role of quality management in software development, setting standards, quality control process and having excellent professionals with experience directly in testing is a good basis for any quality assurance department.
Monday, October 5, 2009
Problems with code coverage
One important testing technique is to monitor how the application is covered by tests. We track how many use cases we have covered, how many customer requests, but the most accurate at least in terms of functionality is code coverage. In determining the code coverage it is recorded what code was run during tests and what has not yet been tested.
Simplest but inadequate and misleading is statement coverage – command coverage.
It only checks whether the command line was executed during testing, to cover conditions it is enough to execute it with any of its evaluation.
Formally: The test set T satisfies the criterion of coverage of commands for a given code K if for every command p belonging to the code K there is a test t from the set T, that during execution of test t will be executed command p.
For tester or programmer is not a problem to achieve very high (even 100%) command coverage of a very buggy program without a single failed test, without a single mistake being discovered.
In spite of this metric is misleading, it is unfortunately popular for its simplicity.
More advanced metric is decision coverage (also branch coverage).
Formally: Consider a graph, where the statements are nodes and transitions between the edges. Then consecutive statements are edge and condition is node of which lead two edges, one for true and one for false value. Then the test set T satisfies the decision coverage criterion for the code K if for each edge h of the mentioned graph, there is a test t from test set T that during execution of test t the run goes through edge h.
Simply put, with decision coverage each condition is node with two edges leading from it.
For example:
We have code with three conditions, one of which is nested.
Then decision coverage graph looks like this:
Blue circles are conditions, green ones are statements.
Even here we can achieve full coverage without the discovery of many errors.
The next stage is condition coverage.
Formally: A set of tests T satisfies the condition coverage criterion for the code K if it satisfies the decision coverage criterion and for each part p of each composite condition, there are tests t and u from the set T that during execution of t p is evaluated as true and during execution of u as false.
If you skip the part where condition coverage has to satisfy the decision coverage criterion, then the case of being not covered by decisions but being covered by conditions occurs when for all possible inputs a condition is evaluated always as positive or negative.
Improvement from decision coverage is that condition coverage takes into account all possible evaluations of conditions, not only if it is true or not.
For example:
Let's have complex condition if (a>0 || b>0).
It takes these two tests to reach decision coverage: {a = 5; b = -2}, {a = 0; b = 0}
These tests do not reach condition coverage because b > 0 is false in both tests.
We need two different tests to reach condition coverage: {a = 5; b = 2}, {a = 0; b = 0}
In this version it is more difficult to find a bug that would not be revealed if code has full condition coverage, although certain types of bugs will still remain undetected even in fully covered code.
The highest level of coverage is path coverage. It finds out whether each of the possible paths in each function has been run and therefore it means a very detailed testing.
Formally: A set of tests T satisfies the path coverage criterion for the code K, if it satisfies condition coverage criterion and for each path C linking the input and output node in the graph of code and containing at most n cycles, there is a test t from the test set T that during execution of t the run goes through path C.
While you cannot detect all bugs only from the code, path coverage provides assurance that all options of the run have been tested.
The problem is the practical inapplicability of this coverage, since its complexity causes an exponential increase in the number of tests.
Try these exercises:
Question 1:
How many tests (runs of code) we need to have a method code_coverage covered by statements; decisions; conditions; paths?
Question 2:
How many tests (runs of code) we need to have a method code_coverage2 covered by statements; decisions; conditions; paths?
Simplest but inadequate and misleading is statement coverage – command coverage.
It only checks whether the command line was executed during testing, to cover conditions it is enough to execute it with any of its evaluation.
Formally: The test set T satisfies the criterion of coverage of commands for a given code K if for every command p belonging to the code K there is a test t from the set T, that during execution of test t will be executed command p.
For tester or programmer is not a problem to achieve very high (even 100%) command coverage of a very buggy program without a single failed test, without a single mistake being discovered.
In spite of this metric is misleading, it is unfortunately popular for its simplicity.
More advanced metric is decision coverage (also branch coverage).
Formally: Consider a graph, where the statements are nodes and transitions between the edges. Then consecutive statements are edge and condition is node of which lead two edges, one for true and one for false value. Then the test set T satisfies the decision coverage criterion for the code K if for each edge h of the mentioned graph, there is a test t from test set T that during execution of test t the run goes through edge h.
Simply put, with decision coverage each condition is node with two edges leading from it.
For example:
We have code with three conditions, one of which is nested.
Then decision coverage graph looks like this:
Blue circles are conditions, green ones are statements.
Even here we can achieve full coverage without the discovery of many errors.
The next stage is condition coverage.
Formally: A set of tests T satisfies the condition coverage criterion for the code K if it satisfies the decision coverage criterion and for each part p of each composite condition, there are tests t and u from the set T that during execution of t p is evaluated as true and during execution of u as false.
If you skip the part where condition coverage has to satisfy the decision coverage criterion, then the case of being not covered by decisions but being covered by conditions occurs when for all possible inputs a condition is evaluated always as positive or negative.
Improvement from decision coverage is that condition coverage takes into account all possible evaluations of conditions, not only if it is true or not.
For example:
Let's have complex condition if (a>0 || b>0).
It takes these two tests to reach decision coverage: {a = 5; b = -2}, {a = 0; b = 0}
These tests do not reach condition coverage because b > 0 is false in both tests.
We need two different tests to reach condition coverage: {a = 5; b = 2}, {a = 0; b = 0}
In this version it is more difficult to find a bug that would not be revealed if code has full condition coverage, although certain types of bugs will still remain undetected even in fully covered code.
The highest level of coverage is path coverage. It finds out whether each of the possible paths in each function has been run and therefore it means a very detailed testing.
Formally: A set of tests T satisfies the path coverage criterion for the code K, if it satisfies condition coverage criterion and for each path C linking the input and output node in the graph of code and containing at most n cycles, there is a test t from the test set T that during execution of t the run goes through path C.
While you cannot detect all bugs only from the code, path coverage provides assurance that all options of the run have been tested.
The problem is the practical inapplicability of this coverage, since its complexity causes an exponential increase in the number of tests.
Try these exercises:
Question 1:
How many tests (runs of code) we need to have a method code_coverage covered by statements; decisions; conditions; paths?
Question 2:
How many tests (runs of code) we need to have a method code_coverage2 covered by statements; decisions; conditions; paths?
Subscribe to:
Posts (Atom)