Tuesday, August 2, 2011
MobileTechnology: App or game not getting pause
MobileTechnology: App or game not getting pause: "//Application or game not getting pause in some devices //hideNotify() and showNotify() not calling.....on interruption... Hi If the a..."
MobileTechnology: App or game not getting pause
MobileTechnology: App or game not getting pause: "//Application or game not getting pause in some devices //hideNotify() and showNotify() not calling.....on interruption... Hi If the a..."
Thursday, July 1, 2010
Wednesday, June 23, 2010
A test methodology for an effective regression testing
A test methodology for an effective regression testing
1. Introduction
There are several definitions and perspectives exist in the industry. The purpose of this article is to bring the best breed of those definitions and methodologies based on the personal experience of the author in software product companies.
Definition: Regression testing is selective retesting of the system; executed with an objective to ensure the bug fixes work and those bug fixes have not caused any un-intended effects in the system.
2. Types of regression testing
There are two types of regression testing that are proposed here, even though it is not being practiced or popular.
A "final regression testing" is done to validate the gold master builds, and "Regression testing" is done to validate the product and failed test cases between system test cycles.
The final regression test cycle is conducted on an "unchanged build for a period of x days" or for a period, which was agreed as the "cook-time" for release. The product is continuously exercised for the complete duration of this cook-time. Some of the test cases are even repeated to find out whether there are failures in the final product that will reach the customer. All the bug fixes for the release should have been completed for the build used for the final regression test cycle. The final regression test cycle is more critical than any other type or phase of testing, as this is the only testing which ensures "the same build of the product that was tested reaches the customer".
A normal regression testing can use the builds for a time period that is needed for the test cases to be executed. However unchanged build is highly recommended for each cycle of regression testing.
3. Selecting test cases for regression testing
It was found from industry data that good number of the defects reported by customers were due to last minute bug fixes creating side effects and hence selecting the test case for regression testing is an art and not that easy.
The selection of test cases for regression testing
a. Requires knowledge on the bug fixes and how it affect the system
b. Includes the area of frequent defects
c. Includes the area which has undergone many/recent code changes
d. Includes the area which is highly visible to the users
e. Includes the core features of the product which are mandatory requirements of the customer
Selection of test cases for regression testing depends more on the criticality of bug fixes than the criticality of the defect itself. A minor defect can result in major side effect and a bug fix for an Extreme defect can have no or a just a minor side effect. So the test engineer needs to balance these aspects for selecting the test cases for regression testing.
When selecting the test cases we should not select only those test cases which are bound to fail and those test cases which has no or less relevance to the bug fixes. You need to select more positive test cases than negative test cases for final regression test cycle as this may create some confusion and unexpected heat. It is also recommended that the regular test cycles before regression testing should have right mix of both positive and negative test cases. Negative test cases are those test cases which are introduced newly with intent to break the system.
It is noticed that several companies have "constant test cases set" for regression testing and they are executed irrespective of the number and type of bug fixes. Sometimes this approach may not find all side effects in the system and in some cases it may be observed that the effort spend on executing test cases for regression testing can be minimized if some analysis is done to find out what test cases are relevant and what are not.
It is a good approach to plan and act for regression testing from the beginning of project before the test cycles. One of the ideas is to classify the test cases into various Priorities based on importance and customer usage. Here it is suggested the test cases be classified into three categories;
1. Priority-0 – Sanity test cases which checks basic functionality and are run for pre-system acceptance and when product goes thru major change. These test cases deliver a very high project value to both engineering dept and to customers.
2. Priority-1 – Uses the basic and normal setup and these test cases deliver high project value to both engineering and to customers.
3. Priority-2 – These test cases deliver moderate project value. Executed part of ST cycle and selected for regression testing on need basis.
There could be several right approaches to regression testing which needs to be decided on "case to case" basis;
• Case 1: If the criticality and impact of the bug fixes are LOW, then it is enough a test engineer selects a few test cases from TCDB and executes them. These test cases can fall under any Priority (0, 1 or 2).
• Case 2: If the criticality and the impact of the bug fixes are Medium, then we need to execute all Priority-0 and Priority-1 test cases. If bug fixes need additional test cases from Priority-2, then those test cases can also selected and used for regression testing. Selecting Priority-2 test cases in this case is desirable but not a must.
• Case 3: If the criticality and impact of the bug fixes are High, then we need to execute all Priority-0, Priority-1 and carefully selected Priority-2 test cases.
• Case 4: One can also go thru the complete log of changes happened (can be obtained from Configuration management engineer) because of bug fixes and select the test cases to conduct regression testing. This is an elaborate process but can give very good results.
This is illustrated in the picture below;
4. Resetting the test cases for regression testing
In a big product release involving several rounds of testing, it is very important to note down what test cases were executed with what build and related information. This is called test case result history. In many organizations not all types of testing and all test cases were repeated for each cycle. In such cases resetting the test cases become very critical for the success of regression testing. Resetting a test case is nothing but setting a flag called NOTRUN or EXECUTE AGAIN with “zero base” thinking.
RESET of test case, are not expected to be done often. Resetting of the test cases needs to be done with following considerations;
a. When there is a major change in the product
b. When there is a change in the build procedure which affect the product
c. Large release cycle where some test cases were not executed for a long time
d. You are in the final regression test cycle with a few selected test cases
e. Where there is a situation the expected results of the test cases could be quite different from previous cycles
When the above guidelines are not met, you may want to RERUN the test cases rather than resetting the results of the test cases. There are only few differences between RERUN and RESET states in test cases, either way the test cases are executed but in case of RESET one has to think “zero base” and expect different result than what was obtained in earlier cycles and therefore those test cases affect the completion rate of testing. In case of RERUN the management need not worry about completion rate as those test cases can be considered complete except for a formality check and are expected to give same results.
To give you an example, if there is a change in Installation of a product, which does not affect the product functionality, then the change can be tested independently by rerunning some test cases and we don't have to RESET the test cases.
RESET is also decided based on how stable the functionalities are. If you are in Priority-1 and have reached a stage of comfort level on Priority-0 (say for example more than 95% pass rate) then you don't RESET Priority-0 test cases unless there is a major change. This is true with Priority-1 test cases when you are in Priority-2 test phase.
4.1 Pre-system test cycle phase
For pre-system acceptance only Priority-0 test cases are used. For each build that is entering the system test, the build number is selected and all test cases in Priority-0 are reset to NOT RUN. The system test cycle starts only if all pre-system test cases (Priority-0) pass.
4.2 System test cycle – Priority-1 testing phase
After pre-system acceptance is over, Priority-1 test cases are executed. Priority-1 testing can use multiple builds. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are high. A RESET procedure during this phase may affect all Priority-0 and Priority-1 test cases and these test cases are reset to NOTRUN in TCDB.
4.3 System test cycle – Priority-2 testing phase
Priority-2 testing starts after all test cases in Priority-1 are executed with an acceptable pass % as defined in test plan. In this phase several builds are used. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are very high. A RESET procedure during this phase may affect Priority-0, Priority-1 and Priority-2 test cases.
4.4 In what way regression testing is related to above three phases?
Regression testing is normally done after Priority-2 testing or for the next release involving only few changes. Resetting test cases during the above phases are not called as regression testing as in my assumption regression comes into picture only after the product is stable. A testing for a release can be decided either by saying a regression testing is sufficient or we can do all phases of testing starting from Priority-0 to Priority-2.
A regression testing for a release can use test cases from all priorities (as mentioned before). A regression testing involving multiple priorities of test cases also needs the test cases executed in strict order i.e. Priority-0 test cases are executed first, Priority-1 next and Priority-2 test cases.
4.5 Why we need to RESET the test cases?
Regression testing uses good number of test cases, which would have been executed already and associated with some results and assumptions on the result. A RESET procedure makes them to NOTRUN so that it gives a clear picture about how much of testing is still remaining, and reflect the results of the regression testing on Zero base.
If test cases are not RESET, then the test engineers tend to report a completion rate and other results based on previous builds. This is because of the basic assumption that multiple builds can be used in each phase of the testing and a gut-feel that if something passed in the past builds, it will pass in future builds also. Regression testing doesn't go with an assumption that "Future is an extension of the past".
5. Concluding the results of a regression testing
Regression testing uses only one build for testing (if not, it is strongly recommended). It is expected that all 100% of those test cases pass using the same build. In situations where the pass % is not 100, the test manager can look at the previous results of the test case to conclude the expected result;
a. If the result of a particular test case was PASS using the previous builds and FAIL in the current build, then regression failed. We need to get a new build and start the testing from scratch after resetting the test cases.
b. If the result of a particular test case was a FAIL using the previous builds and a PASS in the current build, then it is easy to assume the bug fixes worked.
c. If the result of a particular test case was a FAIL using the previous builds and a FAIL in the current build and if there are no bug fixes for this particular test case, it may mean that the result of this test case shouldn't be considered for the pass %. This may also mean that such test cases shouldn't be selected for regression.
d. If the result of a particular test case is FAIL using the previous builds but works with a documented workaround and
a. if you are satisfied with the workaround then it should considered as PASS for both system test cycle and regression test cycle
b. If you are not satisfied with the workaround then it should be considered as FAIL for a system test cycle but can be considered as PASS for regression test cycle.
This is illustrated in the table below;
Current result from regression Previous result(s) Conclusion Remarks
FAIL PASS FAIL Need to improve the regression process and code reviews
PASS FAIL PASS This is the expected result of a good regression to say bug fixes work properly
FAIL FAIL FAIL Need to analyze why bug fixes are not working. “Is it a wrong fix?”
PASS (with a work around) FAIL Analyze the workaround and if satisfied mark result as PASS Workarounds also need a good review as workarounds can also create side effects
PASS PASS PASS This pattern of results gives comfort feeling, that there are no side effects due to bug fixes
Regression test guidelines for patch/upgrade releases
The regression guidelines are applicable for both cases where
a. You are doing a major release of a product, executed all system test cycles and planning a regression test cycle for bug fixes.
b. You are doing a minor release of a product having only bug fixes, and you are planning for regression test cycles to take care of those bug fixes.
There can be multiple cycles of regression testing that can be planned for each release. This is because bug fixes may come in phases or some bug fixes may not work as expected resulting in one more regression cycle.
Mapping Test cases to defect fix
When assigning a “Fail” result to a test case, it is a good practice to enter the defect number(s) along so that you will know what test cases to be executed when a bug fix arrives. Please note that there can be multiple defects that can come out of a particular test case and a particular defect can affect more than one test case.
Even though, it is easy to do the mapping between test cases and defects, using these mechanisms, the test cases that are to be executed for taking care of side effects of bug fixes, may remain as a manual process as this requires knowledge and several other perspectives discussed earlier in this doc.
1. Introduction
There are several definitions and perspectives exist in the industry. The purpose of this article is to bring the best breed of those definitions and methodologies based on the personal experience of the author in software product companies.
Definition: Regression testing is selective retesting of the system; executed with an objective to ensure the bug fixes work and those bug fixes have not caused any un-intended effects in the system.
2. Types of regression testing
There are two types of regression testing that are proposed here, even though it is not being practiced or popular.
A "final regression testing" is done to validate the gold master builds, and "Regression testing" is done to validate the product and failed test cases between system test cycles.
The final regression test cycle is conducted on an "unchanged build for a period of x days" or for a period, which was agreed as the "cook-time" for release. The product is continuously exercised for the complete duration of this cook-time. Some of the test cases are even repeated to find out whether there are failures in the final product that will reach the customer. All the bug fixes for the release should have been completed for the build used for the final regression test cycle. The final regression test cycle is more critical than any other type or phase of testing, as this is the only testing which ensures "the same build of the product that was tested reaches the customer".
A normal regression testing can use the builds for a time period that is needed for the test cases to be executed. However unchanged build is highly recommended for each cycle of regression testing.
3. Selecting test cases for regression testing
It was found from industry data that good number of the defects reported by customers were due to last minute bug fixes creating side effects and hence selecting the test case for regression testing is an art and not that easy.
The selection of test cases for regression testing
a. Requires knowledge on the bug fixes and how it affect the system
b. Includes the area of frequent defects
c. Includes the area which has undergone many/recent code changes
d. Includes the area which is highly visible to the users
e. Includes the core features of the product which are mandatory requirements of the customer
Selection of test cases for regression testing depends more on the criticality of bug fixes than the criticality of the defect itself. A minor defect can result in major side effect and a bug fix for an Extreme defect can have no or a just a minor side effect. So the test engineer needs to balance these aspects for selecting the test cases for regression testing.
When selecting the test cases we should not select only those test cases which are bound to fail and those test cases which has no or less relevance to the bug fixes. You need to select more positive test cases than negative test cases for final regression test cycle as this may create some confusion and unexpected heat. It is also recommended that the regular test cycles before regression testing should have right mix of both positive and negative test cases. Negative test cases are those test cases which are introduced newly with intent to break the system.
It is noticed that several companies have "constant test cases set" for regression testing and they are executed irrespective of the number and type of bug fixes. Sometimes this approach may not find all side effects in the system and in some cases it may be observed that the effort spend on executing test cases for regression testing can be minimized if some analysis is done to find out what test cases are relevant and what are not.
It is a good approach to plan and act for regression testing from the beginning of project before the test cycles. One of the ideas is to classify the test cases into various Priorities based on importance and customer usage. Here it is suggested the test cases be classified into three categories;
1. Priority-0 – Sanity test cases which checks basic functionality and are run for pre-system acceptance and when product goes thru major change. These test cases deliver a very high project value to both engineering dept and to customers.
2. Priority-1 – Uses the basic and normal setup and these test cases deliver high project value to both engineering and to customers.
3. Priority-2 – These test cases deliver moderate project value. Executed part of ST cycle and selected for regression testing on need basis.
There could be several right approaches to regression testing which needs to be decided on "case to case" basis;
• Case 1: If the criticality and impact of the bug fixes are LOW, then it is enough a test engineer selects a few test cases from TCDB and executes them. These test cases can fall under any Priority (0, 1 or 2).
• Case 2: If the criticality and the impact of the bug fixes are Medium, then we need to execute all Priority-0 and Priority-1 test cases. If bug fixes need additional test cases from Priority-2, then those test cases can also selected and used for regression testing. Selecting Priority-2 test cases in this case is desirable but not a must.
• Case 3: If the criticality and impact of the bug fixes are High, then we need to execute all Priority-0, Priority-1 and carefully selected Priority-2 test cases.
• Case 4: One can also go thru the complete log of changes happened (can be obtained from Configuration management engineer) because of bug fixes and select the test cases to conduct regression testing. This is an elaborate process but can give very good results.
This is illustrated in the picture below;
4. Resetting the test cases for regression testing
In a big product release involving several rounds of testing, it is very important to note down what test cases were executed with what build and related information. This is called test case result history. In many organizations not all types of testing and all test cases were repeated for each cycle. In such cases resetting the test cases become very critical for the success of regression testing. Resetting a test case is nothing but setting a flag called NOTRUN or EXECUTE AGAIN with “zero base” thinking.
RESET of test case, are not expected to be done often. Resetting of the test cases needs to be done with following considerations;
a. When there is a major change in the product
b. When there is a change in the build procedure which affect the product
c. Large release cycle where some test cases were not executed for a long time
d. You are in the final regression test cycle with a few selected test cases
e. Where there is a situation the expected results of the test cases could be quite different from previous cycles
When the above guidelines are not met, you may want to RERUN the test cases rather than resetting the results of the test cases. There are only few differences between RERUN and RESET states in test cases, either way the test cases are executed but in case of RESET one has to think “zero base” and expect different result than what was obtained in earlier cycles and therefore those test cases affect the completion rate of testing. In case of RERUN the management need not worry about completion rate as those test cases can be considered complete except for a formality check and are expected to give same results.
To give you an example, if there is a change in Installation of a product, which does not affect the product functionality, then the change can be tested independently by rerunning some test cases and we don't have to RESET the test cases.
RESET is also decided based on how stable the functionalities are. If you are in Priority-1 and have reached a stage of comfort level on Priority-0 (say for example more than 95% pass rate) then you don't RESET Priority-0 test cases unless there is a major change. This is true with Priority-1 test cases when you are in Priority-2 test phase.
4.1 Pre-system test cycle phase
For pre-system acceptance only Priority-0 test cases are used. For each build that is entering the system test, the build number is selected and all test cases in Priority-0 are reset to NOT RUN. The system test cycle starts only if all pre-system test cases (Priority-0) pass.
4.2 System test cycle – Priority-1 testing phase
After pre-system acceptance is over, Priority-1 test cases are executed. Priority-1 testing can use multiple builds. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are high. A RESET procedure during this phase may affect all Priority-0 and Priority-1 test cases and these test cases are reset to NOTRUN in TCDB.
4.3 System test cycle – Priority-2 testing phase
Priority-2 testing starts after all test cases in Priority-1 are executed with an acceptable pass % as defined in test plan. In this phase several builds are used. In this phase the test cases are RESET only if the criticality and impact of the bug fixes and feature additions are very high. A RESET procedure during this phase may affect Priority-0, Priority-1 and Priority-2 test cases.
4.4 In what way regression testing is related to above three phases?
Regression testing is normally done after Priority-2 testing or for the next release involving only few changes. Resetting test cases during the above phases are not called as regression testing as in my assumption regression comes into picture only after the product is stable. A testing for a release can be decided either by saying a regression testing is sufficient or we can do all phases of testing starting from Priority-0 to Priority-2.
A regression testing for a release can use test cases from all priorities (as mentioned before). A regression testing involving multiple priorities of test cases also needs the test cases executed in strict order i.e. Priority-0 test cases are executed first, Priority-1 next and Priority-2 test cases.
4.5 Why we need to RESET the test cases?
Regression testing uses good number of test cases, which would have been executed already and associated with some results and assumptions on the result. A RESET procedure makes them to NOTRUN so that it gives a clear picture about how much of testing is still remaining, and reflect the results of the regression testing on Zero base.
If test cases are not RESET, then the test engineers tend to report a completion rate and other results based on previous builds. This is because of the basic assumption that multiple builds can be used in each phase of the testing and a gut-feel that if something passed in the past builds, it will pass in future builds also. Regression testing doesn't go with an assumption that "Future is an extension of the past".
5. Concluding the results of a regression testing
Regression testing uses only one build for testing (if not, it is strongly recommended). It is expected that all 100% of those test cases pass using the same build. In situations where the pass % is not 100, the test manager can look at the previous results of the test case to conclude the expected result;
a. If the result of a particular test case was PASS using the previous builds and FAIL in the current build, then regression failed. We need to get a new build and start the testing from scratch after resetting the test cases.
b. If the result of a particular test case was a FAIL using the previous builds and a PASS in the current build, then it is easy to assume the bug fixes worked.
c. If the result of a particular test case was a FAIL using the previous builds and a FAIL in the current build and if there are no bug fixes for this particular test case, it may mean that the result of this test case shouldn't be considered for the pass %. This may also mean that such test cases shouldn't be selected for regression.
d. If the result of a particular test case is FAIL using the previous builds but works with a documented workaround and
a. if you are satisfied with the workaround then it should considered as PASS for both system test cycle and regression test cycle
b. If you are not satisfied with the workaround then it should be considered as FAIL for a system test cycle but can be considered as PASS for regression test cycle.
This is illustrated in the table below;
Current result from regression Previous result(s) Conclusion Remarks
FAIL PASS FAIL Need to improve the regression process and code reviews
PASS FAIL PASS This is the expected result of a good regression to say bug fixes work properly
FAIL FAIL FAIL Need to analyze why bug fixes are not working. “Is it a wrong fix?”
PASS (with a work around) FAIL Analyze the workaround and if satisfied mark result as PASS Workarounds also need a good review as workarounds can also create side effects
PASS PASS PASS This pattern of results gives comfort feeling, that there are no side effects due to bug fixes
Regression test guidelines for patch/upgrade releases
The regression guidelines are applicable for both cases where
a. You are doing a major release of a product, executed all system test cycles and planning a regression test cycle for bug fixes.
b. You are doing a minor release of a product having only bug fixes, and you are planning for regression test cycles to take care of those bug fixes.
There can be multiple cycles of regression testing that can be planned for each release. This is because bug fixes may come in phases or some bug fixes may not work as expected resulting in one more regression cycle.
Mapping Test cases to defect fix
When assigning a “Fail” result to a test case, it is a good practice to enter the defect number(s) along so that you will know what test cases to be executed when a bug fix arrives. Please note that there can be multiple defects that can come out of a particular test case and a particular defect can affect more than one test case.
Even though, it is easy to do the mapping between test cases and defects, using these mechanisms, the test cases that are to be executed for taking care of side effects of bug fixes, may remain as a manual process as this requires knowledge and several other perspectives discussed earlier in this doc.
Monday, June 21, 2010
Advance Interview question for software testing.
Below is some advanced level/ Test lead level Software Testing Questions. Answer now and Judge yourself.
- Advantages & drawbacks of path coverage metric of software testing?
- Advantages & drawbacks of decision coverage metric of software testing?
- Drawbacks of statement coverage metric of software testing?
- Main advantages of statement coverage metric of software testing?
- Compare Automation Testing with Manual Testing.
- Define Test Scenario?
- Is there any automated tool going to replace the testers?
- Are automated tool Replace Manual Testers?
- Basic assumptions behind coverage analysis?
- When to automate tests?
- If you are a Lead Automation engineer then what questions you would ask to yourself and your manager while deciding to automate the tests?
- What is :
"Key Word Driven" Method of Testing?
"Test Plan Driven" Method of Testing?
Comparison Testing?
Parallel Testing?
Automated Testing?
a Data Flow Diagram (DFD)?
a Traceability Matrix?
Defect Leakage?
Configuration Management?
Statement Coverage In software testing,?
multiple condition coverage metric of software testing?
the difference between code coverage analysis & test coverage analysis?
the difference between Structural testing & functional testing?
Probe Testing?
the purpose of Automated Test Tools?
Difference between Retest and Regression Testing?
the best sequence of coverage goals as implementation strategy?
Ans 1: Advantages of path coverage metric of software testing:
Path coverage requires extremely thorough testing.
Disadvantages of path coverage metric of software testing:
1) Since loops introduce an unbounded number of paths, this metric considers only a limited number of looping possibilities.
2) The number of paths is exponential to the number of branches. For example, a function containing 10 if-statements has 1024 paths to test. Adding just one
more if-statement doubles the count to 2048.
3) Many paths are impossible to exercise due to relationships of data.
Ans 2: Decision coverage has the main advantage of simplicity & is free from many problems of statement coverage.
Disadvantage of decision coverage is that this metric ignores branches within Boolean expressions which occur due to short-circuit operators.
Ans 3: Drawbacks of statement coverage metric of software testing:
1) It is insensitive to some of the control structures.
2) It does not report whether loops reach their termination condition - only whether the loop body was executed. With C, C++, and Java, this limitation
affects loops that contain break statements.
3) It is completely insensitive to the logical operators (|| and &&).
4) It cannot distinguish consecutive switch labels.
Ans 4: Advantages of statement coverage metric of software testing
1) The main advantage of statement coverage metric is that it can be applied directly to object code and does not require processing source code. Usually the
Performance profilers use this metric.
2) Bugs are evenly distributed through code; therefore the percentage of executable statements covered reflects the percentage of faults discovered.
Ans 5: Automation Testing versus Manual Testing Guidelines:
I met with my team’s automation experts a few weeks back to get their input on when to automate and when to manually test. The general rule of thumb has always been to use common sense. If you’re only going to run the test one or two times or the test is really expensive to automation, it is most likely a manual test. But then again, what good is saying “use common sense” when you need to come up with deterministic set of guidelines on how and when to automate?
Pros of Automation
• If you have to run a set of tests repeatedly, automation is a huge win for you
• It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner
• It gives you the ability to run automation in mainstream scenarios to catch regressions in a timely manner (see What is a Nightly)
• Aids in testing a large test matrix (different languages on different OS platforms). Automated tests can be run at the same time on different machines, whereas the manual tests would have to be run sequentially.
Cons of Automation
• It costs more to automate. Writing the test cases and writing or configuring the automate framework you’re using costs more initially than running the test manually.
• Can’t automate visual references, for example, if you can’t tell the font colour via code or the automation tool, it is a manual test.
Pros of Manual
• If the test case only runs twice a coding milestone, it most likely should be a manual test. Less cost than automating it.
• It allows the tester to perform more ad-hoc (random testing). In my experiences, more bugs are found via ad-hoc than via automation. And, the more time a tester spends playing with the feature, the greater the odds of finding real user bugs.
Cons of Manual
• Running tests manually can be very time consuming
• Each time there is a new build, the tester must rerun all required tests - which after a while would become very mundane and tiresome.
Other deciding factors:
• What you automate depends on the tools you use. If the tools have any limitations, those tests are manual.
• Is the return on investment worth automating? Is what you get out of automation worth the cost of setting up and supporting the test cases, the automation framework, and the system that runs the test cases?
Ans: Test Scenario is the user workflow in the application.
Example: Checking Mail in Gmail is a scenario, where user login, check the mail in inbox and then logoff. This application can have 2 different test case one for login and other one for inbox.
So Test Scenario can consist of different test cases.
Ans: The simple answer to the question ‘Can automated testing replace all manual testing’� is ‘No.’�
Automated functional tests can be used for regression testing (which is a small part of the overall testing effort). If an organization is running the same manual regression tests repeatedly, then the automated tests can replace some of that effort, but they also add the effort to maintain the tests, which is sometimes more than the work required to just run the tests manually. When I say some of the effort, I mean that test failures from an automated test run still must be analyzed manually. Also, any part of the process of provisioning and setting up the machine to run the tests, kicking off the test run, and babysitting it along the way that isn’t automated will still require manual attention.
Ans: The basic assumptions behind coverage analysis tell us about the strengths and limitations of this testing technique. Some fundamental assumptions are listed
below.
* Bugs relate to control flow and you can expose Bugs by varying the control flow [Beizer1990 p.60]. For example, a programmer wrote "if (c)" rather than
"if (!c)".
* You can look for failures without knowing what failures might occur and all tests are reliable, in that successful test runs imply program correctness
[Morell1990]. The tester understands what a correct version of the program would do and can identify differences from the correct behaviour.
* Other assumptions include achievable specifications, no errors of omission, and no unreachable code.
Clearly, these assumptions do not always hold. Coverage analysis exposes some plausible bugs but does not come close to exposing all classes of bugs.
Coverage analysis provides more benefit when applied to an application that makes a lot of decisions rather than data-centric applications, such as a
database application.
Ans: For a test automation the entry criteria are:
* Availability of a stable application under test (around some 80% of test cases passed).
* Availability of automation test tool with the required Add-ins and Patches.
* Availability of stable and controlled test environment.
* Automation test strategy sign-off
-scope (type of tests)
-functionalities (features to be automated)
-assumptions
-features to be automated
* SIT or UAT sign off
* Signed off manual test cases to be provided
* Availability of stable test bed.
Ans : You are a Lead Automation engineer then what questions you would ask to yourself and your manager while deciding to automate the tests
Best approach would be to raise the following questions:
1) Automating this test and running it once will cost more than simply running it manually once. How much more?
2. An automated test has a finite lifetime, during which it must recoup that additional cost. Is this test likely to die sooner or later? What events are likely to end it?
3. During its lifetime, how likely is this test to find additional bugs (beyond whatever bugs it found the first time it ran)? How does this uncertain benefit balance against the cost of automation?
4. Return of Investment.
Ans: Keyword driven testing:
This requires the development of data tables and keywords, independent of the test automation tool used to execute them and the test script code that "drives" the application-under-test and the data. Keyword-driven tests look very similar to manual test cases. In a keyword-driven test, the functionality of the application-under-test is documented in a table as well as in step-by-step instructions for each test. In this method, the entire process is data-driven, including functionality
The merits of the Keyword Driven Testing are as follows,
- The Detail Test Plan can be written in Spreadsheet format containing all input and verification data.
- If "utility" scripts can be created by someone proficient in the automated tool’s Scripting language prior to the Detail Test Plan being written, then the tester can use the Automated Test Tool immediately via the "spreadsheet-input" method, without needing to learn the Scripting language.
- The tester need only learn the "Key Words" required, and the specific format to use within the Test Plan. This allows the tester to be productive with the test tool very quickly, and allows more extensive training in the test tool to be scheduled at a more convenient time.
Demerits of keyword driven testing
The demerits of the Keyword Driven Testing are as follows:
- Development of "customized" (Application-Specific) Functions and Utilities requires proficiency in the tool’s Scripting language. (Note that this is also true for any method)
- If application requires more than a few "customized" Utilities, this will require the tester to learn a number of "Key Words" and special formats. This can be time-consuming, and may have an initial impact on Test Plan Development. Once the testers get used to this, however, the time required to produce a test case is greatly improved.
Ans: The architecture of the “Test Plan Driven” method appears similar to that of the “Functional Decomposition” method, but in fact, they are substantially
different:
* Driver Script
* Performs initialization, if required;
* Calls the Application-Specific “Controller” Script, passing to it the file-names of the Test Cases (which have been saved from the spreadsheets as a
“tab-delimited” files);
* The “Controller” Script
* Reads and processes the file-name received from Driver;
* Matches on “Key Words” contained in the input-file
* Builds a parameter-list from the records that follow;
* Calls “Utility” scripts associated with the “Key Words”, passing the created parameter-list;
* Utility Scripts
* Process input parameter-list received from the “Controller” script;
* Perform specific tasks (e.g. press a key or button, enter data, verify data, etc.), calling “User Defined Functions” if required;
* Report any errors to a Test Report for the test case;
* Return to “Controller” script;
* User Defined Functions
* General and Application-Specific functions may be called by any of the above script-types in order to perform specific tasks;
Advantages
This method has all of the advantages of the “Functional Decomposition” method, as well as the following:
* The Detail Test Plan can be written in Spreadsheet format containing all input and verification data. Therefore the tester only needs to write this
once, rather than, for example, writing it in Word, and then creating input and verification files as is required by the “Functional Decomposition” method.
* Test Plan does not necessarily have to be written using MS Excel. Any format can be used from which either “tab-delimited” or “comma-delimited” files
can be saved (e.g. Access Database, etc.).
* If “utility” scripts can be created by someone proficient in the Automated tool’s Scripting language prior to the Detail Test Plan being written, then
the tester can use the Automated Test Tool immediately via the "spreadsheet-input" method, without needing to learn the Scripting language. The tester need
only learn the “Key Words” required, and the specific format to use within the Test Plan. This allows the tester to be productive with the test tool very
quickly, and allows more extensive training in the test tool to be scheduled at a more convenient time.
* If Detailed Test Cases already exists in some other format, it is not difficult to translate these into the “spreadsheet” format.
* After a number of “generic” Utility scripts have already been created for testing an application, we can usually re-use most of these if we need to
test another application. This would allow the organization to get their automated testing “up and running” (for most applications) within a few days,
rather than weeks.
Disadvantages
* Development of “customized” (Application-Specific) Functions and Utilities requires proficiency in the tool’s Scripting language. Note that this is
also true of the “Functional Decomposition” method, and, frankly of any method used including “Record/Playback”.
* If application requires more than a few “customized” Utilities, this will require the tester to learn a number of “Key Words” and special formats.
This can be time-consuming, and may have an initial impact on Test Plan Development. Once the testers get used to this, however, the time required to
produce a test case is greatly improved.
Ans: In comparison testing, we compare the old application with new application and see whether the new application is working better than the old application or not.
Comparison Testing means comparing your software with the better one or your Competitor.
While comparison Testing we basically compare the Performance of the software.
For ex If you have to do Comparison Testing of PDF converter (Desktop Based Application) then you will compare your software with your Competitor on the
basis of:-
1.Speed of Conversion PDF file into Word.
2.Quality of converted file.
Ans: Parallel Testing
Parallel/audit testing is a type of testing where the tester reconciles the output of the new system to the output of the current system, in order to verify the new system operates correctly.
OR
Comparing our Product / Application build with other products existing in the market. Parallel Testing is also known as Comparative Testing / Competetive Testing.
Testing newly developed system and compare the results with already existing system to check any discrepancy between them.
Ans: Automated Testing:
Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.
Test automation can be expensive, and it is usually employed in combination with manual exploratory testing. It can be made cost-effective in the longer term
though, especially in regression testing. One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation, but research continues into a variety of methodologies for doing so.
What to automate, when to automate, or even whether one really needs automation are crucial decisions which the testing (or development) team has to take.
Selecting the correct features of the product for automation largely decides the success of the automation. Unstable features or the features which are undergoing changes should be avoided.
Ans: Data Flow Diagram (DFD)
Data Flow Diagram is a graphical representation of the "flow" of data through an information system. A data flow diagram can also be used for the visualization of data processing. It is common practice for a designer to draw a context-level DFD first which shows the interaction between the system and outside entities.
The process model is typically used in structured analysis and design methods. Also called a data flow diagram (DFD), it shows the flow of information through a system. Each process transforms inputs into outputs.
The model generally starts with a context diagram showing the system as a single process connected to external entities outside of the system boundary. This
process explodes to a lower level DFD that divides the system into smaller parts and balances the flow of information between parent and child diagrams. Many
diagram levels may be needed to express a complex system. Primitive processes, those that don't explode to a child diagram, are usually described in a
connected textual specification.
Ans: Traceability Matrix
Traceability means that we would like to be able to trace back and forth how and where any work product fulfils the directions of the preceding product.
The matrix deals with the where, while the how we have to do our self, once we know the where.
A traceability matrix is created by associating requirements with the work products that satisfy them. Tests are associated with the requirements on which
they are based and the product tested to meet the requirement.
There can be more things included in a traceability matrix than shown. In traceability, the relationship of driver to satisfier can be one-to-one, one-to-many, many-to-one, or many-to-many.
Traceability requires unique identifiers for each requirement and product. Numbers for products are established in a configuration management (CM) plan.
Traceability ensures completeness, that all lower level requirements come from higher level requirements, and that all higher level requirements are allocated to lower level requirements. Traceability is also used to manage change and provides the basis for test planning.
Ans: Defect leakage
Defect leakage refers to the defect found \ reproduced by the Client or User, which the tester was unable to found.
Defect leakage is the number of bugs that are found in the field that were not found internally. There are a few ways to express this:
* Total number of leaked defects (a simple count)
* defects per customer: number of leaked defects divided by number of customers running that release
* % found in the field: number of leaked defects divided by number of total defects found in that release
In theory, this can be measured at any stage - number of defects leaked from dev into QA, number leaked from QA into beta certification, etc. I've mostly
Used it for customers in the field, though.
Ans: Configuration Management
Configuration Management is a discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specified requirements.
Configuration management (CM) is the detailed recording and updating of information that describes an enterprise's computer systems and networks, including all hardware and software components. Such information typically includes the versions and updates that have been applied to installed software packages and the locations and network addresses of hardware devices. Special configuration management software is available. When a system needs hardware or software upgrade, a computer technician can accesses the configuration management program and database to see what is currently installed. The technician can then make a more informed decision about the upgrade needed.
An advantage of a configuration management application is that the entire collection of systems can be reviewed to make sure any changes made to one system do not adversely affect any of the other systems.
Ans: Statement coverage in Software Testing - Has each line of the source code been executed?
Statement coverage is one of the ways of measuring code coverage. It describes the degree to which the software code of a program has been tested.
All the statements in the code must be executed and tested.
Statement coverage means statement wise we need to give proper test cases.
For your example we need 3 statements coverage’s test cases are needed. And 2 branch coverage’s are needed.
Branch coverage will be on true and false for if statements.
path coverage = branch coverage +1
Ans: Multiple condition coverage metric of software testing
Multiple condition coverage reports whether every possible combination of Boolean sub-expressions occurs. 100% multiple condition coverage implies 100% condition determination coverage.
Drawback of this metric is that it becomes tedious to find out the minimum number of test cases required, especially for very complex Boolean expressions.
Another drawback of this metric is that the number of test cases required can vary to a large extent among various conditions having similar complexity.
Ans: Difference between code coverage analysis & test coverage analysis
Both these terms are similar. Code coverage analysis is sometimes called test coverage analysis. The academic world generally uses the term "test coverage" whereas the practitioners use the term "code coverage".
Ans: Structural testing & functional testing
Structural testing examines how the program works, taking into account possible pitfalls in the structure and logic.
Functional testing examines what the program accomplishes, without regard to how it works internally.
Ans: Probe Testing
It is almost same as Exploratory testing. It is a creative, intuitive process. Everything testers do is optimized to find bugs fast, so plans often change as testers learn more about the product and its weaknesses. Session-based test management is one method to organize and direct exploratory testing. It allows us to provide meaningful reports to management while preserving the creativity that makes exploratory testing work. This page includes an explanation of the method as well as sample session reports, and a tool we developed that produces metrics from those reports.
Ans: Purpose of Automation Testing Tools
The real use and purpose of automated test tools is to automate regression testing.
This means that we must have or must develop a database of detailed test cases that are repeatable, and this suite of tests is run every time there is a change to the application to ensure that the change does not produce unintended consequences.
Ans: Difference between Retesting and Regression Testing
Regression is also retesting, but the objective is different.
yes,
Regression testing is the testing which is done on every build change. We retest already tested functionality for any new bug introduced due to change.
Retesting is the testing of same functionality; this may be due to any bug fix, or due to any change in implementation technology.
Ans: Best sequence of coverage goals as implementation strategy
1) Invoke at least one function in 90% of the source files (or classes).
2) Invoke 90% of the functions.
3) Attain 90% condition/decision coverage in each function.
4) Attain 100% condition/decision coverage
- Advantages & drawbacks of path coverage metric of software testing?
- Advantages & drawbacks of decision coverage metric of software testing?
- Drawbacks of statement coverage metric of software testing?
- Main advantages of statement coverage metric of software testing?
- Compare Automation Testing with Manual Testing.
- Define Test Scenario?
- Is there any automated tool going to replace the testers?
- Are automated tool Replace Manual Testers?
- Basic assumptions behind coverage analysis?
- When to automate tests?
- If you are a Lead Automation engineer then what questions you would ask to yourself and your manager while deciding to automate the tests?
- What is :
"Key Word Driven" Method of Testing?
"Test Plan Driven" Method of Testing?
Comparison Testing?
Parallel Testing?
Automated Testing?
a Data Flow Diagram (DFD)?
a Traceability Matrix?
Defect Leakage?
Configuration Management?
Statement Coverage In software testing,?
multiple condition coverage metric of software testing?
the difference between code coverage analysis & test coverage analysis?
the difference between Structural testing & functional testing?
Probe Testing?
the purpose of Automated Test Tools?
Difference between Retest and Regression Testing?
the best sequence of coverage goals as implementation strategy?
Ans 1: Advantages of path coverage metric of software testing:
Path coverage requires extremely thorough testing.
Disadvantages of path coverage metric of software testing:
1) Since loops introduce an unbounded number of paths, this metric considers only a limited number of looping possibilities.
2) The number of paths is exponential to the number of branches. For example, a function containing 10 if-statements has 1024 paths to test. Adding just one
more if-statement doubles the count to 2048.
3) Many paths are impossible to exercise due to relationships of data.
Ans 2: Decision coverage has the main advantage of simplicity & is free from many problems of statement coverage.
Disadvantage of decision coverage is that this metric ignores branches within Boolean expressions which occur due to short-circuit operators.
Ans 3: Drawbacks of statement coverage metric of software testing:
1) It is insensitive to some of the control structures.
2) It does not report whether loops reach their termination condition - only whether the loop body was executed. With C, C++, and Java, this limitation
affects loops that contain break statements.
3) It is completely insensitive to the logical operators (|| and &&).
4) It cannot distinguish consecutive switch labels.
Ans 4: Advantages of statement coverage metric of software testing
1) The main advantage of statement coverage metric is that it can be applied directly to object code and does not require processing source code. Usually the
Performance profilers use this metric.
2) Bugs are evenly distributed through code; therefore the percentage of executable statements covered reflects the percentage of faults discovered.
Ans 5: Automation Testing versus Manual Testing Guidelines:
I met with my team’s automation experts a few weeks back to get their input on when to automate and when to manually test. The general rule of thumb has always been to use common sense. If you’re only going to run the test one or two times or the test is really expensive to automation, it is most likely a manual test. But then again, what good is saying “use common sense” when you need to come up with deterministic set of guidelines on how and when to automate?
Pros of Automation
• If you have to run a set of tests repeatedly, automation is a huge win for you
• It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner
• It gives you the ability to run automation in mainstream scenarios to catch regressions in a timely manner (see What is a Nightly)
• Aids in testing a large test matrix (different languages on different OS platforms). Automated tests can be run at the same time on different machines, whereas the manual tests would have to be run sequentially.
Cons of Automation
• It costs more to automate. Writing the test cases and writing or configuring the automate framework you’re using costs more initially than running the test manually.
• Can’t automate visual references, for example, if you can’t tell the font colour via code or the automation tool, it is a manual test.
Pros of Manual
• If the test case only runs twice a coding milestone, it most likely should be a manual test. Less cost than automating it.
• It allows the tester to perform more ad-hoc (random testing). In my experiences, more bugs are found via ad-hoc than via automation. And, the more time a tester spends playing with the feature, the greater the odds of finding real user bugs.
Cons of Manual
• Running tests manually can be very time consuming
• Each time there is a new build, the tester must rerun all required tests - which after a while would become very mundane and tiresome.
Other deciding factors:
• What you automate depends on the tools you use. If the tools have any limitations, those tests are manual.
• Is the return on investment worth automating? Is what you get out of automation worth the cost of setting up and supporting the test cases, the automation framework, and the system that runs the test cases?
Ans: Test Scenario is the user workflow in the application.
Example: Checking Mail in Gmail is a scenario, where user login, check the mail in inbox and then logoff. This application can have 2 different test case one for login and other one for inbox.
So Test Scenario can consist of different test cases.
Ans: The simple answer to the question ‘Can automated testing replace all manual testing’� is ‘No.’�
Automated functional tests can be used for regression testing (which is a small part of the overall testing effort). If an organization is running the same manual regression tests repeatedly, then the automated tests can replace some of that effort, but they also add the effort to maintain the tests, which is sometimes more than the work required to just run the tests manually. When I say some of the effort, I mean that test failures from an automated test run still must be analyzed manually. Also, any part of the process of provisioning and setting up the machine to run the tests, kicking off the test run, and babysitting it along the way that isn’t automated will still require manual attention.
Ans: The basic assumptions behind coverage analysis tell us about the strengths and limitations of this testing technique. Some fundamental assumptions are listed
below.
* Bugs relate to control flow and you can expose Bugs by varying the control flow [Beizer1990 p.60]. For example, a programmer wrote "if (c)" rather than
"if (!c)".
* You can look for failures without knowing what failures might occur and all tests are reliable, in that successful test runs imply program correctness
[Morell1990]. The tester understands what a correct version of the program would do and can identify differences from the correct behaviour.
* Other assumptions include achievable specifications, no errors of omission, and no unreachable code.
Clearly, these assumptions do not always hold. Coverage analysis exposes some plausible bugs but does not come close to exposing all classes of bugs.
Coverage analysis provides more benefit when applied to an application that makes a lot of decisions rather than data-centric applications, such as a
database application.
Ans: For a test automation the entry criteria are:
* Availability of a stable application under test (around some 80% of test cases passed).
* Availability of automation test tool with the required Add-ins and Patches.
* Availability of stable and controlled test environment.
* Automation test strategy sign-off
-scope (type of tests)
-functionalities (features to be automated)
-assumptions
-features to be automated
* SIT or UAT sign off
* Signed off manual test cases to be provided
* Availability of stable test bed.
Ans : You are a Lead Automation engineer then what questions you would ask to yourself and your manager while deciding to automate the tests
Best approach would be to raise the following questions:
1) Automating this test and running it once will cost more than simply running it manually once. How much more?
2. An automated test has a finite lifetime, during which it must recoup that additional cost. Is this test likely to die sooner or later? What events are likely to end it?
3. During its lifetime, how likely is this test to find additional bugs (beyond whatever bugs it found the first time it ran)? How does this uncertain benefit balance against the cost of automation?
4. Return of Investment.
Ans: Keyword driven testing:
This requires the development of data tables and keywords, independent of the test automation tool used to execute them and the test script code that "drives" the application-under-test and the data. Keyword-driven tests look very similar to manual test cases. In a keyword-driven test, the functionality of the application-under-test is documented in a table as well as in step-by-step instructions for each test. In this method, the entire process is data-driven, including functionality
The merits of the Keyword Driven Testing are as follows,
- The Detail Test Plan can be written in Spreadsheet format containing all input and verification data.
- If "utility" scripts can be created by someone proficient in the automated tool’s Scripting language prior to the Detail Test Plan being written, then the tester can use the Automated Test Tool immediately via the "spreadsheet-input" method, without needing to learn the Scripting language.
- The tester need only learn the "Key Words" required, and the specific format to use within the Test Plan. This allows the tester to be productive with the test tool very quickly, and allows more extensive training in the test tool to be scheduled at a more convenient time.
Demerits of keyword driven testing
The demerits of the Keyword Driven Testing are as follows:
- Development of "customized" (Application-Specific) Functions and Utilities requires proficiency in the tool’s Scripting language. (Note that this is also true for any method)
- If application requires more than a few "customized" Utilities, this will require the tester to learn a number of "Key Words" and special formats. This can be time-consuming, and may have an initial impact on Test Plan Development. Once the testers get used to this, however, the time required to produce a test case is greatly improved.
Ans: The architecture of the “Test Plan Driven” method appears similar to that of the “Functional Decomposition” method, but in fact, they are substantially
different:
* Driver Script
* Performs initialization, if required;
* Calls the Application-Specific “Controller” Script, passing to it the file-names of the Test Cases (which have been saved from the spreadsheets as a
“tab-delimited” files);
* The “Controller” Script
* Reads and processes the file-name received from Driver;
* Matches on “Key Words” contained in the input-file
* Builds a parameter-list from the records that follow;
* Calls “Utility” scripts associated with the “Key Words”, passing the created parameter-list;
* Utility Scripts
* Process input parameter-list received from the “Controller” script;
* Perform specific tasks (e.g. press a key or button, enter data, verify data, etc.), calling “User Defined Functions” if required;
* Report any errors to a Test Report for the test case;
* Return to “Controller” script;
* User Defined Functions
* General and Application-Specific functions may be called by any of the above script-types in order to perform specific tasks;
Advantages
This method has all of the advantages of the “Functional Decomposition” method, as well as the following:
* The Detail Test Plan can be written in Spreadsheet format containing all input and verification data. Therefore the tester only needs to write this
once, rather than, for example, writing it in Word, and then creating input and verification files as is required by the “Functional Decomposition” method.
* Test Plan does not necessarily have to be written using MS Excel. Any format can be used from which either “tab-delimited” or “comma-delimited” files
can be saved (e.g. Access Database, etc.).
* If “utility” scripts can be created by someone proficient in the Automated tool’s Scripting language prior to the Detail Test Plan being written, then
the tester can use the Automated Test Tool immediately via the "spreadsheet-input" method, without needing to learn the Scripting language. The tester need
only learn the “Key Words” required, and the specific format to use within the Test Plan. This allows the tester to be productive with the test tool very
quickly, and allows more extensive training in the test tool to be scheduled at a more convenient time.
* If Detailed Test Cases already exists in some other format, it is not difficult to translate these into the “spreadsheet” format.
* After a number of “generic” Utility scripts have already been created for testing an application, we can usually re-use most of these if we need to
test another application. This would allow the organization to get their automated testing “up and running” (for most applications) within a few days,
rather than weeks.
Disadvantages
* Development of “customized” (Application-Specific) Functions and Utilities requires proficiency in the tool’s Scripting language. Note that this is
also true of the “Functional Decomposition” method, and, frankly of any method used including “Record/Playback”.
* If application requires more than a few “customized” Utilities, this will require the tester to learn a number of “Key Words” and special formats.
This can be time-consuming, and may have an initial impact on Test Plan Development. Once the testers get used to this, however, the time required to
produce a test case is greatly improved.
Ans: In comparison testing, we compare the old application with new application and see whether the new application is working better than the old application or not.
Comparison Testing means comparing your software with the better one or your Competitor.
While comparison Testing we basically compare the Performance of the software.
For ex If you have to do Comparison Testing of PDF converter (Desktop Based Application) then you will compare your software with your Competitor on the
basis of:-
1.Speed of Conversion PDF file into Word.
2.Quality of converted file.
Ans: Parallel Testing
Parallel/audit testing is a type of testing where the tester reconciles the output of the new system to the output of the current system, in order to verify the new system operates correctly.
OR
Comparing our Product / Application build with other products existing in the market. Parallel Testing is also known as Comparative Testing / Competetive Testing.
Testing newly developed system and compare the results with already existing system to check any discrepancy between them.
Ans: Automated Testing:
Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.
Test automation can be expensive, and it is usually employed in combination with manual exploratory testing. It can be made cost-effective in the longer term
though, especially in regression testing. One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation, but research continues into a variety of methodologies for doing so.
What to automate, when to automate, or even whether one really needs automation are crucial decisions which the testing (or development) team has to take.
Selecting the correct features of the product for automation largely decides the success of the automation. Unstable features or the features which are undergoing changes should be avoided.
Ans: Data Flow Diagram (DFD)
Data Flow Diagram is a graphical representation of the "flow" of data through an information system. A data flow diagram can also be used for the visualization of data processing. It is common practice for a designer to draw a context-level DFD first which shows the interaction between the system and outside entities.
The process model is typically used in structured analysis and design methods. Also called a data flow diagram (DFD), it shows the flow of information through a system. Each process transforms inputs into outputs.
The model generally starts with a context diagram showing the system as a single process connected to external entities outside of the system boundary. This
process explodes to a lower level DFD that divides the system into smaller parts and balances the flow of information between parent and child diagrams. Many
diagram levels may be needed to express a complex system. Primitive processes, those that don't explode to a child diagram, are usually described in a
connected textual specification.
Ans: Traceability Matrix
Traceability means that we would like to be able to trace back and forth how and where any work product fulfils the directions of the preceding product.
The matrix deals with the where, while the how we have to do our self, once we know the where.
A traceability matrix is created by associating requirements with the work products that satisfy them. Tests are associated with the requirements on which
they are based and the product tested to meet the requirement.
There can be more things included in a traceability matrix than shown. In traceability, the relationship of driver to satisfier can be one-to-one, one-to-many, many-to-one, or many-to-many.
Traceability requires unique identifiers for each requirement and product. Numbers for products are established in a configuration management (CM) plan.
Traceability ensures completeness, that all lower level requirements come from higher level requirements, and that all higher level requirements are allocated to lower level requirements. Traceability is also used to manage change and provides the basis for test planning.
Ans: Defect leakage
Defect leakage refers to the defect found \ reproduced by the Client or User, which the tester was unable to found.
Defect leakage is the number of bugs that are found in the field that were not found internally. There are a few ways to express this:
* Total number of leaked defects (a simple count)
* defects per customer: number of leaked defects divided by number of customers running that release
* % found in the field: number of leaked defects divided by number of total defects found in that release
In theory, this can be measured at any stage - number of defects leaked from dev into QA, number leaked from QA into beta certification, etc. I've mostly
Used it for customers in the field, though.
Ans: Configuration Management
Configuration Management is a discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specified requirements.
Configuration management (CM) is the detailed recording and updating of information that describes an enterprise's computer systems and networks, including all hardware and software components. Such information typically includes the versions and updates that have been applied to installed software packages and the locations and network addresses of hardware devices. Special configuration management software is available. When a system needs hardware or software upgrade, a computer technician can accesses the configuration management program and database to see what is currently installed. The technician can then make a more informed decision about the upgrade needed.
An advantage of a configuration management application is that the entire collection of systems can be reviewed to make sure any changes made to one system do not adversely affect any of the other systems.
Ans: Statement coverage in Software Testing - Has each line of the source code been executed?
Statement coverage is one of the ways of measuring code coverage. It describes the degree to which the software code of a program has been tested.
All the statements in the code must be executed and tested.
Statement coverage means statement wise we need to give proper test cases.
For your example we need 3 statements coverage’s test cases are needed. And 2 branch coverage’s are needed.
Branch coverage will be on true and false for if statements.
path coverage = branch coverage +1
Ans: Multiple condition coverage metric of software testing
Multiple condition coverage reports whether every possible combination of Boolean sub-expressions occurs. 100% multiple condition coverage implies 100% condition determination coverage.
Drawback of this metric is that it becomes tedious to find out the minimum number of test cases required, especially for very complex Boolean expressions.
Another drawback of this metric is that the number of test cases required can vary to a large extent among various conditions having similar complexity.
Ans: Difference between code coverage analysis & test coverage analysis
Both these terms are similar. Code coverage analysis is sometimes called test coverage analysis. The academic world generally uses the term "test coverage" whereas the practitioners use the term "code coverage".
Ans: Structural testing & functional testing
Structural testing examines how the program works, taking into account possible pitfalls in the structure and logic.
Functional testing examines what the program accomplishes, without regard to how it works internally.
Ans: Probe Testing
It is almost same as Exploratory testing. It is a creative, intuitive process. Everything testers do is optimized to find bugs fast, so plans often change as testers learn more about the product and its weaknesses. Session-based test management is one method to organize and direct exploratory testing. It allows us to provide meaningful reports to management while preserving the creativity that makes exploratory testing work. This page includes an explanation of the method as well as sample session reports, and a tool we developed that produces metrics from those reports.
Ans: Purpose of Automation Testing Tools
The real use and purpose of automated test tools is to automate regression testing.
This means that we must have or must develop a database of detailed test cases that are repeatable, and this suite of tests is run every time there is a change to the application to ensure that the change does not produce unintended consequences.
Ans: Difference between Retesting and Regression Testing
Regression is also retesting, but the objective is different.
yes,
Regression testing is the testing which is done on every build change. We retest already tested functionality for any new bug introduced due to change.
Retesting is the testing of same functionality; this may be due to any bug fix, or due to any change in implementation technology.
Ans: Best sequence of coverage goals as implementation strategy
1) Invoke at least one function in 90% of the source files (or classes).
2) Invoke 90% of the functions.
3) Attain 90% condition/decision coverage in each function.
4) Attain 100% condition/decision coverage
Traceability Matrix
Traceability Matrix is used in entire software development life cycle phases:
This document is prepared to make the clients satisfy that the coverage done is complete as end to end, this document consists of Requirement/Base line doc Ref No., Test case/Condition, and Defects/Bug id. Using this document the person can track the Requirement based on the Defect id
Note – We can make it a “Test case Coverage checklist” document by adding few more columns. We will discuss in later posts
Types of Traceability Matrix:
Why Bi-Directional Traceability is required?
Bi-Directional Traceability contains both Forward & Backward Traceability. Through Backward Traceability Matrix, we can see that test cases are mapped with which requirements.
Backward Traceability Matrix ensures – We the Building the Product Right.
Traceability matrix is the answer of the following questions of any Software Project:
No traceability or Incomplete Traceability Results into:
1. Poor or unknown test coverage, more defects found in production
2. It will lead to miss some bugs in earlier test cycles which may arise in later test cycles. Then a lot of discussions arguments with other teams and managers before release.
3. Difficult project planning and tracking, misunderstandings between different teams over project dependencies, delays, etc
Benefits of using Traceability Matrix
1. Make use of excel to create Traceability Matrix:
2. Define following columns:
Base Specification/Requirement ID (If any)
Requirement ID
Requirement description
TC 001
TC 002
TC 003.. So on.
3. Identify all the testable requirements in granular level from requirement document. Typical requirements you need to capture are as follows:
Used cases (all the flows are captured)
Error Messages
Business rules
Functional rules
SRS
FRS
So on…
4. Identity all the test scenarios and test flows.
5. Map Requirement IDs to the test cases. Assume (as per below table), Test case “TC 001” is your one flow/scenario. Now in this scenario, Requirements SR-1.1 and SR-1.2 are covered. So mark “x” for these requirements.
Now from below table you can conclude –
Requirement SR-1.1 is covered in TC 001
Requirement SR-1.2 is covered in TC 001
Requirement SR-1.5 is covered in TC 001, TC 003 [Now it is easy to identify, which test cases need to be updated if there is any change request].
TC 001 Covers SR-1.1, SR, 1.2 [we can easily identify that test cases covers which requirements].
TC 002 covers SR-1.3.. So on..
This is a very basic traceability matrix format. You can add more following columns and make it more effective:
ID, Assoc ID, Technical Assumption(s) and/or Customer Need(s), Functional Requirement, Status, Architectural/Design Document, Technical Specification, System Component(s), Software Module(s), Test Case Number, Tested In, Implemented In, Verification, Additional Comments,
1. Risk Analysis phase
2. Requirements Analysis and Specification phase
3. Design Analysis and Specification phase
4. Source Code Analysis, Unit Testing & Integration Testing phase
5. Validation – System Testing, Functional Testing phase
In this topic we will discuss: - What is Traceability Matrix from Software Testing perspective? (Point 5)
- Types of Traceability Matrix
- Disadvantages of not using Traceability Matrix
- Benefits of using Traceability Matrix in testing
- Step by step process of creating an effective Traceability Matrix from requirements. Sample formats of Traceability Matrix basic version to advanced version.
This document is prepared to make the clients satisfy that the coverage done is complete as end to end, this document consists of Requirement/Base line doc Ref No., Test case/Condition, and Defects/Bug id. Using this document the person can track the Requirement based on the Defect id
Note – We can make it a “Test case Coverage checklist” document by adding few more columns. We will discuss in later posts
Types of Traceability Matrix:
- Forward Traceability – Mapping of Requirements to Test cases
- Backward Traceability – Mapping of Test Cases to Requirements
- Bi-Directional Traceability - A Good Traceability matrix is the References from test cases to basis documentation and vice versa.
Why Bi-Directional Traceability is required?
Bi-Directional Traceability contains both Forward & Backward Traceability. Through Backward Traceability Matrix, we can see that test cases are mapped with which requirements.
This will help us in identifying if there are test cases that do not trace to any coverage item— in which case the test case is not required and should be removed (or maybe a specification like a requirement or two should be added!). This “backward” Traceability is also very helpful if you want to identify that a particular test case is covering how many requirements?
Through Forward Traceability – we can check that requirements are covered in which test cases? Whether is the requirements are coved in the test cases or not?
Forward Traceability Matrix ensures – We are building the Right Product. Backward Traceability Matrix ensures – We the Building the Product Right.
Traceability matrix is the answer of the following questions of any Software Project:
- How is it feasible to ensure, for each phase of the SDLC, that I have correctly accounted for all the customer’s needs?
- How can I certify that the final software product meets the customer’s needs? Now we can only make sure requirements are captured in the test cases by traceability matrix.
No traceability or Incomplete Traceability Results into:
1. Poor or unknown test coverage, more defects found in production
2. It will lead to miss some bugs in earlier test cycles which may arise in later test cycles. Then a lot of discussions arguments with other teams and managers before release.
3. Difficult project planning and tracking, misunderstandings between different teams over project dependencies, delays, etc
Benefits of using Traceability Matrix
- Make obvious to the client that the software is being developed as per the requirements.
- To make sure that all requirements included in the test cases
- To make sure that developers are not creating features that no one has requested
- Easy to identify the missing functionalities.
- If there is a change request for a requirement, then we can easily find out which test cases need to update.
- The completed system may have “Extra” functionality that may have not been specified in the design specification, resulting in wastage of manpower, time and effort.
1. Make use of excel to create Traceability Matrix:
2. Define following columns:
Base Specification/Requirement ID (If any)
Requirement ID
Requirement description
TC 001
TC 002
TC 003.. So on.
3. Identify all the testable requirements in granular level from requirement document. Typical requirements you need to capture are as follows:
Used cases (all the flows are captured)
Error Messages
Business rules
Functional rules
SRS
FRS
So on…
4. Identity all the test scenarios and test flows.
5. Map Requirement IDs to the test cases. Assume (as per below table), Test case “TC 001” is your one flow/scenario. Now in this scenario, Requirements SR-1.1 and SR-1.2 are covered. So mark “x” for these requirements.
Now from below table you can conclude –
Requirement SR-1.1 is covered in TC 001
Requirement SR-1.2 is covered in TC 001
Requirement SR-1.5 is covered in TC 001, TC 003 [Now it is easy to identify, which test cases need to be updated if there is any change request].
TC 001 Covers SR-1.1, SR, 1.2 [we can easily identify that test cases covers which requirements].
TC 002 covers SR-1.3.. So on..
| Requirement ID | Requirement description | TC 001 | TC 002 | TC 003 |
| SR-1.1 | User should be able to do this | x | ||
| SR-1.2 | User should be able to do that | x | ||
| SR-1.3 | On clicking this, following message should appear | x | ||
| SR-1.4 | x | |||
| SR-1.5 | x | x | ||
| SR-1.6 | x | |||
| SR-1.7 | x |
ID, Assoc ID, Technical Assumption(s) and/or Customer Need(s), Functional Requirement, Status, Architectural/Design Document, Technical Specification, System Component(s), Software Module(s), Test Case Number, Tested In, Implemented In, Verification, Additional Comments,
Saturday, June 19, 2010
International Scores
International Scores: "Get the latest scores of all the international cricket matches from Cricinfo. Add the Cricinfo International Scores widget now!"
Friday, June 18, 2010
Wednesday, November 4, 2009
SDLC for Web Development
SDLC or Software Development Life Cycle deals with online programs or computer software. Different types of designs were created to cater to the need of individual programmers with different set of skills, demands or business environment. As the name suggests, it is only limited to creating software.
However, our world is not all about software. Everyday, we use the internet and visit different websites to learn, to shop, to do business or even have fun. And the development of a specific website is not just writing anything in HTML with the hope that someone will visit and in your end you will eventually through advertising sales or anything.
SDLC Tutorials
Information Technology SDLCSDLC for Web Development
SDLC for Web Development
Page 1 of 2
SDLC Introduction
SDLC or
However, our world is not all about software. Everyday, we use the internet and visit different websites to learn, to shop, to do business or even have fun. And the development of a specific website is not just writing anything in HTML with the hope that someone will visit and in your end you will eventually through advertising sales or anything.
In this chapter we take a look at different factors and steps in applying SDLC in Web Development. You will be able see how relevant a website could be in
Factors in SDLC for Web Development will also be discussed in this chapter. It should happen in an entirely different environment and there are different players now that instead of creating software, it is now focused on the creation of a fully functional website that has the ability to earn, and inform it is intended audience.
There are different questions that will be asked as well as different conclusions and results from these questions. Each factor will have a deep influence in the outcome of the website and SDLC will carefully outline their roles.
II. Factors
To fully understand the factors that will affect web development in the SDLC framework, we will be asking questions related to web development. These questions are: What? Why? Who? Where? When? How?
These are just simple questions but if you look at them deeper, you will basically answer everything about the website before they are created. These questions are arranged in a way as to how the website will be built.
What? – The most basic of all questions. What kind of website? It all starts with an idea of someone and eventually presenting them to their fellow web designers. It does not really matter what as long as it answers the next question that his/her fellow web designers will ask.
Why? – Although the answer “its cool” will work on fellow web developers, it will never work in the eyes of everyone. Years ago, the web is more of a fad rather than being put to better use. Now, we are more than just connected, being online is our way of life. Most of the time, the things that we need or know can be found online. If you present a colorful website without purpose, you better forget it. You have to give them a good reason why they have to visit your website.
Who? – Maybe you already know who will be spearheading the project but you will need manpower especially when you are tackling a very big project.
Where? – Although the office you will be using right now could be your workspace for the coming months, it will probably take a lot more space when everyone will start working on a website. This would include facilities such as servers, additional computers, things and even food space to support your hungry web developers.
When? – One of the most important aspect in website development in terms of relating to marketers, tech bloggers or anyone who can make the website a very popular one or something despicable. Time frame is very important since an old adage “time is money” will still hold true. Everyday you pay your developers and support staff to create a website.
How? – This part could be translated as in SDLC as the planning stage which means everything has been set and everyone is ready to build a website. Languages and tools are now discussed by developers and they are set to actually apply the concept.
Subscribe to:
Posts (Atom)