Automation Testing

Automation Testing Life Cycle Advantages of this automated software using the above AST life cycle. 1. High Quality to market 2. Low Time to market 3. Reduced testing time 4. Consistent test procedures 5. Reduced QA costs 6. Improved testing productivity 7. Improved product quality AST Requirements >>> • Reqmt / Functional Specification documents • Design Specification documents • Test Traceability Matrix for identifying Test Coverage • Functional/ Non-Functional and test data requirements • Test phases to be automated and % of automation AST Planning >>> • Automated Software Testing (AST) Scope • Tool Evaluation and identification • AST Methodologies and Framework • Prepare and Base lining Scripting standard and ASTPlan AST Environment Setup>>> • AST Test Bed installation and configuration • Network connectivity’s • All the Software/ tools Licenses, Installation and configuration • Coordination with Vendors and others AST Design >>> • Test Script and test data preparation • Test scripts / test data review and unit testing • Integration Testing Test scripts and testing • Base lining under Configuration Management AST Execution and Defect Tracking >>> • Executing AST Test Suit • Capture, review and analyze Test Results • Defects reporting and tracking for its closure AST Maintenance Reports and Acceptance >>> • AST Results and summary reports • Test Metrics and process Improvements made • Base lining of AST Test suits/ scripts/ test date etc for maintenance phase • Getting Acceptance Important Considerations for Test Automation Often when a test automation tool is introduced to a project, the expectations for the return on investment are very high. Project members anticipate that the tool will immediately narrow down the testing scope, meaning reducing cost and schedule. However, I have seen several test automation projects fail - miserably. The following very simple factors largely influence the effectiveness of automated testing, and if not taken into account, the results is usually a lot of lost effort, and very expensive ?shelfware?. Scope - It is not practical to try to automate everything, nor is there the time available generally. Pick very carefully the functions/areas of the application that are to be automated. Preparation Timeframe - The preparation time for automated test scripts has to be taken into account. In general, the preparation time for automated scripts can be up to 2/3 times longer than for manual testing. In reality, chances are that initially the tool will actually increase the testing scope. It is therefore very important to manage expectations. An automated testing tool does not replace manual testing, nor does it replace the test engineer. Initially, the test effort will increase, but when automation is done correctly it will decrease on subsequent releases. Return on Investment - Because the preparation time for test automation is so long, I have heard it stated that the benefit of the test automation only begins to occur after approximately the third time the tests have been run. When is the benefit to be gained? Choose your objectives wisely, and seriously think about when & where the benefit is to be gained. If your application is significantly changing regularly, forget about test automation - you will spend so much time updating your scripts that you will not reap many benefits. [However, if only disparate sections of the application are changing, or the changes are minor - or if there is a specific section that is not changing, you may still be able to successfully utilise automated tests]. Bear in mind that you may only ever be able to do a complete automated test run when your application is almost ready for release ? i.e. nearly fully tested!! If your application is very buggy, then the likelihood is that you will not be able to run a complete suite of automated tests ? due to the failing functions encountered. The Degree of Change ? The best use of test automation is for regression testing, whereby you use automated tests to ensure that pre-existing functions (e.g. functions from version 1.0 - i.e. not new functions in this release) are unaffected by any changes introduced in version 1.1. And, since proper test automation planning requires that the test scripts are designed so that they are not totally invalidated by a simple gui change (such as renaming or moving a particular control), you need to take into account the time and effort required to update the scripts. For example, if your application is significantly changing, the scripts from version 1.0. may need to be completely re-written for version 1.1., and the effort involved may be at most prohibitive, at least not taken into account! However, if only disparate sections of the application are changing, or the changes are minor, you should be able to successfully utilise automated tests to regress these areas. Test Integrity - how do you know (measure) whether a test passed or failed ? Just because the tool returns a ?pass? does not necessarily mean that the test itself passed. For example, just because no error message appears does not mean that the next step in the script successfully completed. This needs to be taken into account when specifying test script fail/pass criteria. Test Independence - Test independence must be built in so that a failure in the first test case won't cause a domino effect and either prevent, or cause to fail, the rest of the test scripts in that test suite. However, in practice this is very difficult to achieve. Debugging or "testing" of the actual test scripts themselves - time must be allowed for this, and to prove the integrity of the tests themselves. Record & Playback - DO NOT RELY on record & playback as the SOLE means to generates a script. The idea is great. You execute the test manually while the test tool sits in the background and remembers what you do. It then generates a script that you can run to re-execute the test. It's a great idea - that rarely works (and proves very little). Maintenance of Scripts - Finally, there is a high maintenance overhead for automated test scripts - they have to be continuously kept up to date, otherwise you will end up abandoning hundreds of hours work because there has been too many changes to an application to make modifying the test script worthwhile. As a result, it is important that the documentation of the test scripts is kept up to date also.  When Test Automation Makes Sense Let’s start with the tests that ideally are automated. These include: Regression and confirmation. Rerunning a test against a new release to ensure that behavior remains unbroken—or to confirm that a bug fix did indeed fix the underlying problem—is a perfect fit for automated testing. The business case for test automation outlined in Software Test Automation by Mark Fewster and Dorothy Graham is built around this kind of testing. 1. Monkey (or random). Tests that fire large amounts or long sequences of data, transactions, or other inputs at a system in a random search for errors are easily and profitably automated 2. Load, volume, and capacity. Sometimes, systems must support tremendous loads. On one project, we had to test how the system would respond to 50,000 simultaneous users, which ruled out manual testing! Two Linux systems running custom load-generating programs filled the bill. 3.Performance and reliability. With the rise of Web-based systems, more and more automated testing is aimed at looking for slow or flaky behavior on Web systems. 4. Structural, especially API-based unit, component, and integration. Most structural testing involves harnesses of some sort, which brings you most of the way into automation. Again, the article I wrote with Greg Kubaczkowski, "Mission Made Possible" (STQE magazine, July/Aug. 2002), provides an example. Other tests that are well-suited for automation exist, such as the static testing of complexity and code standards compliance that I mentioned in the previous article. In general, automated tests have higher upfront costs—tools, test development, environments, and so forth—and lower costs to repeat the test.  QuickTest Professional (QTP) To reduce the time and costs associated with application testing, you need a solution that offers functionality across all environments and business processes. HP QuickTest Professional supports functional and regression test automation that  addresses every major software application and environment. This solution uses the concept of keyword-driven testing to simplify test creation and maintenance.  It enables testers to build test cases by capturing flows directly from the application screens using specialized capturing technology. Test experts also have full access to the underlying test and object properties via an integrated scripting and debugging environment. What's New in QuickTest Professional 9.5 QuickTest Professional 9.5 brings you an exciting array of new features to help you design and maintain your tests and components easier and faster. The new features include: > All-in-one QuickTest Installation > New QuickTest IDE Panes > Process Guidance > Maintenance Run Mode > Checkpoint and Output Objects in the Object Repository > Test Your PowerBuilder Applications in QuickTest > Extend QuickTest Web Support for your Custom Web Objects > Accelerate SAP Testing with Business Process Testing for SAP Applications > Checkpoint and Output Value Steps Supported for Business Components > Tolerance values for Bitmap Checkpoints > Drag and Drop objects from Object Repository > Replace Local Objects with Shared Objects After Export > New Supported Operating Systems and Environments Mercury Quality Center HP Quality Center is designed to address the wide-ranging challenges that your quality assurance professionals face. By leveraging consistent, repeatable and standardized processes, it helps your quality assurance teams turn quality into a competitive advantage. HP Quality Center helps your organization: 1. Gain real-time visibility into requirements coverage and associated defects to paint a clear picture of business risk 2. Manage the release process and make more informed release decisions with real-time KPIs and reports 3. Measure progress and effectiveness of quality activities 4. Collaborate in the quality lifecycle with a single global platform 5. Manage manual and automated testing assets centrally 6. Facilitate standardized testing and quality processes that boost productivity through workflows and alerts 7. Lower costs by capturing critical defects before they reach production  Mercury WinRunner Like most organizations, you are probably looking for ways to efficiently test multiple applications that are constantly evolving and required to work in diverse environments. HP WinRunner provides a powerful tool for enterprise-wide functional and regression testing. This software allows your team to capture, verify and replay user interactions automatically to identify defects and enable your business processes to work flawlessly upon deployment. Your organization can reduce testing time by automating repetitive tasks and optimize your testing efforts by covering diverse environments with a single testing tool.  Mercury LoadRunner Do your applications and systems provide the performance and scalability your business requires? HP LoadRunner can help you prevent costly performance problems in production by detecting bottlenecks before a new system or upgrade is deployed. This software enables you to measure end-to-end performance, diagnose application and system bottlenecks and tune for better performance—all from a single point of control. This helps reduce the costs and time required to test and deploy new applications and systems into your production environment.