Tester Testifies

Importance of Testing in Envision and Planning Phase

2012-02-23T09:51:00.001+05:30

When you are working as a Test Consultant and if you are involved in the early phases of the engagement, it is quite important to deliver the test artifacts on a regular basis to show measurable progress. That can help a long way to win the confidence of the customer and the stakeholders involved. Arguably, the other disciplines (like developers, architects, business analysts) have relatively less to do to prove their worth as they own the key artifacts like Vision scope document & Functional specifications, which are perceived as the most critical ones for the success of the project. Though the test planning is an equally important activity as any other development activity of envisioning and planning phase it doesn’t get the same attention. Overall it gives a wrong impression that the testing team is not adding as much value as the development team in the initial phases of the project. I believe it is important to write a quality test plan that is not verbose but highly customized and designed, keeping in the mind the specific requirements and technology stack for the given project. It is critical for us to educate the customers by evangelizing the importance of testing in the initial phases.

I was in a discussion with the customers and they had this strong opinion that they are paying a lot of money for testing teams to just review requirements and write test cases. They recommended us to write automation scripts only without spending any time on designing test scenarios and test cases. I highlighted our role and contribution in requirement elicitation process to identify the gaps and save the effort upfront rather than discovering them in the build or stabilization phase. To accomplish that goal we need to spend effort in coming up with a detailed test plan and strategy that best fits the unique needs and requirements of the project in hand. We methodologically identify the areas that are best candidates for automation based on the ROI (Return on Investment) to save effort and time during the test execution phase. The quality of the automation scripts depends on the coverage of the underlying test cases / test scenarios and hence it is important to have them. At the end of the discussion I could sense that the customers were still skeptical and I didn’t expect them to get totally convinced either. I bought time and promised them to give a small demonstration of our capabilities by doing some quick POCs (Proof of Concepts).

From the requirements, I picked the most important ones and studied the pattern, the behavior and looked at the testability aspect of them and prepared a list of test techniques. I wanted to reduce my test cases to a number that is manageable and maintainable. I figured out that they had lot of complex process workflows where there are multiple jumps, branches and loops. I couldn’t cover them even if I wrote a whole bunch of test cases and even after doing that it will not guarantee that I didn’t miss an important path. We had the Visio's of those workflows so why not use a “State Transition Table”? I came up with a quick template to represent their most complex workflow and ensured I covered at least every state and every transition once. I had added a flag to be able to filter the main flows/alternate flows.

Then there were a few complex business rules that had lot of conditional clauses and the result depended on the outcome of all those clauses. Again I realized that there can be lot of permutation here and I didn’t want to take chances. “Decision Table” just sounded right to me. I quickly prepared one covering positive and negative flows and here I had a table that was covering it all.

For lot of data entry forms, there were many fields and especially drop downs, radio buttons and checkboxes. I didn’t want my team to randomly pick values and tomorrow miss double-mode types of defects. “Orthogonal array of testing” can be helpful and I thought of using its Pairwise implementation to solve that problem. I used a free tool from Microsoft called PICT and voila from 72 permutations I had now arrived at just 9 permutations scientifically that ensures every pair is covered at least once.

After a couple of days I had put together a few running POCs of these techniques in addition to test cases that used standard techniques like EP & BVA (Equivalence Partitioning and Boundary Value Analysis). I quickly came up with the best way to structure and organize my test artifacts using Microsoft Test Manager. I created a sample Requirement Traceability Matrix using Test Manager. I decided on my test suites and also defined different test configurations as the application is being targeted for multiple platforms.

In 3 days, we were in a conference room with all the customers and stakeholders including our project team. I ran the demos of my test techniques on the real requirements. I showed them we would ensure test coverage and provide them the requirement traceability reports using Test Manager. PICT was like magic to them. State transition tables taught them that testing is much more than randomly clicking buttons and following plain steps. Automation can be done only after you perform these key activities and test cases are not just plain word documents.

The meeting went well and ended with encouraging remarks like “Can you send me the link of that tool? “, “Can you share these with us for Unit testing?”, “We want you to demo this to our test team", “This sounds great”. It was evident from their response that we had their confidence.

Stay tuned for more post. I can be reached at rajkamal@microsoft.com

Disclaimer: This blog is my personal opinion and I apologize if it hurt your sentiments. In the words of my friend Arun (aruncha@microsoft.com) , “When a developer starts in a project he is already 100 miles high whereas a tester is 100 miles buried under the ground and it takes the tester lot of work to reach up there and prove his value before his work is acknowledged “

TechNet: SQL Server 2012 DQS (Data Quality Services) for Testing Teams

2012-02-23T04:35:00.001+05:30

MSDN Technet Article on SQL Server 2012 (Code name: Denali) DQS (Data Quality Services) for Testing team:

http://social.technet.microsoft.com/wiki/contents/articles/7701.sql-server-2012-dqs-data-quality-services-for-testing-teams.aspx

TechNet Webcast: Automated Data Quality Checks Using SQL Server and Visual Studio (Level 300) – Raj Kamal, Microsoft

2011-12-18T12:13:00.001+05:30

TechNet Webcast: Automated Data Quality Checks Using SQL Server and Visual Studio (Level 300) – Raj Kamal, Microsoft

Microsoft IT's business intelligence engineering team has reduced its development and testing phases by automating the data quality audits and finding data quality issues earlier in the development cycle than in the past, using a metadata-driven, user-configurable tool. Attend this webcast to learn about automating data quality checks using Microsoft SQL Server and Microsoft Visual Studio and to hear directly from the development team how this can apply to you.
Presenter: Raj Kamal, Microsoft IT Test Lead, Microsoft Corporation
Raj Kamal is a senior test engineer who specializes in different types of testing techniques, test automation, and testability in different domains like manufacturing, healthcare, and higher education. Raj holds several certifications, including Certified Software Test Engineer (CSTE), Microsoft IT Professional for Microsoft SQL Server Business Intelligence (BI) 2008, International Software Testing Qualifications Board (ISTQB), and APICS certification in supply chain management. Raj also has expertise with Rational and Mercury testing tools, and he has helped teams develop test automation strategies and architectures for other major software companies.

Event: https://msevents.microsoft.com/CUI/EventDetail.aspx?culture=en-US&EventID=1032452088&CountryCode=US

View Online: https://www106.livemeeting.com/cc/MSEventsBMO/view?cn=guest&id=1032452087&pw=3CE735D1

Thanks,

Raj

Saga of Unsung Heroes (QAI STC 2011 - Published)

2011-12-16T20:16:00.000+05:30

This paper is written by the Testers, for the Tester. The Testers, who save millions of dollars by finding bugs, still stay unheard & unsung. These are the ordinary men doing extraordinary deeds by living in a software world that poses several challenges making the „Quality‟ as a job tougher than ever.

This paper is a tribute to all the testers in this world doing a thankless job, those who have spent long hours & many years of their life with a strong belief that every bug they find makes the product better. This paper tries to address #3 major challenges of their (our) life and proposes some fresh solutions to very old problems based on our learning that can reinstate the same belief in new generations of testers

http://www.qaiglobalservices.com/minisites/stc_2011/pdf/Saga_of_Unsung_Heroes.pdf

Agile Test Estimations using Poker Planning (QAI STC 2011 - Published)

2011-12-16T20:12:00.001+05:30

This artifact offers a template and guidelines which can be used by agile teams to commit to stories for a sprint using the proven top-down "Poker Planning" technique (for both Developers and Testing) and the capacity in hand.

The poker player learns that sometimes both science and common sense are wrong; that the bumblebee can fly; that, perhaps, one should never trust an expert; that there are more things in heaven and earth than are dreamt of by those with an academic bent. (David Mamet)

http://www.qaiglobalservices.com/minisites/stc_2011/pdf/Agile_Test_Estimations_using_Poker_Planning.pdf

Leveraging Microblogging for Effective Team Collaboration (QAI STC 2011)

2011-12-16T20:08:00.000+05:30

Proposing the usage of Microblogging as a project communication layer on top of defect and requirement management tools like QC, primarily for Agile projects, having teams spread across geographies. Also to better help teams to include customer, stakeholders & management in day-to-day project conversations and updates.

Check out this link:

http://www.qaiglobalservices.com/minisites/stc_2011/pdf/Leveraging_Microblogging_for_Effective_Team_Collaboration.pdf

SQL Server 2012 (Denali) Data Quality Services (DQS) – Taking Test Practices Places! (QAI STC 2011 - Published)

2011-12-16T20:03:00.000+05:30

If we wait until wrong decisions are made by customers relying on the bad or inaccurate data - we are in trouble. Data quality is often overlooked and yet there is no well-defined testing process or that provides guidance around ways to improve, maintain and monitor the data quality in any data-centric project. With the absence of any specialized testing tools that can aid in testing for data quality makes the entire situation even tougher. However the bad times are now over and with tools like upcoming SQL Server "Denali" Data Quality Services (DQS) we can turn the tables around. This artifact will show you how by using Denali DQS but it can be also applied on the other tools which have similar capabilities.
Check out the link:

http://www.qaiglobalservices.com/minisites/stc_2011/pdf/Denali_DQS.pdf

Web Services Test Automation Guidance using VS 2010 - Unit Test & Web Test

2011-12-11T04:58:00.001+05:30

For all deliverables & details: http://webservicestesting.codeplex.com

Authors

Raj Kamal (rajkamal@microsoft.com) ,

Vipul Mehta (vmehta@microsoft.com ) ,

Ranjit Gupta (rankumar@microsoft.com )

Gunjan Jain (gunjain@microsoft.com )

Microsoft Services - Global Delivery, India

Objective

This artifact will provide Web Services Automation guidance for Functional & Performance Testing leveraging VS 2010. The document will provide recommendation on using either Unit Test or Web Test for functional testing depending on your project and test objective as will be later covered in the document. Load Test will be used for Performance Testing in both cases.

The primary objective is to propose a viable web services test automation approach & reusable libraries to customers, partners and consultant. It will also be a step in standardizing web services automation using VS 2010 platform across the Industry and filling the gaps by extending our tools as suggested in the talk. The document can also be used as a quick jump start guide for beginners to get started on Web Services Automation Testing.

Other than the guidance included in this document, key deliverable will primarily consists of the following:

a) Unit Test & Web Test Template projects for Web services testing

b) Reusable libraries for Unit Test

c) Reusable Add-ins (Extraction & Validation Rules) for Web Test that can be reused to reduce automation development effort drastically.

d) Performance Testing using Load Test once the Web Tests/Unit Tests are in place.

e) Sample Web Services application for beginners to gets started

Pre-requisites:

Ensure you have downloaded and deployed WebServiceAutomation_SampleApp that contains sample Web services projects (WCF/ASMX/WF etc.) that you can use to getting started with Web Services automation.

Verify you have downloaded our sample Template projects WebServiceAutomation_SampleUnitTest and WebServiceAutomation_SampleWebTest that contain sample unit test/web test which you can reuse / customize for your needs

Ensure you have downloaded our Reusable Libraries for Unit Testing (WebServicesTesting.Automation.UnitTest.Helper.dll) & Web Test (WebServicesTesting.Automation.WebTest.Helper.dll)

Visual Studio 2010, SQL Server Express or higher version, IIS (Internet Information Server 6.0 or above

Importance of Web Service Automation

It’s important to first understand the importance of Web Services Automation and the ideal scenarios where it must be recommended.

Web Services (WCF/ASMX) due to their worldwide implementation and popularity but limited test tool support, often raises the question around testing them effectively. Testers are craving to find effective ways to Test Web Services. Even VS 2010 doesn’t have dedicated test types for verifying Web Services (like for UI, there is Coded UI and for DB there is Database Unit Test) which results in low test coverage and non-standardization of proven ways to test web services especially WCF 4.0 Web Services. Web Services Testing in VS 2010 so far has been not been given a first-class citizen treatment and this guidance will bridge that gap by making it much easier and effective.

· In this agile development era, we are moving to a model where UI is ever changing and consummation of web services is taking the center stage and hence this layer attention it truly deserves by shifting the onus from UI testing to Web Services based validation.

Traditionally teams have also used Fiddler and other 3^rd Party tools for the same purpose (many times for manual testing alone) but as Web services have different needs, the knowledge harvested doesn’t get reused to its potential in future engagements

Ideal Candidates

· If your application is consuming APIs exposed by other upstream systems or 3rd Parties

· If your application is exposing APIs to be consumed by the downstream applications

· When your UI is dynamic and constantly changing and cost/maintenance of UI automation is much more.

· When your business logic and rules are primarily implemented using services (ASMX, WCF & Workflow Services) and testing at this layer is much faster and efficient.

· When you need to come up with a common automation framework for web services that can be used to validate both functional and performance aspect of the web services with minimum effort.

Test Approach – Identify your goal (Functional Test or Performance Test or Both)

When we understand and realize the importance of Web Services Automation, the next challenge is to select the right Test approach and Test Type before jumping into automation design and implementation.

Test Objective

It is important to answer the basic question “Why do you want to do Web Services Testing?” and that will help you decide the right kind of Test you should choose to automate them. Let’s begin with few possible answers

To validate the functional behavior of your application / APIs

To validate the performance aspect of your application/ APIs

To first validate the functionality and then also test the performance aspect of it. (a) + (b)

Now when you know about your test objective, you should also look at the technologies being used for your Web Services as that can make a difference as well. If you use Microsoft platform, chances are your services are built on:

ASMX – Legacy aps

WCF – New apps

WF – Window Workflow Foundation

Choosing the appropriate Test Type (Unit Test or Web Test)

Please find below the recommendation on the type of test you should decide depending on your test objective defined in the section above

Unit Test – provides you with flexibility to directly access your Web services APIs programmatically using familiar c# and test their behaviour based on your requirements, business rules etc. We will be covering this in depth in the section below

Web Test – also lets you invoke your web services methods, though not through better known record and playback feature of Web Test but by directly adding services method call and parameters, properties. We will be covering this in depth in the section below

At a high level, you can say that unit test is more of glass box testing whereas Web Test gives you ‘black box’ view.

As shown in the image below, Unit Test is an excellent choice when:

You have complex web services involving lot of business rules and logic which you need to validate and primary focus is on detailed functional validation

You are using WCF based services that includes RESTFul API, services returning non-xml kind of data

Web Test is a better choice when:

You have relatively simpler web services where you objective is to test simpler / straightforward functionality and behaviour and primary focus is to test their performance down the line

You are working on legacy app that are based on ASMX kind of services which have a better support in Web Test

Test Types vs. Test Objective Recommendation

Sample Lab Application: Web Service Automation

As many of you might be new to Web Services automation, we have created and shared sample Web Services applications that will help you better understand our helper functions/ Add-ins and sample unit test/web test shown later in the document.

If you already have sample Web Services deployed in your machine, you can skip this section.

Sample Application: Student Administration

This is a sample Student Administration application that comprises of simple functionalities around enrolling students and displaying their details back to the user. The UI app is built using Win forms and services are implemented using both ASMX and WCF for demonstration purpose. There are other sample projects to demonstrate features like Authentication, Caching, Workflow, Asynchronous calling etc.

Sample Student Administration Application UI

Sample Application project structure in TFS

Please refer to Appendix: Deploying Sample Web Services Application to know more about our sample application and instructions around deployment.

At this point we assume you have deployed our sample web services application. In subsequent sections, we will present the various approaches and steps involved in Web Services Automation Design, Development & Execution that you should take to achieve your objective while following the automation standards and guidelines that are recommended.

Web Services Automation – Unit Test Walkthrough

If “Unit Test” was selected as the Test approach for Web service automation based on the analysis and your test objective as covered in above section: Test Approach then proceed with the walk through.

Creating a new Unit Test Project / Reusing Sample Template Unit Test Project

You can create a new project from scratch or using our existing sample template project but it’s important to add your project to source control for version management and change tracking.

SL No.	Topics	Description
1.	Have you identified & marked the automatable tests as PLANNED in Microsoft Test Manager (MTM)	Open your test cases in Test Manager that you want to automate and mark Automation Status = “Planned” for those Whereas the test which are not good candidates for automation are marked as “Not Automated” to show that they are out-of-scope for automation. This is an optional step if you don’t want to run your test from Microsoft Test Manager and map your automation to manual test cases. We however recommend this.
2. a	Have you created new Test project & Added it to Source Control (TFS)?	Every automation project must be source controlled. Add project to Source Control by clicking the check box as shown and give appropriate name.
2.b	Alternatively, Open existing Test project and using it as Template	Launch Visual Studio, Browse and open existing ‘WebServiceAutomation_SampleUnitTest’ template project and add it to Source control. You can change the properties as per your needs. For this sample template project we have SampleUnitTestCases.cs that contains all our sample reusable unit test cases. Please find below a short description on the folder structure of the template project which will be used for case study: 1. Config: All your application configuration details that you need to run your test will go to app.config Example: Also refer to: http://msdn.microsoft.com/en-us/library/a65txexh(v=VS.100).aspx 2. Common Libraries References: Add references to our reusable helper library “WebServicesTesting.Automation.UnitTest.Helper.dll” for Unit Test Web Services Testing (provided as part of the release and documented below) . 3. Service References: Add / update services reference to point to your web services and provide a logical name 4. Scripts: Your actual test are stored in these .cs files along with components like Test Initialize, Test Cleanup etc You can create a folder structure to organize your test methods by grouping them in multiple class files depending on the size of the project. It is advisable to group test belonging to same feature/functional module to be grouped together in a .CS file and given an appropriate name. 5. Solution Configurations & Compilation Symbols With our template solution you get additional solution configuration which will help you run your same test for both functional and load test seamlessly by controlling the sections of the code that you may want to execute for functional and load testing For example. When you do load test, you might not want to do certain validation which makes sense for functional testing alone and these configurations will control that by just selecting the appropriate solution configuration as shown below. a) Custom Solution configurations b) Configuration – Functional Testing As shown below it uses a compilation symbol #WEBSERVICES_FUNCTIONAL_TESTING that is used in our test methods to control the behavior. c) Inside your test method, you will have code like this to control the behavior based on the solution configuration file that is selected For more reference: http://www.codeproject.com/Articles/49448/Use-Visual-Studio-Conditional-Compilation-to-manag.aspx

Unit Test Automation Design & Development (WCF & ASMX based services)

Now that you have your test project in place and our sample reusable unit test cases for automating common web services test scenario, in this section we will be walking you through few sample reusable tests that you can customize it for your own needs or create new unit test for your project scope. The approach shown below would remain same for WCF as well as ASMX based services and we have shown examples of both.

Follow the below steps for Automation design guidance:

SL No.

Steps

Description

Global Settings

a) Set the test context which provides information about and functionality for the current test run.

b) Test Initialize: To perform steps before running each test like instantiating objects, defining global variables including connection strings etc.

c) Test Cleanup: To run code after running each steps

2a.

Add New Test Methods / Reuse Existing Template Test Methods

(Simple Lookup Verification)

Sample Test scenario: Verifying Web service call response value against expected results

Below is sample data driven test method for web services testing that comes with our template project for Unit Test. You can add your new test methods by modeling the below example.

Step 1.

Create a new Test method / Modify our existing sample test method

Step 2.

Instantiate and invoke your web service method (in this example: ASMX based service) as shown in the code snippet below and get the results in a variable of appropriate data type

In our sample example, the web service returns an integer datatype which we cast to String for comparison. Your services method can return any data type like int, double, decimal, bool, string etc and we can use the same helper method below to validate the response.

Step 3.

Get your expected results from data source of your choice as shown below.

Step 4.

Reuse our appropriate library / helper method for validating the web service response against expected results.

In this example CompareString is a reusable method in our helper / library that can be called from your test methods to save effort related to validating the results and marking the test as pass/fail.

Note: Depending on your test scenario, you need to choose different helper/library methods. The complete documentation about our reusable library method is given in sections below

Add New Test Methods / Reuse Existing Template Test Methods

(Simple Transactions Verification)

Sample Test scenario: Verifying Web service call results in database operation like Insert

Below is sample test method when tester needs to validate if the web service call results into a Database Insert by validating the database for ensuring the transaction was committed. Instead of Insert, you might have different operations like update, delete etc. for which we have reusable helper methods in our library.

Here the web service method Enroll_Student (in this case WCF based Service) is inserting a record into the database and the expected result is the query against the database to validate the operation to ensure transaction was committed.

Reuse our appropriate library / helper method for validating the web service response against expected results.

In this example Verify_DBInsert is a reusable method in our helper / library that can be called from your test methods to save effort related to validating the database insert and marking the test as pass/fail.

Note: Depending on your test scenario, you need to choose different helper/library methods. The complete documentation about our reusable library method is given in sections below

Add New Test Methods / Reuse Existing Template Test Methods

(Handling Complex Object Types)

Use this approach as a generic way of comparing / validating web services response of various data types including user defined/abstract data types like Struct, Classes as well as primitive data types like Int, String, Dictionary, List etc.

DTO (Data Transfer Objects) Having the results of a remote method available in an object instance makes it easy to pass this data to a test module or to compare it with the desired outcome

It works by converting the service response (whether abstract or primitive data types) and expected results into standard generic objects which then are finally converted into XML for comparison with the help of our reusable library methods as described below in the example

Sample Test scenario: Verifying Web service response which returns a Struct object against test data from database.

Step 1: Create Data structure for your object which your service returns as

shown below

(In this case it returns a Struct which contain elements like

First_Name, Last_Name etc.

Step 2. This is an optional step for better organizing your code which invoked your web services and returns the response to the main test method for verification.

Step 3: Add your test method

Step 4. You can read your data source (in this case database) and convert it into generic DTO object using reusable method in our library called DTO_PopulateDBResults which takes Dataset as input parameter. DTO Type should be same as the type of your complex object (in this case Student which is struct)

List<Student> ExpectedStudentDTOList = null;

ExpectedStudentDTOList = helper.DTO_PopulateDBResults<Student>(dt);

Step 5. In this step you call you Action code to get your web service response

Actions.DTO_Action complexObj = new Actions.DTO_Action();

Object myObj = complexObj.ComplexObjectSample();

Step 6. Convert web services response into generic DTO using our generic library method called DTO_PopulateObject which accepts Object and structure of the object defined above as parameters.

object ActualStudentDTOList = helper.DTO_PopulateObject<Student>(myObj, s1);

Step 7. Finally the generic DTO objects returned from expected results i.e. Test data source and Web services response are converted into XML and compared using our reusable library method called DTO_XMLComparison and accordingly marks the test method as pass/fail

bool result = helper.DTO_XmlComparison(ExpectedStudentDTOList, ActualStudentDTOList, typeof(List<Student>));

Assert.AreEqual(true, result, "Expected does not match actual");

Step 8. Expected and Actual XML files with the test method name are created under your folder c:\TestResults for your future reference. Please create this folder if it doesn’t exist

Make your scripts data-driven

Irrespective of kind of test methods you have created above

Data-Driven Test

Ensure every variable is parameterized by passing the test data to each test using data-driven capabilities in VS 2010.

Right-click on the test method and select Properties, to view the properties of the test

To start making this a data-driven test, we have to first create a data source for the test to draw from. On the Data Connection String property, click the ellipse button to start the New Test Data Source Wizard

There are three options for creating a data source:

Database -- which allows you to connect to a database or Microsoft Excel spreadsheet

CSV File -- which allows you to use a comma-delimited file

XML File -- which allows you to use an XML file

When you add a data source A Data Source attribute is added to the Test method.

[DataSource("System.Data.SqlClient", "Data Source=(local);Initial Catalog=Sample_DB;Integrated Security=True", "Sample_App", DataAccessMethod.Sequential), TestMethod]

Map your automation test to your test cases in MTM

This is an optional step if you don’t want to run your test from Microsoft Test Manager and map your automation unit test to manual test cases. We however recommend this.

Repeat the above steps for all your test scenarios against your web services to be tested. After you write and debug your scripts its important to run them in a batch or as part of a suite. There are various way to do that and factors to keep in mind as we will see in next section.

Web Services Automation – Web Test Walkthrough

If “Web Test” was selected as the Test approach for Web service automation based on the analysis and your test objective as covered in above section: Test Approach then proceed with the walk through.

Creating a new Web Test Project / Reusing Template Web Test Project

You can create a new project from scratch or using our existing sample template project but it’s important to add your project to source control for version management and change tracking.

SL No.

Topics

Description

Have you created new Test project & Added it to Source Control (TFS)?

Every automation project must be source controlled.

Add project to Source Control by clicking the check box as shown

Alternatively,

Open existing Test project and using it as Template

Launch Visual Studio, Browse and open existing “ WebServiceAutomation_SampleWebTest” template project and add it to Source control. You can change the properties as per your needs.

Please find below a short description on the folder structure of the template project which will be used for case study:

1. Common Libraries References:

a) Add references to our reusable helper library “WebServicesTesting.Automation.WebTest.Helper.dll” for WebTest based Web Services Testing (provided as part of the release and documented below)

b) Reusable Validation Rules

c) Reusable Extraction Rules

2. Service References: Add / update services reference to point to your web services and provide a logical name

3. Scripts: Your actual test are stored in these .webtest files.

You can create a folder structure to organize your test methods by grouping them in multiple class files depending on the size of the project. It is advisable to group test belonging to same feature/functional module to be grouped together in a .CS file and given an appropriate name.

For this sample template project we have many reusable webtest that can be reused / customized for your needs.

4. Test Data: You can choose to have you test data as CSV or XML or Database as support by Visual Studio Testing framework for making your test data driven.

WebTest Automation Design & Development (WCF & ASMX based services)

Now that you have your test project in place, in this section we will be walking you through our sample reusable Web test for automating common web services test scenario that you can customize it for your own needs or create new additional Web tests for your project scope.

Follow the below steps for Automation design guidance:

S No.	Steps	Description
1a.	Add New Web Test / Reuse Existing Template Web Test (Dataset Verification for WCF/ ASMX Service)	**Sample Test scenario: Verifying Web service method that returns a dataset against the expected test data from Database If you are adding a new web test from scratch then follow the steps: Step 1. Open your web service method request XML by launching WCFTestClient** utility or any other way you are comfortable with. Step 2. Add a new Web Test or Reuse our existing template Web Test and customize them Step 3. Right click the node and press “Add Web Service Request” and provide your web service URL Step 4. Right click the node and press “Add Header” and provide SOAPAction value as shown below by retrieving it from request XML of Step 1. Step 5. Add <string body> a) From your request XML as shown below, extract Envelope and Body (you shouldn’t copy Header part) <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"> <s:Body> <Display xmlns="http://tempuri.org/" /> </s:Body> </s:Envelope> b) Add this to to you string body now Step 6. Add our reusable Validation and Extraction Rule called DatasetValidation that reads the expected data from database using SQL query and validate it against the response from web services by using XPathExpression Note: Depending on your test scenario, you may need to choose different Extraction and Validation Rule complete documentation about our reusable library method is given in sections below
1b.	Add New Web Test / Reuse Existing Template Web Test (Method Prototype Verification for WCF/ ASMX Service)	**Sample Test scenario: Verifying Web service method parameters and response values against expected test data. Repeat Step 1 to 5 from 1 a) above as explained Step 6. Add our reusable Extraction Rule called ParameterExtraction and provide XPathExpression for example Enroll_Student and it will retrieve all the parameters name and their respective values and store it inside ContextParam Step 7. Step 6. Add our reusable Validation Rule called ParamValidation** and provide ParamValues that contain expected test data as shown below along with contextParam name as defined in Step 6. Step 8 & 9. As shown in this web test you can also add our reusable ServiceResponseExtraction and ServiceResponseValidation rules from our helper class for validating the response values against expected test data. Step 10. After you run this web test, the result will show Pass / Fail with context tab showing the discrepancies if any. Note: Depending on your test scenario, you may need to choose different helper/library methods. The complete documentation about our reusable library method is given in sections below
2.	Make your scripts data-driven	Data-Driven Test Ensure every variable is parameterized by passing the test data to each test using data-driven capabilities in VS 2010. To start making this a data-driven test, we have to first create a data source for the test to draw from. On the Data Connection String property, click the ellipse button to start the New Test Data Source Wizard There are three options for creating a data source: Database -- which allows you to connect to a database or Microsoft Excel spreadsheet CSV File -- which allows you to use a comma-delimited file XML File -- which allows you to use an XML file

Automation Execution & Deployment (Unit Test/Web Test)

Ensure you have configured test settings to capture logs, code coverage, test impact etc. and you also need to keep in mind that your script should be able to successfully run in multiple environment (for ex. Test to UAT)

After automation design and debugging is completed, you can now run and deploy your automation scripts using the below steps:

SL No.	Areas	Description
1.	Configuring Test Settings	Test Settings a) Data and Diagnostic: Enable appropriate diagnostic are selected as shown below (Code Coverage, Event Log, Test Impact etc.) b) Define other settings like Roles, Test Timeouts, Setup and Clean scripts
2.	Command Line Execution (MSTest.Exe)	*Grouping & Classifying Test Methods using various parameters* It lets you run groups of tests based on their assigned categories without the requirement to maintain test lists. Go to Test View and you can filter and view your tests by setting these parameters MSTest.EXE Options: MSTest.exe is the command-line command that is used to run tests. This command has several options that you can use to customize your test run. You can specify these options in any order on the MSTest.exe command line. Example: mstest /testcontainer:"C:\ WebServiceAutomation_SampleUnitTest" /test:testproject32\generic Refer to http://msdn.microsoft.com/en-us/library/ms182489.aspx
3.	TCM.exe or Test Execution from Microsoft Test Manager (Optional though recommended)	a) If you have mapped you automation unit test with test cases in Microsoft Test Manager then you can use TCM.exe as shown below to run your web service test and it will also update your test cases results in MTM without any custom code. You don’t need to use MSTest.exe in that case Example: tcm run /create /title:title /planid: id /collection:CollectionURL /teamproject:project (suiteid:id /configid:configid \| /querytext: query) [/settingsname:name] [/owner:owner] [/builddir:directory] [/testenvironment:name] [/login:username,[password]] [/include] Refer to : http://msdn.microsoft.com/en-us/library/dd465192.aspx & http://msdn.microsoft.com/en-us/library/ff942469.aspx b)You can also Run Automated Tests from a Test Plan Using Microsoft Test Manager. To run the latest version of your automated test, you must make sure that the correct build is selected for your test plan. You must also use a physical or virtual environment to run your automated tests. This is out of scope of this guidance however we recommend this the most. Refer to: http://msdn.microsoft.com/en-us/library/dd553270.aspx

Web Service Performance Testing using Unit Test/Web Test + Load Test

Now when you have your Web Services Test automated either using Unit Test or Web Test as suggested in this guidance above, you can go ahead and create you Load Test to performance test your web services.

SL No.	Areas	Description
1.	Configuration – Load Testing	a) Ensure you have selected our custom solution configuration for Load Test i.e. Debug – Load Test b) As shown below it uses a compilation symbol #WEBSERVICES_LOAD_TESTING that is used in our test methods to control the behavior. c) Inside your test methods, you will have code like this to control the behavior based on the solution configuration file that is selected
2.	Creating Load Test using Web Services Unit Test or Web Test (Designed in above sections)	Step 1. Add a new Load Test Step 2. Give a scenario name for your Load test Step 3. Specify your load pattern (Constant Load/ Step Load) Step 4. Choose an appropriate test mix Step 5. This is the most important step where you have an option to select Unit Test or Web Test to be added to your Load test depending on the kind of approach you had selected for your Web services automation above Press Add and select your “Web Services Unit Test” OR “Web Services Web Tests” created in the sections above that you want to performance test Press OK and adjust Distribution of tests. Step 6. Refer to below section for adding custom web services counters other than standard counters that you may want to monitor Step 7. Specify Load run settings Step 8. You load test is created Our sample Unit Test and WebTest template project has this reusable SampleLoadTest.loadtest sample load test for you to model your load tests around.
3.	Running Load Test and analyzing results	For Web services Load testing, we recommend you adding following custom counters against the web services server (where the web services are hosted): ServiceModelEndPoint 3 ServiceModelEndPoint 4 ServiceModelOperation ServiceModelServices Please follow Adding custom counters set using Load Test Editor for detailed steps. Step 1. Open you load test. Right click you counter set and add “Custom Counter Sets” Step 2. Right click and now click “Add Counters” Select Performance Category to include counters related to Web Services as shown below Step 3. Repeat it for all the categories as shown below. Tip: You can enable performance counters for a WCF service through the app.config configuration file of the WCF service as follows: <configuration> <system.serviceModel> <diagnostics performanceCounters="All" /> </system.serviceModel> </configuration> *Sample Load Test Summary Report*

Web Services Helper Documentation for Unit Test & Web Test Add-ins

This section provides documentation of the complete list of reusable helper functions (WebServicesTesting.Automation.WebTest.Helper.dll) and web add-ins (WebServicesTesting.Automation.UnitTest.Helper.dll) for your reference and usage. We have shown few of these in sample walkthrough above to give you a glimpse of the same.

Category	Title	Description	Unit Method Signature & Example invoke call	Web Custom Extension (Extraction & Validation Rules)
Lookup Verification	Verifies the Scalar value or multiple scalar values of even different data types returned by services against expected results	This method will let you compare the scalar values of different types like <INT>, <STRING> etc. returned by the service method call converting them to <string>against the expected values	public void CompareStringArray(int condition, string query, string[] ArrgetAgeActual) Example: obj.CompareStringArray(2, query, ArrgetAgeActual);	For scalar value Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam Validation Rule ServiceResponseValidation Context Parameters 1.Expected 2.ContextParam For multiple values Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the dictionary returned by web service from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the values of <DICTIONARY> data type returned by services against the expected results	This method will let you compare the <Dictionary> <key, value> values returned by service method call against the expected values <key, value>.	public void CompareDictionaryWithExpectedArray(Dictionary<int, string> ActualDic, string[,] ArrgetAgeExpected) Example : obj.CompareDictionaryWithExpectedArray(col_Symbols, ArrgetAgeExpected);	Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the dictionary returned by webservice from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the values of <LIST> data type returned by services against the expected results	This method will let you compare the values of data type <LIST> returned by service method call against the expected values	public string[] CompareList(string query) Example: string[] ArrgetAgeExpected = obj.CompareList(query);	Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the List returned by webservice from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the result set returned by services against expected results	This method will let you compare the result set returned by service method call against the expected result set	Method Prototype public void Verify_Dataset(string ConnString, string Query, DataSet dst) Method Invocation UnitTest help = new UnitTest(); help.Verify_Dataset(conn, query, set);	Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the dataset returned by webservice from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the row count of the values returned by Web Services against expected result	This method will let you validate the row count of the values/list/dataset returned by service method call against the expected row count	Method Prototype public void Verify_RowCountDataSet(int expected, int actual) Method Invocation UnitTest help = new UnitTest(); help.Verify_RowCountDataSet(expected, actual);	Extraction Rule RowCountExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the row count and stores the value to Context Param specified by the user. Validation Rule RowValidation Context Parameters 1.ConnString 2.Query 3.ContextParam 4.ContextParam This method validates the value set to the context param and the one returned by the specified query
Metadata Verification	Verifies the called web service method has schema as expected	This method validates the XML file to make sure it fits to your application's needs to prevent any exception in your code. To check whether the XML document conforms to an XSD Schema, the document must be validated against that XSD Schema	Method Prototype public bool Verify_Schema(string targetNameSpace, string xsdPath,string requestXml) Event Handler to raise error private static void ValidationEventHandler(object sender, System.Xml.Schema.ValidationEventArgs arg) Method Invocation UnitTest help = new UnitTest(); bool validation = help.Verify_Schema(targetNameSpace ,XsdPath,requestXml);	Validation Rule SchemaValidation Context Parameters 1.targetNameSpace 2. xsdPath
Metadata Verification	Verifies that web service methods exist and their parameters definition (like input parameters, return type) against expected result	This method will let you smoke test / BVT if the expected list of Web Service methods exists and their definition like input parameters and return type matches.	Method Prototype public void Verify_methodPrototype(object myObject, string Methodname, string paramlist, string returnTypelist) Method Invocation UnitTest help = new UnitTest(); help.Verify_methodPrototype(obj, methodlist, paramlist, returnTypelist);	Extraction Rule ParameterExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the parameter passed to the request and set to the contextparam Validation Rule ParamValidation Context Parameters 1.ParamValues 2.ContextParam This method validates the value set to the context param and the Values passed to the ParamValues The above extraction & validation rule is used to validate the parameter passed The below extraction & validation rule is used to validate value returned by the webservice Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam Validation Rule ServiceResponseValidation Context Parameters 1.Expected 2.ContextParam
Transactional Verification	Verifies that called web service method creates the output file as expected	This method verifies the output file (e.g. CSV, XML, TXT) has got created by the web service method call	Method Prototype public void Verify_FileCreation(string FilePath) Method Invocation UnitTest help = new UnitTest(); help.Verify_FileCreation("D:\\employees.xml");	Validation Rule FileCreationValidation Context Parameters 1.FilePath
Transactional Verification	Verifies that web service INSERTS/CREATES a new record into the database	This method verifies the web service call results into new record insertion in a given database	Method Prototype public void Verify_DbInsert(string ConnString, string Query, string expected) Method Invocation UnitTest help = new UnitTest(); help.Verify_DbInsert(conn, query, expected);	Extraction Rule ParameterExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the parameter passed to the request and set to the contextparam Validation Rule DbInsertValidation Context Parameters 1.ConnString 2.ContextParam 3.Query 4.ContextParam This method validates the contextparam and the values returned by the query
Transactional Verification	Verifies that web service UPDATES existing records into the database	This method verifies the web service call results into update of existing record(s) in the given database	Method Prototype public void Verify_DBUpdate(string ConnString, string Query, string expected) Method Invocation UnitTest help = new UnitTest(); help.Verify_DBUpdate(conn, query, expected);	Extraction Rule ParameterExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the parameter passed to the request and set to the contextparam Validation Rule UpdateDbValidation Context Parameters 1.ConnString 2.ContextParam 3.Query 4.ContextParam This method validates the contextparam and the values returned by the query
Transactional Verification	Verifies that web service DELETES records into a database	This method verifies the web service call results into deletion of existing record(s) from a given database	Method Prototype public void Verify_DBDelete(string ConnString, string Query) Method Invocation UnitTest help = new UnitTest(); help.Verify_DBDelete(conn, query);	Validation Rule DeleteDbValidation Context Parameters 1.ConnString 2.Query This method verifies the rows returned by query is 0
Workflow service	Verify that web service executed the workflow and end result is as expection	_________For this one we just need a test method in our test file instead of a separate helper method____	Sample code to verify the result returned by the workflow IDictionary<string, object> _d = new Dictionary<string, object>() { { "userArgument", arg } }; _d = WorkflowInvoker.Invoke(new Workflow1(), _d); Int32 outPut = (Int32)_d["OutputValue"]; Assert.AreEqual(expected, outPut); output can be validated with expected for its correctness	WorkFlow cannot be invoked. We can invoke the web service to verify the result returned by service method against the expected result. Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam Validation Rule ServiceResponseValidation Context Parameters 1.Expected 2.ContextParam
Authentication	Verifies that user credentials & certificate are validated against the server to authenticate the user trying to access the web service	This method verifies the user credentials and certificate to ensure that he is authenticated to get the results from the web service	var ServiceendPoint = new EndpointAddress(new Uri("http://localhost:91/Service1.svc"), EndpointIdentity.CreateDnsIdentity("MyWebSite")); var binding = new WSHttpBinding(); binding.Security.Mode = SecurityMode.Message; binding.Security.Message.ClientCredentialType = MessageCredentialType.UserName; var result = new Auth_Service.Service1Client(binding, ServiceendPoint); try { result.ClientCredentials.ServiceCertificate.Authentication.CertificateValidationMode = X509CertificateValidationMode.None; result.ClientCredentials.UserName.UserName = auth_username; result.ClientCredentials.UserName.Password = auth_passwd; time = result.GetServertime(); }
Encryption	Verifies that web service is encrypting the message/transport layer when returning data	This method will verify that web service is encrypting and decrypting the data as required by business.	Configuration appConfig = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None); ServiceModelSectionGroup serviceModel = ServiceModelSectionGroup.GetSectionGroup(appConfig); Console.WriteLine("Configuration Name : Address : Binding : Contract"); foreach (ChannelEndpointElement endpoint in serviceModel.Client.Endpoints) { if (endpoint.Contract == "Auth_Service.IService1") { binding_used = endpoint.Binding; break; } else { } } Auth_Service.Service1Client result = new Auth_Service.Service1Client(); result.ClientCredentials.ServiceCertificate.Authentication.CertificateValidationMode = X509CertificateValidationMode.None; result.ClientCredentials.UserName.UserName = auth_username; result.ClientCredentials.UserName.Password = auth_passwd; string time = result.GetServertime(); TestContext.WriteLine("Bindings used by this contract is " + binding_used);
Caching	Verifies that data returned by called web service is returned from cache	_________For this one we just need a test method in our test file instead of a separate helper method____ Caching is frequently used as an optimization in distributed systems. It can be used to avoid re-doing computations or complex database access when the results remain valid for an extended period of time. In this case, subsequent requests for the same information can be served with the cached version rather than repeat the processing with the associated overheads.	string url = "http://localhost/Cache/Service1.svc/xml/12"; Uri uri = new Uri(url); WebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = "GET"; HttpWebResponse response = (HttpWebResponse)request.GetResponse(); Stream receiveStream = response.GetResponseStream(); StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8); string firstResponse = readStream.ReadLine(); Above code snippet shows to retrieve server response. To test caching two request should be made befor cache expires and should be validated.	Extraction Rule CacheExtraction Context Parameters 1.XpathExpression 2.ContextParam 3.Delay To test caching two request should be made before caching expire and one after caching expire. Parameter Delay has to be passed for the second request And plugin DelayPlugin should be associated with the second service CacheOutcomePlugin ContextParameters 1.ContextParam1 2.ContextParam2 3.ContextParam3
Caching	Verifies that data returned by called web service is returned from database when cache is expired	_________For this one we just need a test method in our test file instead of a separate helper method____	string url = "http://localhost/Cache/Service1.svc/xml/12"; Uri uri = new Uri(url); WebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = "GET"; HttpWebResponse response = (HttpWebResponse)request.GetResponse(); Stream receiveStream = response.GetResponseStream(); StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8); string firstResponse = readStream.ReadLine(); Above code snippet shows to retrieve server response. To test caching two request should be made before caching expire and one after caching expire and should be validated.
Non-XML Data	Verifies that called web service method sends/returns non-xml data (like blob, image) as expected	This method will verify that web service method is storing or retrieving the non XML data like blob from file system or database	Refer our reusable [TestMethod] public void VerifyNonXMLData()
Asynchronous messaging	Verifies that called web service method if implemented as asynchronous then returns the data as expected at a later point in time	This method verifies the asynchronous behavior of web service method as expected and implemented.	Refer our reusable [TestMethod] public void VerifyAsync_Response()
Exception and Error handling	Verify that service returns appropriate error message when there is a fault exception like underlying database connectivity issue to ensure exception and error handling	This method will verify that web service method that is called handles the exceptions like timeout, connectivity/networking issue as expected	<serviceDebug includeExceptionDetailInFaults="True" /> Set the IncludeExceptionDetailInFaults property to true to instruct to return managed exception information to the client in SOAP faults to ease debugging try { WCFServiceRef.Service1Client obj = new WCFServiceRef.Service1Client(); obj.Display(); } catch (FaultException e) { string str = e.Message.ToString(); TestContext.WriteLine(str); }	Extraction Rule ExceptionExtraction ContextParameter 1.contextParam Validation Rule ExceptionValidation ContextParameter 1.contextParam 2.ErrorMsg ExceptionPlugin ContextParameter 1.contextParam Plugin is used to pass/fail the test case contextParam name provided in extraction/validation/plugin should be same
Merge	Verifies the result obtained from multiple web services calls by merging them matches against consolidated expected output	This method verifies when similar result is obtained from multiple web services and then need to be validated against single expected result set Think of a scenario where you are getting similar result from various source - each a different web service call. The tester wants to validate the consolidated output from these services by comparing it against expected values	WCFServiceRef.Service1Client obj = new WCFServiceRef.Service1Client(); string expected = "5 Test"; string exmaple1 = obj.mergeExample().ToString(); string example2 = obj.mergeExample1(); string actual = exmaple1 + " "+ example2; Assert.AreEqual(expected, actual);	Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam The extraction rule is used with both the request to extract the response from the server Associate the validation rule with the 2^nd request Validation Rule MergeTwoServcResponseValidation ContextParameter Context Parameters 1. ContextParam1 2.ContextParam2 3.expected
DTO based Validations	DTO (Data Transfer Objects) based validation is a generic way to validate web services response of any data type against expected result by converting them into generics objects	It works by converting the service response (whether abstract or primitive data types) and expected results into standard generic objects which then are finally converted into XML for comparison with the help of our reusable library methods as described below in the example	public List<T> DTO_PopulateObject<T>(object DTOObject, object myObj) public List<T> DTO_PopulateDBResults<T>(DataTable dt) where T : class public bool DTO_XmlComparison(Object ExpectedDTOList, Object ActualDTOList, Type ObjectType) Examples: ExpectedStudentDTOList = helper.DTO_PopulateDBResults<Student>(dt); object ActualStudentDTOList = helper.DTO_PopulateObject<Student>(myObj, s1); bool result = helper.DTO_XmlComparison(ExpectedStudentDTOList, ActualStudentDTOList, typeof(List<Student>));

Tools to speed up Web Services Testing

WCF Load Test

This tool helps us in accelerating the generation of Skeleton unit test which can be used for load test of Application.

This tool takes the WCF or Fiddler2 message log file and a WCF client proxy, or a WCF interface contract, and generates a unit test that replays the same sequence of calls found in the message log file for WCF and ASMX services. The code generated is easily modifiable so that data variation can be introduced for the purpose of doing performance testing.

The tool will also integrate with Visual Studio 2005/2008 Team Edition for Software Testers and Visual Studio 2010 Ultimate and installs a wizard for creating the message log and processing it from within Visual Studio.

Tool can be downloaded from

http://wcfloadtest.codeplex.com/releases/view/51762

This tool is an add-in to VS and after running the .msi, open Visual studio and from the test menu, create a new test .It contains a template for the WCF Test.

1. Create a new Test Project and select the WCF test from the templates available for the test.

2. Click on Next

3. Give the path of the executable, it’s your application exe files, in this case it’s the sample App application which we are using for the generation of unit test.

4. Click on Run, will run the application and load the data as expected.

5. Test the application; change the values in the List of Students to fire the service with different set of Data. Once this is done, close the application.

And click on Next on the ‘New WCF Service Test Generator wizard’

6. There will be screen which depicts what all methods were invoked when we navigated the application with different set of inputs.

7. Click Next, Now we need to add the Dll/exe which define the contracts which are implemented for the WCF service which we have consumed.

8. Click on finish and the Unit tests are created for actions which we have performed on the Application.

Web Test Plug In for Silverlight + RIA based services

When testing Silver Light applications, we frequently run into dealing with non-Html content types. This is because most of the Silver Light applications use WCF RIA web services. Typically, this will be a binary format called MSBin1. Also, Silver Light applications may use other content types such as JSON, SOAP and REST to communicate with server. In general there is significantly greater use of Web Services and JSON in today's web applications.

The Web Test Plugins for Web Service Messages leverage these extensibility points to provide the following functionality for all these content types:

1. Select the Web Service plugin to enable Web Service message plugins before recorder shows up when we create a new web test project.

2. After Recording is completed it adds a reference to the runtime assembly to project, and adds the web service plugin to the Web Test

3. In recording result we can see a new tab named as “Web Service”

4. Right click on the selected node, and select “Add Extraction Rule” or “Add Validation Rule” from the context menu.

Reference- <http://blogs.msdn.com/b/mahipalkante/archive/2010/04/07/web-test-plugins-for-web-service-messages.aspx>

Web Services Automation Benefits

· Increase the quality of the application and find issues/bugs quicker and easily by validating web services for functionality without relying or testing the UI layer too much which is many times slower.

· Many times when UI is under development or for V1.0 projects where UI is changing a lot often, UI automation doesn’t give you’re the ROI in short term and Web Services automation can help you quickly validate the functionality and business layer.

· The effort and investment on automating Web Services for functionality will also pay off later to quickly build you performance test suite with lesser effort and reuse.

· This will also help to scale up when doing performance testing to ensure that web services behave as per SLA and as-expected when put under lot of stress and load.

· As a performance engineer, you will be able to do effective performance (load/stress) testing of web services along with validating their behaviour.

· As developer/consultant, you will be able to use web services automation as part of continuous Integration to improve the build and code quality.

· As a support engineer, you can also run automation tests quickly to find the culprit such an unintentional regression impact and that will help in debugging and troubleshooting critical failures.

Appendix

i. Getting Started with VS 2010 Unit Test & Web Test

Walkthrough: Creating and Running Unit Tests

Web Performance Test Walkthroughs

Introduction to WCF Testing

ii. Deploying Sample Web Services Application

Please find below the steps to get started with our sample web services

Step 1: Deploy sample DB

From our sample solution, run Create_Sample_DB.sql which will create the Database and Tables required for our sample app.

Step 2. Deploy our services

a) Sample WCF Service –

This project contains the WCF services will be consumed by Sample_APP_UI. It contains the contract as well as the implementation of the contracts. This project also uses the LinQ feature for integration with the database.

Below is sample step to deploy services on your machine

i) Launch solution and publish “WcfService” project to FileSystem

ii) Add an application to your IIS and provide the File system directory and App pool details

iii) Browse and test your services in browser or using WCFTestClient

Browser: From IIS, select your application and press Browse

WCFTestClient:

- Launch VS Command Prompt and type WCFTESTCLIENT

- Add URL of your web service

- Click on any test method, provide request parameter and invoke

b) Sample ASMX Web Services

This project contains the services which are built on ASMX types. These services are also consumed by the Sample_App_UI.

For deployment follow the steps given in above step (a).

Note: You can skip this as well if your scope doesn’t include testing ASMX based services

c) Sample Windows Workflow Foundation Services

This project contains the sample WCF service which is consumed by the workflow application.

For deployment follow the steps given in above step (a).

Note: You can skip this as well if your scope doesn’t include testing Workflow based services

d) Sample Authentication Project

This project contains the WCF service which entails the authenticate service. We have used the “UserNamePasswordValidator” validate method for the verification of the client credentials to authenticate the client.

The main intent is to check the authentication of the client, so this service has 1 contract implemented “GetServertime” which returns the current time at the server.

Note: You can skip this as well if your scope doesn’t include testing Authentication of your services

e) Sample Non-XML Data Project

This project contains the WCF service which is required for returning the non xml data from the service. For this case we are using the service to return the image file using the stream.

There are two response from the service implemented under this project.

- Buffered response

- Streamed response

Note: You can skip this as well if your scope doesn’t include testing of Non-XML based response from web services

f) Sample Web Services Caching Project

This project contains a WCF service which contains the service representing the cache behaviour of the WCF service. For this project we have leveraged the Asp.Net cache in the WCF. The response is cached for 60 seconds if the request URL doesn’t changes. On changing the parameter to the request URL or elapse of 60 seconds the call again to this service returns the fresh response from the service.

Note: You can skip this as well if your web services don’t use caching

g) Sample Asynchronous Web Services Project

This project shows another feature of the WCF asynchronous call where in client sends a request to the server and continues with the regular task and on completion of the service i.e. response comes from server the required action is performed using the response received.

For this example we have used client making the Asynchronous call using the call-back mechanism. This project contains the contract as well as the Windows client which uses the service return.

Note: You can skip this as well if your web services don’t use Asynchronous calls

Step 3. Run Sample Client apps

a) Sample UI application

This is the main project in the solution which consumes different services WCF and ASMX. This project is a sample app to demonstrate the services when consumed from UI and their return

Note: Please update the service reference to point to the URL of the our sample web services (WCF and ASMX) as shown below by using “Configure service reference”

You can now run sample UI app that consumes WCF and ASMX based services.

b) Sample Authentication Console Application

This project in the solution contains the client which will test service, and provide the valid credentials so that the service returns the valid return.

This is a console based application which hosts the Auth_Service and send the valid credentials to test it.

c) Sample Workflow Console Application

This is a Workflow project which uses the WCF services with in the workflow, and executes the rest of action in the workflow.

This project is just a sample to be demonstrated on how the services are integrated with the WF app.

iii. Sample Restful Services

iv. Request/Request XML for services not supported by Fiddler/ WCFTestClient

Working on a project where we create WCF Services there may be a need to capture the raw xml passed in to the service operation and also capture the reply xml sent back by the service. Our reusable code captures raw xml even for the scenarios (e.g. Non Xml data being returned by service) where WCF Test Client or fiddler are not able to trace the request\response xml.

Reusable Code snippet:

ImageServc.StreamingServiceClient ClientHandle = new ImageServc.StreamingServiceClient();

ResponseXmlInterceptingBehavior responseXmlInterceptingBehaviour = new ResponseXmlInterceptingBehavior();

ClientHandle.Endpoint.Behaviors.Add(responseXmlInterceptingBehaviour);

OutputFile = ClientHandle.GetData("Dryden");

string requestXml = responseXmlInterceptingBehaviour.ResponseXmlInterceptor.RequestXml;

v. Binding & Hosting configuration & consideration

This project one of the features of WCF to host the WCF service on the console. In this project we have used the basic test for the call of contract/method of WCF which is hosted on the console app.

Please refer the snapshot which will give you an idea on hosting the service on the console.

Another sample unit test

vi. Data driven testing

Use below link if you need more information about Data driven testing using Coded UI and access/store data in MS EXCEL from Coded UI

Data Driven Testing by Mathew

http://msdn.microsoft.com/en-us/library/ee624082(VS.100).aspx

http://msdn.microsoft.com/en-us/magazine/cc163536.aspx

Test Automation Guidance using VS 2010 Coded UI

2011-12-11T04:33:00.001+05:30

Author:

Raj Kamal (rajkamal@microsoft.com)

New !!! On special demand, sharing the below content in PDF format as well as sample reusable Coded UI Framework for your user

http://blogs.msdn.com/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-01-42-73-CodedUI/2086.Automation-Guidance-_2800_VS-2010-Coded-UI_2900_-ExternalCommunity.pdf

http://blogs.msdn.com/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-01-42-73-CodedUI/0167.CodedUISampleFramework.rar

Objective

The objective of this artifact is to provide a L200 test automation guidance leveraging VS 2010 Coded UI features to foster a continuous, consistent and standard automation approach. The document can also be used as a quick jump start guide for beginners.

However the bigger objective is Build, Deploy and Test leveraging Visual Studio Lab Management features, which will allow Automation Testers to reap the benefits of the following

· Integrating with the build process for automatic build deployment

· Execution on VMs

· Capabilities of getting the test automation results reported as part of Test Manager (MTM) for better reporting.

Pre-requisites:

Ensure you have ‘CodedUISampleFramework’ sample Coded UI project

For VS 2010 installation & Configuration, go to Appendix: VS 2010 Installation Reference section

If you are new to Visual Studio Coded UI, go to Appendix: Getting started with VS2010

Ensure your application is supported by Coded UI, go to MSDN link

Jump to Case Study, if your objective is to get started with Coded UI using our recommended framework and practices.

Test Automation Process using Visual Studio 2010

It is not enough just to automate your tests but there is also an important need to automate the build process as well as automatic installation and deployment. This will eliminate manual intervention required to trigger the automation due to build and deployment dependencies.

Automated Build, Deploy and Test framework will also enable us to find out the build quality at the earliest by publishing the test results for every run against a new build.

You can run the test suites that are most important for your continuous builds because too many tests can delay the completion of build. Make sure that you run these tests for the current iteration.

Figure 1: The topology required for setting up Build, Deploy and Test environment using various VS components like Build Controller and Agents for Build, Test Controller and Agent for distributing test and collecting results, Virtual environment for Deployment and Execution (Reference taken from MSDN)

To run unit or functional automation tests, we can create a set of environments that are managed Hyper-V images in Microsoft Test Manager. Tests that are created at the start can be used as part of continuous integration.

1. For Daily build, we want to run very minimal set of test which are like Build Verification test or Smoke test

2. For Weekly builds, we want to run “Regression test” which will be more in number.

Figure 2: Steps for Build, Deploy and Test workflow

If you plan to run your automated tests by using Microsoft Test Manager, you must use a physical or virtual environment that contains a set of roles to run your tests from your test plan.

**Build, Deploy and Test using Visual Studio Lab Management will be demonstrated using the below case study in future editions of this document.

We will now learn Coded UI automation features as well as our recommended automation framework and best practices in rest of the document.

Case Study: Coded UI Automation

Case Study

You have been given the task to test the basic functionality of Microsoft Bing www.bing.com and automation is in scope. You know that Microsoft Visual Studio Coded UI can be used to automate your functional tests. We assume that you have already written your functional test using MTM

We are going provide guidance on Coded UI automation using a real world case study. We assume you are familiar with Microsoft Test Manager for writing test cases (otherwise please jump to Get started section in Appendix)

Here is a sample snapshot of your test cases in MTM:

In subsequent sections, we will present the various approaches and steps involved in Automation Development Lifecycle that you should take to achieve your objective while following the automation standards and guidelines that are recommended.

Step 1: Automation Planning

Like Project Planning or Test Planning, Automation planning is equally important to ensure that the teams are not directly jumping into automation scripting without taking all the important factors into consideration which are extremely critical for providing accurate estimates, deciding the right framework, choosing the right approach, the long term maintenance of automation and getting early ROI (Return on Investment).

This activity will be primarily focused on identifying the feasibility and the scope of automation.

Figure 3: Steps involved in Automation Scope and Planning

Taking our case study, we will walk you through each of these steps below:

a) Identify Automatable Test Cases, Scenarios & plan Order of Execution

We must clearly know beforehand the test cases that are good automation candidates, the scenarios that can be converted into library methods for reuse and the sequence of execution if the tests are dependent on each other.

SL No.	Checklist	Description	Y/N
1.	Have you identified and marked the automatable tests as PLANNED in Test Manager?	Open your test cases in Test Manager that you want to automate and mark Automation Status = “Planned” for those For the case study, we have marked the automation candidate as “Planned” as shown below. Whereas the test which are not good candidates for automation are marked as “Not Automated” to show that they are out-of-scope for automation.	☐
2.	Have you identified your data-driven test cases?	Identify your test cases that are data-driven (this will be required during automation design) For our case study, look at the example of one data-driven test related to Bing where you want to perform search multiple times for the list of values provided as Test data	☐
3.	Have you identified the common functionalities/scenarios OR SHARED STEPS which can be automated for Reusability?	Ensure that you identify the Shared Steps or Common functionalities that can be later be converted into methods inside your application specific libraries to reduce automation effort and redundancy For the case study, here is one of the examples of Shared step or common functionality which can be added to the automation library. Note: This will be used during automation design	☐
4.	Have you identified the precedence / sequence of execution for your automated tests?	Ensure that you mark the sequence / precedence for the automated tests that have to be run As shown below, define the order of execution for your automaton scripts using OrderID field	☐

b) POC (Proof of Concept)

Now that we know the scope, we must also ensure that the application or technology in hand is actually supported by VS Coded UI Test by doing a POC on key scenarios. Many a times there might be custom controls and other components that are not automatable / supported by Coded UI out-of-the-box or need customization or are not worth automating due to low ROI. The outcome of this activity should help you decide your approach and next steps. You may mark the non-automatable test cases as out-of-scope for automation.

SL No.	Checklist	Yes/No
1.	Have you done a feasibility study to ensure VS 2010 Coded UI supports your application? Identify key scenarios of your application and ensure that they are supported by VS 2010 Coded UI by coding few scripts and checking the results.	☐
2.	Does your application have 3rd party controls or custom controls? Ensure that you call out the tests which are not supported by VS 2010 as “out-of-scope” for automation.	☐
3.	Testability: Have you worked with the development team to ensure that the properties of the objects are exposed and set appropriately for smooth automation? It is extremely important to work with the development team to work on the testability right from the design phase to make automation easier later in the cycle.	☐

c) Automation Estimation

The estimation will decide the success or failure of your automation project as it has been observed that automation can take much more time than anticipated if all the key tasks involved in automation lifecycle development are not clearly thought of while estimating.

The following major activities in each phase of automation must be estimated.

Automation Initiation

· Tool/Framework Feasibility Study

· POC

Framework Design/Implement

· Design and Implement Various Framework Components

· Testing the Framework Components and End to End Testing

· Training Resources on the Framework/Tool

· Environment Setup

Script Development Phase

· Script Creation

· Script Debugging

· Script Review

· Review Comments Incorporation

Script Execution Phase

· Environment Setup

· Result Analysis

Follow the below checklist while coming up with you initial estimation apart from effort for automation design and execution.

SL No.	Checklist	Y/N
1.	Check if you have included automation planning effort in your estimation including POC (Proof of Concept)- if any	☐
2.	Check that you include environment preparation (installation, configuration) effort into estimations.	☐
3.	Have you identified the Shared Steps or Common functionalities that can be later on converted into methods inside your application specific libraries to reduce automation effort and redundancy	☐
4.	Ensure that you include effort for building framework reusable components (Libraries)	☐
5.	Ensure that you include extra effort required if third party or custom controls are there in the application.	☐
6.	Ensure that you include effort for code review, walkthroughs and compliance with the standards defined. Ensure that you call out the tests which are not supported by VS 2010 as “out-of-scope” for automation.	☐
7.	Ensure that you include effort for code review, walkthroughs and compliance with the standards defined	☐
8.	Ensure that you include effort for debugging and lead time to resolve bugs discovered in the product (troubleshooting and workarounds)	☐
9.	Ensure that you include effort for reporting and result analysis.	☐

Now that we have the clarity on the scope, feasibility and estimates, we can move the next logical step which is creating your automation solution

Step 2: Creating a new Coded UI Project & Adding project to Source control

You can create a project from scratch or by using an existing template but it’s important to add your project to source control for version management and change tracking.

SL No.

Topics

Description

Y/N

Have you created new Coded UI project & Added it to Source Control (TFS)?

Every automation project must be source controlled.

Add project to Source Control by clicking the check box as shown

Project Check-in Path:

All application / project specific automation code tobe maintained under respective ‘project node’ of the TFS being used for the projects.

Recommended Source Control Path:

<Project-Node>/Main/Test/Test Automation

Refer to Appendix: TFS Check-In Policies and Best Practices for details.

☐

Open existing test project and using it as Template

Launch Visual Studio, Browse and open existing ‘CodedUISampleFramework’ template and add it to Source control. You can change the properties as per your needs.

Please find below a short description on the folder structure of the template project which will be used for case study:

1. Config: All your application configuration details and playback settings.

etc will go here

Example:

Also refer to: http://msdn.microsoft.com/en-us/library/a65txexh(v=VS.100).aspx

2. Library: All your libraries- application level, data access, exceptions, utilities will go here.

Please refer to Step 3

3. Resources: All your keys / strings that’s need to globally defined can be added here rather than hardcoding their values in code.

Example: The resource file key “SearchTimeOut” can be used to set playback settings as per your project needs without hardcoding it in your code.

4. Scripts: Your actual test are stored in these .cs files along with components like Test Initialize, Test Cleanup etc

5. Test Data: These will contain your .CSV or .XML test data files for your data-driven automation test

6. UIMaps: These will contain your test object repository along with any test methods or assertions (if any)

7. Main Class file: One class that acts as a single point interface to all the test methods and contains instantiations of the UI maps

☐

With next step, we are now entering into the most important phase of Automation Lifecycle Development which involves designing your automation framework. It is worth spending time on coming up with a good design rather than doing lot of rework later due to shortsightedness and eagerness to start scripting. I would strongly recommend you spend a good effort in coming with design/framework that suits your project requirement before writing your automation test.

Step 3: Automation Design & Development

Automation design and framework is one of the very important pillars of your automation success and it serves as a guiding light for your team members to follow a standard and consistent automation approach across the team.

In this section we will be walking through the automation framework design for Coded UI projects using our Case Study’s template project.

Figure 5: Proposed 3-Tier Architecture for maintaining automated test that promotes high modularity and maximum reuse with minimum maintenance effort in longer run

Follow the below steps for Automation design guidance:

SL No.

Steps

Description

If you have created project from scratch, add Multiple UI Map files

(In the template project, we already have 2 UIMap files for Case Study)

Managing Object Repositories:

Single UI Map File:

By default only one UIMap File is created per project and it is appropriate for a project of small size

Multiple UI Map File: (We recommend using Multi UIMap files for modularization)

For large application, we should use multiple UI Map. Using multiple UI Map files can provide the following benefits:

- Each map can be associated with a logical subset of the application. This makes changes easier to manage.

- Each tester can work on a section of the application and check in their code without interfering with other testers working on other sections of the application.

- Additions to the application UI can be scaled incrementally with minimal effect on tests for other parts of the UI

You can add new UIMap files by selecting Add new Items option and then selecting CodedUI Test Map type and renaming it as per your needs.

The system will launch Coded UI Test Builder which can be used to add object properties to your UIMap file. Press Generate Code button (Alt+G).

For Case Study, you will find the UIMap files as shown below:

Open UIMap .CS files and add User friendly names to constructor:

Go to your Main Class file (Bing.cs) in our Case Study and instantiate each Multi UIMap file

Please refer to Appendix: Managing objects using UITestEditor for more details

Add/Create Libraries to your project

(For Case Study, We have already created libraries for Bing)

Managing Libraries:

Application Functional Libraries using Multiple UI Map files

Ensure common functionalities identified above in planning phase are automated as reusable methods which are then invokes from your Coded UI Test methods

Create functional libraries as per functional modules for your application.

Common Library / Helper Classes:

a) Ensure common utilities like SQL connection, Excel/ XML, String/Date handling etc. are referenced from common library repository in your automated solution (instead of creating them from scratch)

All common libraries need to be browsed and added to automation solution as “References”

b) Ensure that common helper classes which are technology specific like WPF, WCR, SharePoint, Silverlight, .Net etc. are referenced in your solution (if any).

Refer to Appendix: Managing Common Libraries in TFS

For our case study, we have created/referenced following libaries based on the Shared Steps and scenarios identified from ‘automatable test cases’ selected in Step 1 (Automation Planning)

Example of Application specific library methods

Example of Common/Utilies based methods:

Add your CodedUI scripts

Test Methods

Automate your ‘automatable test cases’ as CodedUI Test Methods using application libraries, utilities and UI Maps defined above

a) Set the test context which provides information about and functionality for the current test run.

b) Test Initialize: To perform steps before running each test like instantiating UIMap files, setting TestContext for logging etc.

c) Add your Test Methods

As you can notice above that we have highlighted certain things which are not recommended like using static playback interval or hardcoding test data. We will see in next section how to handle these.

d) Test Cleanup: To run code after running each steps

Adding Checkpoints

Assertions

For validation in Coded UI Testing, you need to use “Assertions”

- Creating assertions using CUIT (Coded UI Test Recorder): You can create assertions without coding by using out-of-the-box feature provided with Coded UI

Choose the property you want to validate and Click “Add Assertion”.

- Creating assertions by hand coding: You can create assertions by writing you own code and capturing properties against expected values to validate at the run time.

Verify.WaitForControlPropertyEqual(BING.SearchResultsUI.searchText, "Text", "Bill Gates", 15000);

e.g. We have a method in our Utility “Verify” that waits for control property to be equal to some text value for certain period and returns PASS if found otherwise FAIL

Make your scripts data-driven

Data-Driven Test

Ensure every variable is parameterized by passing the test data to each test using data-driven capabilities in VS 2010.

Right-click on the Coded UI test and select Properties, to view the properties of the test

There are three options for creating a data source:

Database -- which allows you to connect to a database or Microsoft Excel spreadsheet

CSV File -- which allows you to use a comma-delimited file

XML File -- which allows you to use an XML file

When you add a data source A Data Source attribute is added to the Coded UI Test.

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\data.csv", "data#csv", DataAccessMethod.Sequential), DeploymentItem("data.csv"), TestMethod]

For our Case study, let’s look at one of our data-driven test:

a) We have a method in our data access library (Utilies) to read the data from data sources using the test context

b) Another method as part of Application library returns the data from data source and inputs the same to application using the above data access library method

c) Test data source which is stored in .CSV format

d) Our Data-driven test method calls the above method as part of our functional

Now I suppose you have your automation tests candidates designed and developed. After you write and debug your scripts its important to run them in a batch or as part of a suite. There are various way to do that and factors to keep in mind as we will see in next section.

Step 4: Automation Execution & Deployment

After automation design and debugging is completed, you can now run and deploy your automation scripts using the below steps:

SL No.	Areas	Description
1.	Configuring Test Settings	Test Settings a) Data and Diagnostic: Enable appropriate diagnostic are selected as shown below (Code Coverage, Event Log, Test Impact etc.) b) Define other settings like Roles, Test Timeouts, Setup and Clean scripts
2.	Deploy your test to multiple environments	Deployment to different environments: We want to run the same test across multiple environments to ensure everything is still working correctly. Development-> Test -> UAT -> STAGE Example: We want to run these tests after each deploy to each environment. Each environment has as a different URL, username and password. public UIMap() { // Bind the parameters using Environment Variables. string testServer = Environment.GetEnvironmentVariable("TestServer"); string testUserName = Environment.GetEnvironmentVariable("TestUserName"); string testUserPassword = Environment.GetEnvironmentVariable("TestUserPassword"); } Also refer: http://blogs.msdn.com/b/gautamg/archive/2010/02/23/how-to-get-same-test-running-under-different-environments.aspx
3.	Command Line Execution (MSTest.Exe)	*Grouping & Classifying Test Methods using various parameters* It lets you run groups of tests based on their assigned categories without the requirement to maintain test lists. Go to Test View and you can filter and view your tests by setting these parameters MSTest.EXE Options: MSTest.exe is the command-line command that is used to run tests. This command has several options that you can use to customize your test run. You can specify these options in any order on the MSTest.exe command line. Example: mstest /testcontainer:"C:\TestProject2\generictest1.generic" /test:testproject32\generic Refer to http://msdn.microsoft.com/en-us/library/ms182489.aspx

With this, we come to the end of our case study where we walked you through the complete automation development life cycle that involved Automation Planning, Automation Design and Development, and finally Automation execution and Deployment.

Automation scripts are just ‘Code’ at the end of the day and we can’t deny the importance of having high quality of code and best practices adopted for making our automation tests not only readable and easy to troubleshoot but also for long term maintainability.

Coding standards and best practices

We recommend following the below checklist and recommendations for your automation code. It is the responsibility of the entire team to follow these and imbibing a culture of peer and lead review of the automation code on a regular basis.

Checklist

SL No.	Topic	Description
1.	Object Synchronization over static wait/delay	These synchronization points will make the scripts more resilient and robust. Wait for Control options: Ensure that you insert synchronization points by waiting for the object to be ready/exist/enabled etc. before trying to perform any critical action. Example: WaitForControlReady(),WaitForControlExist() etc *Static* *Delay: Ensure that you use WaitForControl options as much as possible but in some unavoidable cases where you must have to hard wait/sleep use* Playback.Wait() API instead of Thread.Sleep() API.
2.	Exceptions and Error Handling	Ensure that you use the exception handling feature in all your scripts that C# provides i.e. try, catch and finally to do exception handling. The try encloses the statements that might throw an exception whereas catch handles an exception if one exists. The finally can be used for doing any cleanup process. The general form try-catch-finally in C# is shown below try { // Statement which can cause an exception. } catch(Type x) { // Statements for handling the exception } finally { //Any cleanup code }
3.	Logging	Sample Logging Code: 1: [TestMethod] 2: public void CodedUITestMethod1() 3: { 4: Console.WriteLine("Console.WriteLine()"); 5: Console.Error.WriteLine("Console.Error.WriteLine()"); 6: TestContext.WriteLine("TestContext.WriteLine()"); 7: Trace.WriteLine("Trace.WriteLine()"); 8: Debug.WriteLine("Debug.WriteLine()"); 9: } If you check the test result window, you should see output like – Each of these ways have their own advantages and disadvantages - TestContext.WriteLine: Advantage - Shows up property in the separate section in test result. It is meant for this purpose only. Disadvantage - You will have to pass TestContext around from you main test class to other classes (like UIMap class in Coded UI Test) to use it. Recommendation – Prefer this over other methods for tracing in test code. Trace.WriteLine: Advantage – Shows up in VS Output window too during debugging. Disadvantage - The message here could be lost among other trace messages from other components (or product code). Recommendation – Use it in your product code but avoid in test code. That is good way to separate out messages from product code vs test code. You can use this though in certain scenarios where either TestContext is not there or you want to determine the sequence of operation between test & product code Debug.WriteLine: Same as Trace.WriteLine except that this shows up only for Debug build and is no-op in Retail build. Console.WriteLine and Console.Error.WriteLine: These, though work, are tricky. The test harness redirects the Standard Output and Standard Error to capture the output\error from your product code and not for tracing. So, the recommendation is to avoid using these. In short, use TestContext.WriteLine wherever possible.
4. 1.	Naming Conventions	When you create a method, use a meaningful method name instead of the default name. A meaningful name helps identify the purpose of the method. In the application under test, use meaningful names when you create the UI controls. This gives more meaning and usability to the automatically generated control names.
5.	Comments	Have extensive and meaningful comments. Remember that code written by you will be around for a long time and has to be understood, extended and debugged by others who do not have the entire context you had when the code was originally developed. Unless there are good comments, it is hard for even the owner to understand the code few weeks or months later.
6.	Startup & Cleanup Scripts	Using Code: Create a test in your test project. In the test class, create methods with the [TestInitialize] and [TestCleanup] attributes. They will be run before/after each test method. Or, if you want to run before/after all test in that class, create static methods with [ClassInitialize] and [ClassCleanup] attributes. Lastly, to run before/after all tests in the assembly, create static methods with [AssemblyInitialize] and [AssemblyCleanup] attributes.

We are also providing a compilation of best practices collected from various forums which can be used as a reminder when writing coded UI test.

Do’s and Don’ts of UI Automation

Do(s)

o Each recorded method should act on a single page, form, or dialog box. Create a new test method for each new page, form or dialog box.

o Use the Coded UI Test Builder whenever possible.

o Explicitly set focus to the window on which the test Case is expected to input data.

o When possible, limit each recorded method to fewer than 10 actions. This modular approach makes it easier to replace a method if the UI changes.

o Create a separate UIMap file for each module in your application under test. For more information, see Testing a Large Application with Multiple UI Maps.

o If you are creating assertions by coding with the API, create a method for each assertion in the part of the UIMap class that is in the UIMap.cs file. Call this method from your test method to execute the assertion.

o Capture screen shots for failures. Helps in debugging.

o Leave the UI in a known state after a test Case is done executing. E.g.: Close the wizard even if the test Case is to test the first screen of a wizard by cancelling out of the wizard after the first screen.

o Validate that the object being created using the UI with the one being stored in the backend system

o Log before you do something, not after.

o In code reviews for test cases, or new framework automation support review the log messages, is it clear what the test did from just the log?

o Log the origin of messages. If the framework logs, say that it logged, if the test case logged, say that.

o Have two assertion modes that are configurable: 1. Fail instantly. 2. Delay assertion of failure until the test has completed. Steps might be mutually exclusive, you’re losing coverage.

o Log non-fatal exceptions so as not to abort a test, bubble fatal exceptions up to the test and let the exception assert failure.

o Log every valid UI operation being performed in the test Case.

o Use resource files for all data required to be entered in the UI.

o Use Refactoring: Refactoring implies removing a block of code that is being used repeatedly throughout the program into a single function. This helps in reuse of code and makes the code cleaner, and improves readability.

o Use Constructor of classes for meaningful things: In any class avoid unnecessary overloading of constructors. Minimize the amount of work done in a constructor. Constructors should mainly be used for initialization of members. It is always good practice to have accessory methods (get/set) to expose member variables and also initialize them.

o Using string. Compare () instead of "==" or "Equals ()": It is advisable use string.compare() as in that case we don't have to worry about NullPointerException, unlike using Equals() -- you have to make sure that the object that calls Equals() is not null.

o Using List<class name> instead of Arrays: Arrays are not type safe, so it is advisable to use the inherent .Net 2.0 collection objects like IList<T>, ICollections<T>. They provide additional type-checking during compile time.

o Use “this.”: Whenever you use a member variable of class, it is a good practice to put "this" in front of the variable. This improves readability and avoids conflicts when the parameter name and the member variable names are same.

o Using .NET attributes for test cases: Attributes provide a mechanism to enable additional data to the Test Code during runtime and it also helps in keeping information associated with the Test Case in the code itself.

Don’t(s):

o Do not modify the UIMap.designer.cs file directly. If you do this, the changes to the file will be overwritten.

o Do not log multiple times.

o Do not sleep after every UI operation.

o Do not give unusually long timeouts for every major operation.

o Fail immediately after an unexpected screen.

o Do not re-launch the UI application for every test Case as it is a time consuming process.

o Test cases should never have hard-coded strings. This includes control ids, menu paths, etc. Even control IDs and menu paths change over time.

o Do not use sleep

o Avoid too many parameters in a Method/Function. It is best practice to avoid passing too many parameters to a function or method. This indicates lack of abstraction. In this case encapsulating the parameters in a separate class or struct may help.

o Avoid Lengthy Functions: If a function is longer than a page, it is a good practice to break it up into smaller functions. However, if the code has a switch statement or straight line code with no loop back, then it is still OK, but in almost all other cases, the code can be simplified by splitting it up into multiple functions.

o Avoid Hard-Coded strings: Avoid the usage of hard coded strings in the code. Instead use the constants utility to store them.

Appendix

i. Getting Started with VS 2010 Coded UI

Getting Started
Online Help	Testing the User Interface with Automated UI Tests
Forum	Visual Studio UI Automation Testing (includes CodedUI)
Product Overview	Coded UI Test
Webcasts & Videos	Functional UI Testing Visual Studio Team System 2010: Manual Test To Coded UI Test
Guidance and References	Best Practices for Coded UI Tests
Coded UI Test Bloggers	Anutthara Bharadwaj Gautam Goenka Balachander G. Subramaniam Rubel Singhal Rituparna Paul Team Blog
What’s new in Beta2	What’s new for testers doing UI Automation

ii. Visual Studio 2010 Installation

This step is required to ensure that environment is setup and configured to get started with automation,

- Installation: Have you installed VS 2010 Ultimate?

Go to Microsoft.com Download page

- Installation: Have you installed VS 2010 Feature Pack 2

Refer to http://msdn.microsoft.com/en-us/vstudio/ff655021.aspx to find out new features in FP2

- Installation: Have you connected to you Source control (TFS node)?

Go to VS 2010 Ultimate -> Team -> Connect to Team Foundation Server

Add your TFS server name, Choose Project Collection, Select your team project and say Connect

iii. TFS Check-in Policies & Best Practices

- Ensure that policies enforced by project administrator are met

Tip: For example “comment” must be entered every time checking the code

http://msdn.microsoft.com/en-gb/library/ms181460.aspx

Apart from that, following are available by default

o Builds: Requires that the last build was successful before a check-in.

o Code Analysis: Requires that code analysis is run before check-in.

o Testing Policy: Requires that certain tests are completed before check-in.

o Work Items: Requires that one or more work items be associated with the check-in.

http://msdn.microsoft.com/en-gb/library/ms181411.aspx

http://msdn.microsoft.com/en-gb/library/ms245460(v=VS.100).aspx

- Ensure that any custom policies enforced by administrator are met.

- As a best practice, check-in the files at the end of the day

- Code Branching and Merging

One of the most common branching strategies is to align branches with product releases. A branch holds all the software development assets for a single release. Occasionally, you must merge updates from one release to another, but they usually never merge. You discontinue a branch when you discontinue its release.

The mechanics of branching in TFS are simplified to a single right-click à Branching and Merging | Branch command.

Figure 6: Courtesy: ALM Rangers Codeplex Branching guidance

If you have more than one work (DEVELOPMENT) branch, forward integration to all work branches should occur as soon as any branch integrates into the MAIN branch. Because the MAIN branch is kept stable, forward integration is safe..

Refer to http://msdn.microsoft.com/en-us/library/ee782536.aspx and http://tfsbranchingguideiii.codeplex.com/ for detailed guidance from ALM Rangers team

- Ensure Reverse Integration and Forward Integration process when multiple teams (Onsite Offshore are working together on same engagement)

FORWARD INTEGRATE (FI) – merges from parent to child branches.

REVERSE INTEGRATE (RI) – merges from child to parent branches

Refer to http://branchingguidance.codeplex.com/wikipage?title=Code%20Promotion%20Best%20Practices

- Gated Check-ins: Ensure the solution is built before the automation code check-in

By using a gated check-in for the MAIN branch, you help make the reverse integration much easier because quality gates help avoid conflicts or errors in the MAIN branch. For more information, see http://msdn.microsoft.com/en-us/library/dd794776.aspx and http://blogs.msdn.com/b/patcarna/archive/2009/06/29/an-introduction-to-gated-check-in.aspx

http://intovsts.net/2010/04/18/the-gated-check-in-build-in-tfs2010/

When this trigger is selected for a build definition, any check-in that is made to a file that is mapped in the associated workspace will trigger a verification build. If the build succeeds, the changes will be submitted to the repository. If the build fails, the changes are not allowed to be submitted and must be fixed and resubmitted.

iv. Coded UI Frameworks

Details of each UI Framework Supported - Differences, Known Issues and Workarounds
Web	IE Plugin Part 1 IE Plugin Part 2 UI Test Framework Firefox support
WinForms	WinForms Plugin Part 1 WinForms Plugin Part 2
WPF	WPF Plugin Part 1 WPF Plugin Part 2
SharePoint	UI Test Framework Sharepoint support
MFC/Win32	UI Test Framework MFC support

v. Managing common libraries in TFS

All common code and libraries to be maintained at a centralized node which wil be accessible (read-only) to all the projects/apps to be included for reusability.

Note: Only the automation group member will have the add/edit permission on this node to control the change.

Common TFS Project & Node: < placeholder >

Library Path: Automation/Libraries

vi. Data driven testing

Use below link if you need more information about Data driven testing using Coded UI and access/store data in MS EXCEL from Coded UI

Data Driven Testing by Mathew

http://msdn.microsoft.com/en-us/library/ee624082(VS.100).aspx

http://msdn.microsoft.com/en-us/magazine/cc163536.aspx

vii. Managing object repository using UI Test Editor

This is a new feature launched in FP2 which lets you better manage your object repository.

The Coded UI Test Editor toolbar offers the following commands:

1. Find helps to locate UI actions and controls.

2. Delete removes unwanted UI actions.

3. Rename changes the names for test methods and controls.

4. Properties Opens the Properties Window for selected item.

5. Split into a new method lets you modularize the UI actions.

6. Move Code adds custom code to your test methods.

7. Insert Delay Before adds a pause prior to a UI action specified in milliseconds.

8. Locate the UI Control identifies the location of the control in the UI of application under test.

9. Locate All helps to verify control property and significant changes to the application’s controls.

Refer to: http://msdn.microsoft.com/en-us/library/gg269469.aspx

viii. Extending Coded UI

VS 2010 Coded UI support extensibility for technologies which are not supported out-of-the-box like Excel or custom controls.

Refer this example for WPF calendar support

http://blogs.msdn.com/b/tapas_sahoos_blog/archive/2010/12/10/adding-new-control-support-in-a-coded-ui-test-plugin.aspx

ix. Support & Help

- Discussion forum (External)

http://social.msdn.microsoft.com/Forums/en/vsautotest/threads

Intelligent selection of test data by applying learning from Marketing Research

2011-12-10T23:34:00.001+05:30

1. Abstract

2. Prospects Hot Candidates

3. Proposed Solution

4. Process: Intelligent Sampling Process Flow & Engine

5. Sampling Algorithms selected for Database Sampling

6. Cost/Benefit Analysis for the selected algorithms

7. Extended Features

8. Benefits

9. References

Abstract
When the data volume is not too huge, the entire data of the population can be validated as a part of the test phase (Unit testing/ Integration Testing/ System testing). Such a total study gives an excellent confidence on the quality and coverage which can be very expensive method, because validating all the data involves huge hardware cost, resource dependency and the time required for the completion of the activity. Remember, too, that the objectives of a project do not always require an absolutely exact account of the entire population.
Sampling is one approach which can be adopted when the data is voluminous. Note that sampling does not mean that you were not equally interested in all the items in the population. On the contrary, you would like to study all of them, but you pick the sample for practical reasons. Perhaps you have a population of millions of objects and it is impossible to reach even a major part of them. Also in those cases (with populations of, say, up to 10,000) where you might choose to study every object, the sampling study may be a prudent choice, because it saves your time and you can then use the time you save to study the sampled items more carefully.

Few months back, we were in the middle of testing our application, which processes more than 1.50 TB of data and takes approx 36 hours of time when we received this great news that Microsoft was voted as the Best Employer by BT TNS Mercer’s Best Companies to work for in India & it was no doubt a moment of pride but i wasn’t interviewed and none of my friends were either but it struck us how they were still able to echo our feelings so well (for most of us i spoke to were not surveyed)
After thinking it over for a couple of days we realized that this very same concept is very much applicable to our IT solution as well. Why do we have to process 1.5 GB of data and wait for 36 hours to run or why should i make up the data by generating it randomly and taking the risk of compromising on the coverage when i have access to customer/ production equivalent data (non PII)?

The Big Question:
“Why didn’t we do intelligent sampling yet?”

The Big Question:
“Choosing Sample Test Data from Population/ Customer data is not a challenge but how do you ensure that sample is a GOOD sample”
I agree, “GOOD” is a quite subjective term. Let me rephrase
“How do you ensure that the sample chosen is a GOOD representative of your entire data population?”
Prospects Hot Candidates:
Bigger the Size of font in the image below, more is the importance & applicability.
Proposed Solution
Best Sample selection based on your requirement is the true objective. There are multiple solutions in silo to cater to specific scenarios.
What better place to have this solution than in Visual Studio DbPro
Process: Intelligent Sampling Process Flow & Engine
1. Selection Criteria
  i) Population Selection
  The data set which needs to be sampled should be chosen first. This can be chosen based on the parameters of
  1. Size – the dataset for sampling should be of reasonable size (>500 MB). This is to make the process of data sampling worth the effort
  2. Complexity – If the dataset to be sampled is of lesser complexity, data generation approaches may be more suitable. It is for complex data set which involves some business rules plus data integrity to be maintained where subset data selection would help.
  3. Test cases to be run – Subset data selection is for a scenario where number of test cases to be run is high and considerable amount for data regression is required
  4. Variety in data – The dataset to be sampled contains a variety in data. For Eg. Databases of an enterprise system would be ideal. On the other hand one which contains only Taxonomy related data or domain values may not be suitable for sampling
  ii) Business Rules / Requirements or user input / Sampling Frame
  Whenever a sample data is selected from an existing database, one needs to ensure that the underlying business rules are intact. In order to cater to negative test case scenarios, some sample breaking the rules is also fine. However on the whole, business rules have to be maintained.
  Most datasets where sampling is required also have some ‘business rules’ attached to them. For Eg. Expense related calculations may need checks of exchange rates; billing related calculations may need checks of status and so on. If there are business rules which are critical to the transformation of the system to be tested, these can be input by the user, to ensure that the sampling happens accordingly.
  iii) Get Expected Cost Associated From Sample from the User.
  The user should also be able to select an expected associated cost to the sample which would be created. This cost would be based on the size/time taken for the sample generation and coverage based on data variety included in the sample. The expected associated cost entered by the user would be used to prioritize on the sample algorithms to be run, which are known to provide the expected cost. Running these algorithms with the expected cost as an input would help reach the required end result and propose different data sampling options to the user.
2. Sampling Method
  i) Data Profiling Required - Domain/Transactional Data
  In order to go for effective sampling, we also recommend categorizing the existing dataset into tables containing ‘Master’^* data and those containing transactionaldata.This could be an outcome of the Data Integrity relationship detected, or a step within the sampling engine while applying the algorithm. This categorization helps in selecting required amount of samples from each of these tables as well as selecting an appropriate sampling method. For instance, if a table contains date/calendar information, thus making it fall under a master category, would not need to be sampled or reduced; where as one containing daily transactions of an account can be a candidate for sampling. Detection of data set impacting the coverage is critical before coming up with the sampling plan.This would be done based on
  1. Profiling the data and checking for columns which contain distinctively unique values which are sure to impact the test case/data coverage.
  2. Get input from the user based on his prior knowledge.
  Once this input is provided, save it to use during generating the sampling plans.
  ii) Generate Plans with associated cost
  Generate the different plans possible for data selection and provide the‘cost’ associated with it based on the rules applied prior to sampling This cost would be calculated based on following factors:
  - Expected size of the sample
  - Expected data coverage of the sample.
  The plans along with the indicator of cost associated with each of the plan would be provided to the user. Thereafter, the user could either choose the best recommended plan, or go for another based on his discretion.
  
  iii) User to choose plan or go for regeneration
  If the user is not satisfied with the plans and their cost, he can ask for a regeneration of plan which would involve trying out a different set of inputs or providing different columns for driving data coverage.
3. Sampling Engine
  The sampling engine is expected to perform all of the above steps after scanning through the target population, as well as consuming specific inputs provided by the user. Next Steps are:
  i. Implementing the sampling plan
  ii. Reviewing the sampling process
  The sampling engine is the core of the data selection process. Stages involved within the sampling engine:
  i) Building the Sampling Rules
  - The sampling rules would be built based on the Data Integrity Checks run on the database as well as after profiling the data for any exceptions to business rules and categorizing data sets into master or transactional.
  - This could also include setting any delta rule condition (or in other words reducing the scope of data to be sampled
    When a Data Volume is sampled already, the next time the Sampling Engine Runs on the same dataset, the sample is generated for the new added data rows only. This would save time, effort and money and would give more ROI to the proposed feature inclusion in VSTF DB Pro.
  - These sampling rules would form the underlying conditions which need to be satisfied by the selected database.
  - This ensures that after selecting a sample from the source system, when we run our ETL process on it, the ETL should not break due to incorrect sample set.
  ii) Apply algorithm
  - The algorithms chosen for application within the sampling engine are covered in next section (Section 7). Each of the above algorithms would be applied to the source system data and cost for the same would be calculated
  iii) Calculate Sampling Cost :
  We define sampling cost based on two parameters
  - Sample Size
  - Data coverage
  The Sampling Cost would be calculated based on the Size (This could be Row Count of each table available in the sample or the actual Size in MB/GB). This percentage reduction in Size would act as one parameter in Cost. Data Coverage in the sample would be detected differently by differently algorithms. If the algorithm is probabilistic, frequency of data would be a key in determining coverage, if statistical algorithm is used, distinct values in columns would form key criteria and in T-Way, the requirements would be primary. This would be covered in detail in the subsequent section.
  iv) Check for correctness and update the Sampling Rules
  Identify if the sample satisfies the basic rules and if the user wishes he/she could opt for re-sampling, and this time fine-tune the rules as well as inputs provided.
  v) Log Errors
  Any errors encountered during the process of profiling/sampling method generation would have to be logged and reported.
  The diagram below shows an end-to-end mock of the screens that could show up during this process of sampling method generation and selection and the various inputs that need to be provided by the user and the actions that take place.
Sampling Algorithms selected for Database Sampling
Within any of the types of frame identified above, a variety of sampling methods can be employed, individually or in combination. Factors commonly influencing the choice between these designs include:
- Nature and quality of the frame
- Availability of auxiliary information about units on the frame
- Accuracy requirements, and the need to measure accuracy
- Whether detailed analysis of the sample is expected
- Cost/operational concerns
a) Probabilistic Sampling Methods - Multi-Stage Sampling
In most real applied application, we would use sampling methods that are considerably more complex than simple methods like (random sampling, cluster, stratified, systematic etc). The most important principle here is that we can combine the simple methods in a variety of useful ways that help us address our sampling needs in the most efficient and effective manner possible. When we combine different probabilistic sampling methods, we call this Multi-Stage sampling. By combining different sampling methods we are able to achieve a rich variety of probabilistic sampling methods that can be used in a wide range of social research contexts. This leads to less Sampling Error by relying on multiple randomizations by choosing random samples of preceding random samples.
3-Stage Multi-Stage Sample Process
1. Select the clusters from the population
2. Select all strata from each clusters
3. In next stages, select additional random sample from selected strata units and so on
4. Finally all ultimate samples selected are chosen
b) Statistical Sampling – Purposive Sampling
The difference between non-probability and probability sampling is that non-probability sampling does not involve random selection and probability sampling does. Most sampling methods are purposive in nature because we usually approach the sampling problem with a specific plan in mind. In purposive sampling, we sample with a purpose in mind. We usually would have one or more specific predefined groups we are seeking. Useful when you need to reach a targeted sample quickly and where sampling for proportionality is not the primary concern.
For ex. have you ever run into guys in a mall asking people if they could interview them? Most likely they are conducting a purposive sample. They might be looking for Caucasian females between 30-40 years old as that’s their need and they do that by estimating
One caveat is that you are also likely to overweight subgroups in your population that are more readily accessible.
c) T -Way Sampling Method
T-way Testing is a type of interaction testing which requires that for each t-way combination of input parameters of a system, every combination of valid values of these t parameters must be covered by at least one test case. It involves selecting test samples in such a manner that it covers all the t-wise interactions between the parameters and the possible values of a given system.
For example, Let us consider Employee Entity which contains attributes like Employee Discipline and Employee Role. Also, consider the Country Entity has attributes like CountryName and IsActive. Correspondingly, the possible values Employee Entity and Country Entity are as shown below.

The total Number of Combinations possible for values between Employee and Country = 3*2*3*2 = 36 combinations. We can also say, that our code will work if I have 1 combination, but neither of them strikes an optimal balance between Quality Assurance and Delivery!
Applying the t-way testing methodology and setting t=2, we get the Dataset which has only 9 combinations

Similarly, when t=3, the number of combinations increase but the data coverage also increases and hence the sample size increases, and it strikes an optimal balance between the size of the dataset and the code coverage. A sample run would generate 14 records for the current example.
We propose, set t=3 to find out the optimal data combination both in terms of coverage and time.
Cost/Benefit Analysis for the selected algorithms
This section will be used internally by the system to determine best algorithm for a selected population by doing a cost/benefit of each algorithm statistically
a) Cost Benefit for Statistical Algorithms

b) Cost Benefit for Statistical Algorithms

c) Cost Benefit for T-Way Algorithm
Extended Features:
7.1 Avoid Re-sampling of the Sampled Data to increase data coverage
Sampling schemes should be generally defaulted to Without Replacement (‘WOR’). No element should be re-selected more than once in the subsequent sampling attempts if it’s already sampled in one of the past attempts (unless RESAMPLI NG = 1 is explicitly configured by the user). By Default, Re-Sampling Flag is set to 0 to ensure that already sampled data doesn’t always get picked in every sample run and rest of the data should get a fair chance of getting selected.
For example, if we catch fish, measure them, and immediately return them to the water before continuing with the sample, this is a WR design, because we might end up catching and measuring the same fish more than once. However, if we do not return the fish to the water (e.g. if we eat the fish), this becomes a WOR design.
Marking already sampled data helps ensuring that the already sampled data is not picked in the future runs unless we run out of data. This basically follows the assumption that the sampling method chosen by the user / system picks the best available sample for the first run but say for the second run we don’t want to pick the already sampled data (selected based on run #) and so on.

7.2 Weighted Sampling
Weights can be provided by the user in the proposed user configuration settings to give more value / importance to a specific column which is more important based on the business need.

More generally, data should usually be weighted if the sample design does not give each individual an equal chance of being selected. In many situations the sample fraction may be varied by stratum and data will have to be weighted to correctly represent the population.
Real World Example, a simple random sample of individuals in the United Kingdom might include some in remote Scottish islands who would be inordinately expensive to sample. A cheaper method would be to use a stratified sample with urban and rural strata. The rural sample could be under-represented in the sample, but weighted up appropriately in the analysis to compensate.
Benefits
The following are direct /indirect benefits to the various stakeholders like Dev, Test, PM’s & Business Community.
- Overall Test Execution Time Reduction
- Higher Requirement & Test Coverage
- Higher Code Coverage
- Effective Unit Testing & Functional Testing -Boundary Values, Equivalence Partitioning
- Reduces Hardware Cost for IT
- High confidence while delivering to Test, UAT & Production
- Increased Domain knowledge & business understanding of the resource while choosing & profiling the sample data
- Earlier detection of potential Data Quality issues by analyzing the data (while profiling) for sampling
References
http://en.wikipedia.org/wiki/Sampling_%28statistics%29
http://doi.acm.org/10.1145/1102351.1102456
http://www.ai.mit.edu/projects/jmlr/papers/volume3/scheffer02a/scheffer02a.pdf
http://www.uiah.fi/projects/metodi/152.htm
http://en.wikipedia.org/wiki/Quantitative_marketing_research
http://en.wikipedia.org/wiki/Expectation-maximization_algorithm
http://www.gabormelli.com/RKB/EM_Algorithm

Authors

Raj Kamal (rajkamal)

Web Services Test Automation Guidance using VS 2010 - Unit Test & Web Test

2011-12-10T21:19:00.001+05:30

All your wait is over. Finally its out http://webservicestesting.codeplex.com/

<<Please refer to Web Services Automation Guidance (VS 2010 Unit Test & Web Test).PDF http://webservicestesting.codeplex.com/releases/view/78473#DownloadId=312126 for detailed documentation>>

The artifact will provide Web Services Automation guidance for Functional & Performance Testing leveraging VS 2010. The document will provide recommendation on using either Unit Test or Web Test for functional testing depending on your project and test objective as will be later covered in the document. Load Test will be used for Performance Testing in both cases.

The primary objective is to propose a viable web services test automation approach & reusable libraries to customers, partners and consultant. It will also be a step in standardizing web services automation using VS 2010 platform across the Industry and filling the gaps by extending our tools as suggested in the talk. The document can also be used as a quick jump start guide for beginners to get started on Web Services Automation Testing.

Have fun guys. We want to hear from you.

Thanks,

Raj, Vipul, Ranjit & Gunjan

Microsoft Case Study: Visual Studio 2010 Automation Testing (Raj Kamal)

2011-03-03T10:39:00.001+05:30

“We wanted a test process that would be faster and more productive. We achieved that by using the automated tools in Visual Studio 2010.”

Raj Kamal
BI Engineering Automation Team Lead, Microsoft IT

http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseStudyID=4000009261

Testability: ‘Black Box’ required for ‘Black Box’ testing

2010-12-30T13:30:00.001+05:30

Can’t see the forest for the trees

Thought

“We don’t have runway left,” said his co-pilot HS Ahluwalia – the last thing recorded in the cockpit. The plane then plunged into a forested gorge and was engulfed in flames. Black box only helped unravel the truth that captain ignored co-pilot's plea to abort landing which led to this unfortunate event.

This was nudging me for quite some time and I was thinking when we testers test all the time (and lot of which is of type ‘black box’) and ironically, we don’t have a ‘black box’ present in most of our IT applications. While testing complex systems, teams end up spending a good amount of time figuring out what’s happening under the hood in order to determine the application’s actual behaviour & this is especially true when a production issue pops up and it takes a lot of time for support & test to not only repro the issue but also to find out what would have happened behind the scenes to quickly find out the root cause so that the fix can be provided (if required) while customers can’t stop biting their nails waiting to hear from IT.

And then while researching on the topic I stumbled upon this:

“Think of it as a flight data recorder, so that any time there's a problem, that 'black box' is there helping us work together and diagnose what's going on," Bill Gates said during a speech at the Windows Hardware Engineering Conference. I understand Bill was saying this from Windows point of view but isn’t this if not more but at least equally applicable to IT apps where we want to reduce the development and support cost? Shouldn’t we have a “black box” too to enable our testers do “black box” testing in lesser time?

This paper will be targeting testability for machines (by ensuring automation tools can validate the implementation of feature with reliability & ease) and humans (customers, users, testers and developers) at system level in the given test context.


	“Testability helps as “Every Test hour saved is a Microsoft hour saved”

Problem

“Situation is very similar to that of a man who doesn’t even know that he is suffering from some serious disease while instead of going for check-up he blames other factors for his weakness and such symptoms “

For “poor testability” in many scenarios acts as a slow poison (& not always comes as a sudden death), it is often hard to detect the same & testers, who might be ignorant of the fact, can attribute it to the wrong problems and will keep trying to address their pain points by working out solutions such as stretching extra hours, hiring more resources, expensive automation, risk based testing, need for better estimation and planning etc. without any real long term success. This vicious cycle keep haunting them release over release.

The point is that lot of teams don’t even realize testability is their pain point and hence it doesn’t get the attention it truly deserves.

And for the testers who do realize testability is their culprit, they know better that it is difficult to fix testability, once the damage is already done (for legacy apps) & its keeps getting procrastinated forever as cleansing code is NOT fancy, fashionable & most importantly it’s not cheap. Even when our development team acts like Good Samaritans & come to tester’s rescue to offer refactoring/redesigning to improve the testability (lot of good stuff has been written by testability gurus like Misko), but interestingly & surprisingly, the test teams & customers might not necessarily always agree for the obvious reasons as it will can be risky to make changes in a functional product & running the risk of introducing any regression issues which can’t be adequately tested due to lack of time & they might settle to live with bad testability.

Testability if not tackled right from design phase of the application then it can be a major problem to handle them in future releases and this paper focuses on improving testability for not only V1 project but also for legacy applications where testability can still be improved by proposing solutions other than design changes.

Scope: Testability @ System Layer (Black Box)

In real world, testing is nothing but attempting to USE the product and sincerely hoping it to work and not being able to determine the behaviour reliably and getting stuck (I call it ‘unintentional discoverability of poor testability’). Now testers are customer’s best friends (or so we want them to be), who get paid to put themselves in end-user’s shoes to test the product from their point of view & hence testability is quite important at this layer because this is how the product is going to be used in real world. The challenge DFT (Design for Test) is facing today is the need to look beyond component level & focus on system requirements, & the life cycle maintenance support. This brings in the need to ensure testability at system level despite of the fact that testability is easier to perform at granular level for controllability and observability.

A products user interface is one of the most challenging areas to increase testability. It’s also the place where our customers will be using the most. It’s important to make your user interface code as testable as possible.

Testability is more important than ever at functional layer because of the correlation that exists between Testability and other crucial factors like usability, accessibility, security and discoverability.

As usual, there's no silver bullet; some approaches may be better, but there's always some limitation that blocks from adopting in all possible contexts. There is lot of great stuff written on the use of DFT, TDD, Mock Objects, Emulation Layers and Test Plug-ins for improving testability but for the scope of this discussion we are not covering those.

Solutions

To improve automated testability, we will talk about few solutions including building a ‘black box’, merging test and development code and intelligent selection of test data

Developing ‘Black box’

‘Black Box’ can be developed using Test Hooks. Test hooks are quite different from Mock Objects, however. Hooks are significant anytime you need to find the details of any process

“Good test hooks are one of those "I'll know it when I see it" things. Basically, if it gives you the data you need when you need it, it’s good”

There can be two kinds of applications where ‘black box’ can be very useful especially for (a):

a) Applications with Thick UI

If you system’s business logic is separated from UI you can directly hit the web services / API to test the logic with the help of custom harness BUT if not then you need a way to tunnel through the UI and other layers to reach the business logic. That’s when testability at system layer (UI) sometime become extremely important for applications having business logic present inside the UI layer as well (though which is not a good practice but for legacy apps it might be true.

b) Applications with Thin UI

A test hook can be useful here even if you are calling the APIs directly; for ex., you can find out the specific intermediate results.

Approaches:

While developing black box to increase the visibility, you need to focus on “Observability” and “Controllability” components of the classic SOCK model from ‘black box’ point of view. Observability here is watching system behaviour and reaction for each input given to the application & Controllability is primarily your ability to control the state and execution path of your application.

Adding (optional) detailed logging using Test Hooks

Turning on/off option for detailed logging will help in observability:

i) Validating business logic when there are lot of intermediate steps involved and the testers want to validate the logic at each step then this can be instrumental in going and tracking it exactly as it happens.

ii) Visualization of application execution flow (behind-the-scene-action) and understanding what’s exactly happening under the hood and when (e.g. timestamps). This will avoid the need to make guesses and investigating various layer one by one to find out the real culprit

iii) This can also be used as a kind of self-documentation of the features where someone new in the team can understand the application data and control flow by just looking at these elaborate logs.

Optional parameters to override default settings

This can improve controllability when let’s say tester want to change the default timeout by overriding it so that the test can be run faster. These additional parameters will empower our tester to control the state of the application which can be used to easily test the application.

Adding “Self-Testable” features

These can be the methods explicitly written and added to the code and they will be publically exposed and documented as features so that testers/support or even users can run these to diagnose the behaviour of basic yet critical functionalities.


	“Think of these as like Hard Disk Failure Test which can be run by the user to troubleshoot the problem without the need to engage the support professionals to do preliminary investigation.”

The hardest part of designing a black box might be whether to leave it in the production build or not. Likewise, allowing your customers to turn on debug logging and send you the results will save time for everyone. Many support teams and advanced business users are perfectly willing to do this, as long as it's under their control and they can see what they're sending you. They can act as self-testable features that will be shipped with the software for easier FD/FI (Fault Detection/Fault Isolation). If you do leave your black box in, treat them as the feature they are.

Providing “Alternate” execution paths

a) It is the ability to perform same operation by using multiple options so that when say main option is not working, it doesn’t become a showstopper and the same can be tested using alternative option. For example: a keyboard shortcut might come in handy when the UI control is not working.

b) Lot of times testing an operation might be tough say from UI whereas same could be easily done from Command line and hence consciously making sure that there are multiple means to achieve the same results go a long way to help testers test easily.

One Code: Merge test automation and development code

“After all code is just a piece of code and it knows no boundary. “

Too many times the silos of development and testing are not intersecting until it's 'too late'. Consider having test automation code and development code as part of the same Visual Studio solution.

I'm a huge advocate of merging test and development code for a variety of reasons.

1. Automation ‘Rat Race’ Challenge: Situation arises when automation test code constantly tries to catch up with the new changes made to the code due to requirement changes / bug fixes, which mostly breaks the automation and causes rework and maintenance effort.

Now with refactoring the changes in the UI can be easily propagated to test solution even before the test script runs, fails and discover it.

2. Whenever the code is changed, the impacted test code can be seen and identified for any potential issues with dependencies created between them. With this the regression can be easily identified whenever the code change is made and testing team can run the impacted tests alone.

3. We get maximal code reuse, our test utilities can use the exact same code to create and modify data, and it promotes ownership between test and development in both directions. This way quality can be pushed upstream when we have a solid automation infrastructure which can be shared by both developer and tester to improve the overall product quality.

4. ‘One Click Automation’ can be achieved where every development build (Daily or Weekly) can seamlessly trigger automation (unit/functional) using VS 2010 features.

Like many things, this is about people working together and if you have a good level of DEV/QA cohesion, it goes A LONG way.

Intelligent Test Data can improve testability as well

One of the important aspects of testability is to achieve complete code path coverage and simulate situations in a cost efficient manner. The monster that would need to be addressed when there are infinite permutations and combinations and one can’t be certain about the testability as exhaustive testing is impractical and random testing is only risky. We wrote a Think Week paper last year to cover this topic in depth “Intelligent Selection of test data by applying learning from Marketing Research There we have proposed strategies to choose the test inputs /data in such a way that it can help you achieve high coverage with minimum testing effort.

Benefits

Overall testing efforts can be reduced when features become easier to test.

The ‘black box’ and ‘merging test and development code’ approach can make our tests simpler and automation much more robust.

‘Black box’ can become supportability features for support professional and advanced users (easier Fault Detection & Fault Isolation)

IT support cost and effort can be reduced by empowering support professionals by providing them ‘black box’ which can be used to diagnose issues with lesser dependency on IT Engineering team (Ticket Reduction)

Conclusion

In this paper I have made an attempt to highlight testability as a major challenge which needs to be given its due importance to help reduce overall testing cost/effort and also to improve customer UX (user experience) by making testing/using the product simpler.

Implementing changes suggested in the paper would no doubt take some substantial effort (varying from project to project depending upon the complexity of the apps) but for legacy apps, it can be done in an incremental manner whereas for new apps it can be built into the code from day 1 as a best practice.

References

http://news.cnet.com/8301-13860_3-10052412-56.html#ixzz10S1RPf00
http://news.cnet.com/Microsoft%20to%20add%20black%20box%20to%20Windows/2100-1016_3-5684051.html#ixzz10nq2SAIl

http://blogs.msdn.com/b/micahel/archive/2004/08/04/207997.aspx

http://blogs.msdn.com/b/dustin_andrews/archive/2008/01/11/driving-testability-from-the-ui-down.aspx

Biography

Raj is a Test Consultant specializing in different types of testing techniques, test automation and testability in different domains like Manufacturing, Healthcare and Higher Education. He holds an APICS certification in Supply Chain Management. Expertise with Rational and Mercury testing tools, he has helped teams develop test automation strategies and architectures for such companies as Cognizant Technology Solutions and Oracle Corporation. He also provides training in automated testing architectures and design. He is QAI (CSTE) & ISTQB Certified. He has a master's degree in Computer Applications. He is currently working at Microsoft, India, Business Intelligence COE. He has earlier represented Microsoft and Oracle at International test conferences as a Speaker.

His passion @ http://www.itest.co.nr

Testing a product….Just like movies

2010-08-01T13:41:00.001+05:30

I am not a movie buff but i have no shame in admitting that movies do inspire me. I learn a lot from movies. Most of the movies (good or bad) carry some lessons which are so very applicable to our personal as well as professional life. Lately I was thinking that there is a lot common between movies and software testing assignments. I am a tester by profession and hence this post is dedicated to the analogies I have tried to draw between movies and software testing. The thought process was that “can we learning something from other professions?”. I invite every reader (tester and non-tester) to add points to the list below and share their thoughts.

Here are the 10 things which I feel are common between movies and our profession (software testing):

#1. Movies typically have good and evil characters (Heroes and Villains; Friends & Enemies)

Our dear tester are heroes most of the time but they can also be even villains at times. They are the ones who make sure that the user’s expectations are met. Testers many times have an opinion that all the forces in the world are conspiring against them to stop them from delivering a quality product. The so called “Quality” is their sole mission and they will destroy whatever comes in the way. For the rest of the team (esp. developers and project managers) they act like villains by playing the role of quality police where they wont let the release the product on time.

#2. Movies always have a plot, to start with there will always be some problems, with lot of twists and in the end somehow everything falls in place

Software testing project too kicks off with a scope (or a plot), here as well things obviously don’t go as planned, unexpected things take place, mistakes are made, then there is panic, and then from nowhere a hero comes to rescue by suggesting solutions/work around and in the end somehow we still deliver the product and celebrate big time

#3. Movies are by-design entertaining. They typically have all the necessary ingredients to keep the audience engaged

So is software testing. Each project put you into situations which makes you laugh (sometime at your own stupid mistakes), it makes you angry when you feel somebody is not helping (or making things worse for you), sometimes it even makes you cry (when despite of putting all the hard work things don't work for you) and for some geeks like me…it makes you fall in love with your work or the technology you are working on :)

#4 Movies are more convincing when they are based on real life stories

Ever noticed that movies makes long lasting impression when you know that they are inspired by some REAL stories. Probably because you then trust them more or may be because you can correlate to characters better.

When someone is testing a product just like a REAL user (by truly simulating their behavior & thinking how they might interact with the product when at work), it is for sure going to be much more trustable, convincing and believable in eyes of all the stakeholders and even to the test team itself. It’s not a bad idea to come up with different user personas of your application users and then to they play them sincerely while testing the product.

#5. Now days movies are released world-wide needs to appeal to global audience to be a real block buster (Remember Avatar ? or Titanic?)

Does Globalization / Localization testing rings any bells? Yes, that is our version of doing a world premiere. I will leave this point here for you to think about it.

#6. Entire movie can’t (or at least shouldn’t be) be shot ONLY at one location

Producers spend unimaginable amount of money in creating sets, shooting outdoors (think of taking all the crew to New Zealand or other exotic destinations), just to produce the effect which is the need of the script and the story. Don't your agree that the effort and the money we spend in test environment is well justified and shouldn’t be compromised? We can test the application in some local test environment (nothing like real environment) which is no doubt cheaper but it would never produce the same results as when it will be actually used in production. We don’t want that to happen. Do we?

Wait !! I know what you must be thinking ? “Sets” in movies are not REAL either. Yes, but they LOOK real. Right? something that is really close to REAL and that’s what Performance testing tools like Load Runner enable us to simulate that REAL behavior.

#7. If movies are stretched too much, audience loses interest & the message the producers wanted to convey is lost (or at least doesn’t remain as impactful as it could have been)

Reminds you of waterfall model days ? A release going on for months and years. In today's agile world even 3 months has started to look like ages. This is an era of faster delivery."Shorter the better” is the mantra. If software testing goes on for too long, testers start to lose focus (excitement starts diminishing, work life begins to look monotonous), your customers go on hibernation mode while you are busy developing and testing and then they wake up all of a sudden when you deliver and then one finds that there is a disconnect and end users have even forgotten that something was coming up and i wont be surprised if users have switched to competitors or worse, comes up with their solution to the very problem you were trying to solve (You software/application is now obsolete)

#8. Script, Story, Lyrics, Screen Play , Direction, Acting and tons of other things makes a movie successful

Movies are great examples of excellent teamwork. A movie won’t work if say story is bad, doesn’t matter how good was say direction or acting. These are all important ingredients of the the recipe.

Now think, can we test the product successfully if requirements itself are invalid or say code is crappy or design isn’t what it should be or lets say if we are testing with bad test data. All these are absolutely mandatory for the product to be successful. After all “Quality is EVERYONE’s business, and not just Tester’s”

#9. No movie is ever perfect (There are critics always waiting to pass the verdict)

Lesson is that whatever you do there will be suggestions/feedback and criticism and you cant make everyone happy. Having said that, critics are really important to look back and identifying the area where we could have still done better. So don’t lose your heart if despite of putting all the hard work you are still asked to improve (‘Era of continuous improvement :P)

#10. All successful movies have sequels :)

So true in our profession too.

A software testing stage act (funny) - written and directed by yours truly :)

2010-07-26T23:59:00.001+05:30

I would be lying if I say I always wanted to do this and as far as i can remember, i was never known for participating in any extra-curricular activities, leave alone plays or stage acts. Lately it somewhere started to grow on me as I started watching (and appreciating) movies and more importantly when my sister (Savita , the cinematographer) instilled confidence in me. Thankfully i got my first break , when i was,asked ( let’s rather say forced ) to write and direct a play for Microsoft’s Tester’s Day. Like many others, i thought of Googling (oops …Binging) for it but i didn’t find one good play on software testing (good for me ) and i finally gathered the courage to write one. I am sharing the script with you all if you want to play it again :)
Plot:
A bunch of unusual testers unintentionally making the life of developers really terrible.

Actors:

<Testing Team>
Quality Advocate :: Ranchoddas

He is awesome force in testing that will stand for nothing else but quality. Ranchoddas is often called the quality police and are at times known to be too quality focused. He will enforce quality all over, from the build and release to the documentation. Even if no-one asked for his input. He will rip any spec to pieces; enforce quality in every meeting they attend and settle for nothing but the most ardent quality in every aspect of the software development life cycle.
One liner: He lives, breathes and sleeps quality.
Ranchoddas is respected and required on every project but he often over-looks commercial aspects meaning he can be known as troublesome, unbending and inflexible. He often has a reputation of sticking this his nose in or bullying. He also believes they are running the project.
Drama queen :: Sambhavna
Drama Queen over-reacts with each bug found. Whether it is a spelling mistake or a catastrophic failure The Drama Queen always reacts the same.

The Drama Queen can be seen hurtling towards the programming team, returning with at least 5 of them whilst wildly pointing at the screen and shouting 'Look, Look, it's broken'. In reality it's often 'just a bug' that is fixed immediately with little or no fuss. The Drama Queen really gets programmers backs up as they are never sure whether their code is really that bad or whether they've just overlooked something. Cry Wolf?

Socialiser :: Ritesh

One Liner: Let's get a drink and chat about it.
Laughing, chatting, enjoying their work and having fun. The Socialiser is often referred to as The Moral Officer due to his key role as team outing organizer and coffee break manager.
The socialiser basically knows everyone, everywhere
Last but not the least is our old Chaturlingam who has a knack of finding unusual bugs. He is most feared by developers. All he has to say in his defence is: “I don't go looking for defects. Defects find me. “

EXPLORER :: Chaturlingam

Chaturlingam who has a knack of finding unusual bugs. He is most feared by developers. All he has to say in his defense is: “I don't go looking for defects. Defects find me. “
He is what's known as a responsive thinker. He set himself a charter, often defined from a test case and they explore the app looking for interesting scenarios and paths reacting to information being given back by the app and in turn altering their test ideas. The defects he find are often show stoppers, are truly inventive and are often so difficult to fix that programmers have exploded in rage. The Explorer can test anything, in any state and at any time. They aren't bound by the constraints of a checklister.They are free. They explore.

<Development team>

ANGRY DEVELOPER:: Rakhi -
She is an aggressive and impatient person by nature. She of course doesn’t like being bugged too much by the test team. Though she doesn’t say it openly she disdains the test team for their lack of technical knowledge and for doing a job which she feels is not worth appreciating.

SEASONED DEVELOPER:: Anil
All my sympathies to Anil who is a senior campaigner who has got tired of arguing with test in his long career and decided to take an easier way out of not getting into confrontation mode. He tries to convince, fails miserably and in the end other party wins and he accepts it whether he was wrong or right.
Anil doesn’t like to go into confrontation modes with testers or for that matter with anyone. He is a composed person who is definitely not witty and he often finds himself giving up in the end by accepting the other’s terms. In short, people feel sympathy for him coz he really can’t fight back even when innocent

Opening Scene:

Developers are looking drowsy, sleepy and tired in the morning (as they were working till 6 in the morning). Testers are fresh and come late to start a new day.

< SCENE #1>: Installation

Chaturlingam to Sambhavna: (Chaturlingam looking excited to start the testing; Sambhavna looking tensed as she is installing and she’s looking anxious)
Hey Sambhavna, What’s taking the installation so long? Is it done yet?
Ranchoddas: (interrupts suddenly as if was just waiting for it)
Are you kidding? I am not going to anyone touch the build till my design inspection bugs are closed and I see the signed off Tech Design Spec doc. In the name of our Test Process Improvement, I can’t compromise on these key deliverables
<in high pitch to developer>
Ranchoddas: <Stands up and looking at Rakhi who is looking very lazy at the moment>
Why Bug tracker stills shows my design bugs are still active and Tech Spec is not updated? You have already wasted our 15 mins this morning
Rakhi: (trying her best to look reasonable)
Ranchoddas, Tech Design Specification is checked-in and complete. Check again.
Ranchoddas: (with sarcasm)
Yes, I can see the Tech Design Spec but that was for the last release and only thing you seem to have updated is the RELEASE VERSION NO. Don’t provoke me to reject the build.
Anil (waking up suddenly & trying to solve the matter; trying to get sympathy):
Guys…We just finished coding only this morning at 6 a.m. so we would have missed it. You start with the installation and give us time till EOD and we will update the tech spec
Ranchoddas: (looking surprised; no pity that devs were working till morning)
Are you suggesting me that you have not done Unit Testing in that case? I want to see Unit Test report now? Mister quality is not only our responsibility.
Rakhi (feeling trapped):
There was a requirement change yesterday night and because of that we didn’t get enough time to code and unit test.
Ranchoddas: (Giving an expression as if he is back stabbed):
OMG!!! So there was a CR nobody bothered to inform the test team why the test team was not in the loop? What is wrong with our PM? What about the impact analysis.? And when will we update our test cases? I need to escalate this right away to senior management. Nobody breaks law in my territory.
Ritesh: cool man. That’s ok. The CR is in my work item & I have updated it in my test case. I think we should start off with our testing
Anil (looking relaxed again. Passes a broad smile to Ritesh)
Ranchoddas, I am resolving your tech spec comments. It won’t happen again.
Ranchoddas (shrugs his shoulders)
Fine.

< PAUSE OF few second. Team is doing their usual stuff>

Drama Queen: <doing the installation; she squeals and runs toward developers>
The build failed. Oh God why today? Deadline is so close. It always happens to me. I don’t know what we are going to do next.
Anil (Looking relaxed after seeing the error and speaks in a soft tone of contempt for testers):
Yes Sambhavna, It will fail coz you don’t have D drive on your machine
Drama Queen:
Mister don’t get technical with me. How on the earth was i supposed to know that?
<Rakhi gives a disgusting look to Sambhavna. Test team looks embarrassed;. Sambhavna doesn’t know what was wrong with that. Anil has no words and everyone gets back to work>

< PAUSE OF 10 second. Team is doing their usual stuff. Devs have gone back to their seats>

Sambhavna: Not again !!!! There is another problem. After I install the build, dialog box says “Installation recommends reboot. Do you want to reboot now?”
Rakhi (casually) What is the problem. Say NO !!!!
Sambhavna: (Looking irritated)
There is only one button available and that says. “YES”
Rakhi : oops…looks like a minor miss
Rakhi gives a smirk, and Sambhavna is puzzled.

<ANNOUNCEMENT: so finally the installation succeeded. There were few bugs here and there but the team is still managing quite okay>

< PAUSE OF 10 second. Team is doing their usual stuff. >

< SCENE #2>: (BVT & Functional Testing)

Ranchoddas:
I can see two bugs where “how found” and “found in environment” is not entered. Can you fix it right away? I am not going to tolerate such process compliance, I tell you!
Ritesh:
Sorry!!!. That has to be me. Leave it to me dude, I will take care of it.

< PAUSE OF 10 second. Team is doing their usual stuff. >

Ritesh:
Guys, as you know our last release had just 5 Severity 1 bugs production, our management is really happy with our performance and has given us budget for an outing. How about a lunch today.?
Ranchoddas:
Did we scope for this time out in our schedule? Are you sure we will not be missing out on our time to test by such outings? I suggest we take an informed decision on this. Send an email with voting option, so that we have this documented somewhere for future reference.
Ritesh:
Ok boss. Whatever u say. Let’s do it fast. I have to check the reservation then.
Rakhi:
Lets go to KFC. Finger licking good.
Anil:
Today is Tuesday. Can we go to a Veg restaurant
Ritesh:
Hmm.. How about Little Italy, Mozarella, Ohris or Malgudi ?
Ranchoddas:
Ohris? All the way to Banjara Hills? *sarcastically*
Do you want this to be a full day activity????
Sambhavna:
How can you even think about it? I am not confident about the code quality and there is so much to test. Too little time and too much to test such a crappy code!
Anil:
Hello madam? How can you call my code a crappy code? Do you even have an idea about the amount of design and planning that has gone into it? For starters, Each module is designed in way that it can be reused!!!
Sambhavna:
Hello Mister! Let your code be usable first. We can think about reusing it later!
Ritesh:
<<Consoling Anil …>> Chill out dude .. it is not as crappy …<<little pause>> as it was last time .. !!! (Glancing the test team members)
Chaturlingam:
I agree with our drama queen for once....lunch can wait...there are too many bugs.. I need to call for an immediate triage? We can wind that up and then go for lunch.
TEST TEAM IN CHORUS: Then it will DINNER !!!!
<ANNOUNCEMENT : Triage meeting is called out>

< SCENE #3>-- (THE TRIAGE)
Chaturlingam:
To start with let’s looks at Defect ID #3601, I get a system error when I give an invalid file (negative testing)
Anil:
Man..…go and see the error report first to see the root cause
Chaturlingam:
Really? This is what the error is!

Anil: (Looks embarrassed)
Err… I will take care of it.
Sambhavna:
Look, look, it’s not working. I swear, I saw it coming. I knew it would break. Thanks god i found it in test otherwise i don’t know what would have happened in production. How could you guys miss it?
Rakhi:
What’s the issue?
Ranchoddas:
Don’t show them. First log it in the tool. That’s the process.
Rakhi # (Gives up and walks to drama queen’s cabin):
Can I have a look at it?
Ranchoddas:
Don’t you dare fixing the code on her machine?
Rakhi:
Dude. I cannot change the code through an exe. Remember?
Ranchoddas:
Of course I know that! But I cannot trust you guys at all!
Sambhavna (finally explains);
See the spelling of “ORGANISE” . it has to be “ORGANIZE”.
Rakhi (Almost cursing herself):
What’s the big deal? I thought we have customer in UK as well.
Sambhavna:
For your information, 51 % of the customers are in US and hence it has to be US English
Rakhi:
Plz go log a bug
Sambhavna:
S1 / P1 ?
Rakhi (seeing her patience tested):
S3/P3
Sambhavna:
How is it S3? Can you not see this is such a severe bug?
Shijavi (takes out a doc which he keeps ready all the time)
Wait a minute. I have a reference, exactly for such situations. Microsoft’s Severity /Priority guidelines
<Anil & Rakhi are surprised that he carries such a thing all the time>
Ranchoddas:
As per Slide 4, line 2, this bug is S2/P2 to say the least
Anil (seeing they are losing the ground)
Agreed.
Ritesh: (to developer):
Boss …, looks like there is indeed some problem, but not big enough
Anil (looking tensed):
What happened?
Ritesh:
Look at this dialog box. Is it not silly ?
….. <<getting a call in between …>> … <<resume after little pause>>
I will log a S2-P3 bug, I’m sure you can fix it!

Ranchoddas:
Can’t you take personal call later ?
Ritesh:
Oh..dude, we can manage this triage meeting. Managing home triage is more difficult...!!
Ranchoddas:
Okay … I wanted to ask, how is it a S2-P3 man? Can you not see that it’s a Fatal error? Has to be a S1!
Ritesh:
Oh common, its sure is not a show-stopper! No panic dude !..<<little pause>> Btw …., I have one more.

Look at this screen, it’s throwing error that “Keyboard failure” but asking me to press F1 to continue …. No big deal. Your code is not crappy dude, I think, I should change the keyboard & try this scenario
Ranchoddas:
Dude, what is a Sev 1 bug in your dictionary!!
Chaturlingam:<Mighty pleased with himself>
Okay. I now have an awesome bug. Something which stands up to my reputation and will give you guys sleepless nights
<Rakhi & Anil make a face and wait for Chaturlingam to continue>
I put up a 30GB file for copy from the Source to Destination folder, and then I put some load on the system as it is copying. I then tried to remove the network cable and see what happens, and viola, I have the bug! See this! Do you expect my grandkids to come and test this?

And this ? What were you doing when you created this dialog box in your code?

Rakhi & Anil: <Look at each other and say at the same time>
Oh come on, man! Who on earth would even think of doing something as foolish as this!!!
Rakhi:
Did you log this bug? I am going to close this as Not-Repro!
Anil:
Its okay Rakhi. Lets fix it! Lets ensure that a graceful error is thrown when someone this foolish does something this weird on the application!
Sambhavna:
Okay, now my bug! This one took off my confidence on the code totally! What on earth am I supposed to do?

Rakhi:<smirks>
Press ‘Proceed’ and see na. I ensure it would format your system!
Sambhavna:
I have logged this bug. Fix it! What if a customer gets this bug? Imagine what would happen to our reputation if we ship this?
Rakhi:
Oh well, no customer will get this bug! I ensure that this happens only on your machine! This block of code would get executed only for yours and Ranchoddas’s login! But now that you proved to be smart enough to find it, I will fix it!
Sambhavna:
See this one! I told it I don’t want those drivers on my machine! God knows what you guys have coded into the drivers!

Anil:
Uncalled for but I accept this one!
Ranchoddas:
If all these weren’t enough, I have one now. And because of this one, I reject the build. Please give a new build!

Rakhi & Anil: <Too stumped to say anything>
Rakhi:
That’s it! Enough is enough. I am going to set up a 1:1 with my manager. Cannot work with these loonies anymore!
Ranchoddas:
Can someone send minutes of this meeting ?
Chaturlingam:
Boss … you are the quality “Guru” .. you can take care of sending minutes.
Ritesh:
Hey guys … itz been long battle today, let’s chill out. It seems some tester’s day is going on and looks like they are providing free lunch J lets go.

Narrator:
Thus ended yet another eventful day in the lives of these team members.
In spite of all the weird bugs, the stringent processes, the long coffee breaks and the amazing histrionics, this software was shipped and was reported to have touched a user base of 1M customers!
Anil is now a Dev lead, and it is heard that he has extra long sessions with his shrink, all for his sanity
Rakhi has moved away from the team, into a totally different discipline!
All the test team members are still flourishing in the same team, and continuing to give sleepless nights to their Devs!

PS: By the way, did i tell you that this play was quite a success at the event? Feel free to use it -- Raj, Microsoft, India , raj.kamal13@gmail.com

Using Code coverage in Black Box testing ain’t no rocket science :) Overcome your CODEPHOBIA

2010-07-20T20:26:00.001+05:30

I deserve few more comments here as I travelled 6 km and 30 mins extra to write this blog as in the excitement of writing this i forgot to get down at my stop and ended up at a place where the driver had to tell me that it was the last stop :P

“Yeah our developers are not doing code coverage and hence our code looks ugly” OR “we are not doing white box testing so we really cant measure code coverage" Heard it before ? Now Rocky (our tester) was in one such meeting recently and he was the only odd man out who believed otherwise. It took us some time to realize that it was the “CODEPHOBIA of the testers which is the major resistance when it comes to measuring code coverage and not as much as the actual process”.

This reminds me of one of my favorite commercial which has this one liner “Jhooth bolte hai woh log joh kehte hai unhe darr nahi lagta, darr sabko lagta hai. darr se aage bado, kyunki darr ke aage jeet hai” (For people who don’t understand Hindi: “People who say they aren’t afraid of anything are plain lying. Everybody gets scared (and its very human) but the key is to face it and then only you emerge as a winner”

When i interviewed the teams, I came to this understanding that the testers feel when they are no experts in .Net, Java, Ajax etc then how can they measure and improve the code coverage. I don’t completely agree by this notion as measuring code coverage doesn’t necessarily need very strong programming knowledge. Provided that you have a basic understanding of any programming language like C, Perl etc and a strong willingness to learn, you can still ramp up pretty fast to the extent which is required for you to be able to measure/improve code coverage. Understanding code is much easier than writing it :) and all it needs is aptitude and basic understanding of the syntax and semantic of the language in question. I am saying this based on my own experiences where I went an learned the technology when it was required to be able to accomplish targets like measuring and improving code coverage.

Now if you are saying “Why we are even talking about code coverage”?

Requirement coverage is ensured by tracing the requirements to your test cases and ensuring user needs are met but what about the traceability between the code and the test. In other words, what happens if there is code which is not exercised by your test and results into unexpected behavior at customer’s end.

In short because Test Coverage = Requirement Coverage + Code Coverage

Lets accept it that “requirements” and “code” are two different entities and validating just one of these would be unfair to the other one.

Now if agree that code coverage can be done by black box testers and there are no major obstacles then lets move on to “How” part. Well, so if there is a will then there is a way and fortunately in this case there are many ways. We need to choose what fits best for us.

Step 1: Measure you baseline and set target: is to see where you stand now by measuring the BASELINE code coverage for your project (as you cant improve what you cant measure) and then decide your realistic TARGET code coverage number. Now don’t get me wrong for highlighting keyword “realistic”. Yes, I am no conservative and everybody want 100 out of 100 (no less) but the practitioners would tell you its same as “exhaustive testing” which we know doesn’t come for free. So you got to decide on a number by seeing the criticality of your application and a trade off between the money spent in increasing code coverage vs. risk of missing a defect due to insufficient code coverage.

for e.g. if your test (manual + automated) provides you 40 % coverage today then it becomes your baseline and you can set a target say 70-80 % for the future releases by doing all calculation on (investment vs. ROI)

Step 2: Improve and measure (continuous)

Then you can go and measure release-over-release IMPROVED code coverage and then compare it against the baseline to detect the trend (if there are any improvement), till you reach your target. Once you achieve your target code coverage number, you can raise the bars to take it to next level and continuously improve it.

There are tools that can be used to start these trace using command line options which will instrument and track the code execution (statement coverage, path coverage, branch coverage etc) based on the tests being run (both automated & manual) and then help you generate reports to see the impact/coverage of your tests. I have given few links below which will point to some of these tools and the tool selection/adoption is at the reader’s discretion.

Is there any process which can be used to achieve this. Answer is YES. Read on.

Code Coverage process:

I have come up with a generic iterative code coverage process which we can be used for measuring & improving code coverage while test execution (black box or white box).

1. Start the Trace using Code Coverage tool

    Here you install, configure and start the CC tool before starting your test execution.

2. Test Execution – Phase 1

    You are running you test cases or scenarios or conditions (manual or automated whatever)

3. Measure Coverage & Identify areas where coverage is low

At this point lets say when you have done one pass of testing or have run your test cases once, you can Generate code coverage reports using the CC tool in question and IDENTIFY rooms for improvement where code coverage is not good.

4. Add Test cases / conditions to improve code coverage

Now you know which segment of the code was not covered by our test so you can go and either

a) write new test cases or scenario to cover those parts

b) Include the test data which can exercise those missing conditions

5. Test Execution – Phase 2

   You start again and this time you run these new test cases / conditions with new test data.

6. Measure Coverage again

You generate code coverage reports AGAIN and you see if there are improvements

Loop: Keep repeating steps 2 to 6, till you reach your target. End the trace

Pointers to few CC tools:

http://www.codecoveragetools.com/index.php/coverage-process/code-coverage-tools-c.html

http://www.codecoveragetools.com/index.php/coverage-process/code-coverage-tools-java.html

If you still with me then you are among those who are willing to learn new things in life and i wish you a ''Happy Code Coverage”. Go celebrate it. If it helps then why not use it. Technical knowledge shouldn’t be an excuse for not doing it. \

After all it is no rocket science.

PS: CODEPHOBIA in my dictionary is “fear of code; not only writing even reading or trying to understand it”

Bhopal’s man-made disaster (Testing Tragedies #2: Learning from past)

2010-07-19T21:05:00.001+05:30

Bhopal’s gad tragedy is one of the most sad example of ‘things-went-wrong’ due to human negligence. We primarily blame the company, the management (& rightly so) but one thing we often overlook is that other than human error it is the system (machines) that failed us too. If the quality was given its due importance, we could have possibly averted this. This blog is part of the series where we are trying to learn from our past error in various fields and apply those learning in software testing world. In Churchill’s words “Those that fail to learn from history, are doomed to repeat it.”

History

The Bhopal disaster or Bhopal Gas Tragedy is the world's worst industrial catastrophe. It occurred on the night of December 2-3, 1984 at the Union Carbide India Limited (UCIL) pesticide plant in Bhopal, Madhya Pradesh, India

Loss:

Government agencies estimate 15,000 deaths

Bad Quality

Attempts to reduce expenses affected the factory's employees and their conditions. Kurzman argues that "cuts ... meant less stringent quality control and thus looser safety rules. A pipe leaked? Don't replace it, employees said they were told ... MIC workers needed more training? They could do with less. Workers were forced to use English manuals, even though only a few had a grasp of the language”

Timeline, Summary

21:00 Water cleaning of pipes starts.
22:00 Water enters tank 610, reaction starts.
22:30 Gases are emitted from the vent gas scrubber tower.
22:30 First sensations due to the gases are felt—suffocation, cough, burning eyes and vomiting.
1:00 Police are alerted. Residents of the area evacuate. Union Carbide director denies any leak.
2:00 The first people reached hospital Symptoms include visual impairment and blindness, respiratory difficulties, frothing at the mouth, and vomiting.
2:10 The alarm is heard outside the plant.
4:00 The gases are brought under control.
7:00 Immediate death toll raises to 2,259
7:00 A police loudspeaker broadcasts: "Everything is normal".

Flaw:

During the night of December 2–3, 1984, large amounts of water entered tank 610, containing 42 tons of methyl isocyanate(MIC). The resulting exothermic reaction increased the temperature inside the tank to over 200 °C (392 °F), raising the pressure to a level the tank was not designed to withstand. This forced the emergency venting of pressure from the MIC holding tank, releasing a large volume of toxic gases into the atmosphere. The gases flooded the city of Bhopal, causing great panic as people woke up with a burning sensation in their lungs. Thousands died immediately from the effects of the gas and many were trampled in the panic

Incident Logs & My Observations:

Factors leading to the gas leak include:

The use of hazardous chemicals (MIC) instead of less dangerous ones
Plant's location near a densely populated area

      [raj]: Its a perfect example of poor risk management. We keep saying that test plan should have an important component which should talk about risk identification, risk assessment, risk mitigation and risk contingency but do we really do our part with sincerity.

      When you have alternatives one should always evaluate the worst case impact of each risk and then decide the alternative which is least impactful to the end users as user’s impact in many cases can be much more important than the cost effective   solution which might look better deal to us. In software world we can say that if there is a trade-off then choose the option which is say slightly more time consuming or costly over cheapest solution if it can pose some risk to the end users which can ultimately become too expensive.

Storing these chemicals in large tanks instead of over 200 steel drums. Large-scale storage of MIC before processing

      [raj]: Poor capacity planning. Over utilization of resources can lead to fatigues and fatal outcomes as it happened in this case. It is crucial to understand the threshold of each resource (software, hardware or humans) and understanding how much you can stretch them before they will collapse or explode. Resources are like a rubber band in your hand which can be stretched to meet your needs as long as you are not going beyond its threshold but if you keep stretching it will break and will hurt your finger real bad because of its rebound speed)

Safety systems being switched off to save money—including the MIC tank refrigeration system which alone would have prevented the disaster
lack of skilled operators due to the staffing policy
there had been a reduction of safety management due to reducing the staff, there was insufficient maintenance of the plant and there were only very loose plans for the course of action in the event of an emergency

      [raj]: Cost controlling by resource reduction (software, hardware or humans) should NOT be done blindly. This is more applicable now when the economies are struggling, as many times it is seen as the simplistic and the most obvious solution but it can have such adverse impacts as we have witnessed in this tragedy. We all should have moral responsibility toward our users and we should think twice before taking such measures. There can be other ways to save cost but leaving critical systems running in modes where they are not maintained or giving in the hands of people who are not trained to do is nothing less than playing with the life of our users.

The MIC tank alarms had not worked for four years
There was only one manual back-up system, compared to a four-stage system used in the US

[raj]: This only emphasizes the need of practicing security measures constantly to ensure that there are enough preventive techniques in place to avoid such accidents. Example in software could be anti-viruses or the scans/tests to be performed on periodic basis to know the possible threats in time.

Learning

      #1 Risk management should not just be done as a formality. Its not something that should only concern management or a project manager.

           It should be one of the most important item which should be raised, managed, controlled and monitored by every team throughout project lifecycle.

     #2 Choices made by humans CHANGE the course of future

         When you do risk management and decide on a choice, always remember that each choice you make can decide the fate of millions of users and hence ALWAYS think of your users while taking a call.

         Risk management is not just about you and your team, its also about the impact of that risk on your end users.

     #3 Over utilization beyond a limit can backfire. Know your boundaries and use them diligently. Cost reduction is important but constant overloading can instantly fail your system if not used judiciously.

          for example, In software world, load testing is important but stress testing is even more to understand when your software is going to break so that we can be better prepared for the failures. “Hope for the best; Prepare for the worst :P”

    #4 Cost reduction when done without understanding the impact on the end users can be counter productive.

    #5. For any application which involves direct or indirect physical interaction with users, SAFETY testing is must and needs to be done constantly to avoid accidents. e.g. Healthcare industry, Aviation etc.

    #6 Security testing is extremely critical when there are chances of sabotage in the system. In software it can be a malicious user who is trying to harm the system for his interest or to hurt the users.

    #7 Testability should be built into every system to find out the exact root cause of the behavior when something goes wrong.

         Even today the interested parties haven’t been able to reproduce the exact conditions that resulted into this gas tragedy.

         Only if the software has logging mechanism which can be used to find out the exact sequence of events that occurred before the failure, the root cause can be determined unanimously without leaving any room for speculation.

Thoughts for you

[raj]: If you think we have really learnt from past then read the comments below. Forget about applying the learning into other industries or fields, the very same company’s toxic waste is still lying there waiting to explode anytime again.

In software testing, I learn that if a product crashes or causes a major loss to the customer, then there will no guarantee that preventive or even corrective actions will taken for sure. History has proved that we humans are so good at repeating our mistakes and suffering from it again and again without really fixing it. If you want to be good tester, make sure making mistake once is acceptable, repeating it again and again is sheer stupidity.

Clean-up operations: Lack of political willpower has led to a stalemate on the issue of cleaning up the plant and its environs of hundreds of tonnes of toxic waste, which has been left untouched. Environmentalists have warned that the waste is a potential minefield in the heart of the city, and the resulting contamination may lead to decades of slow poisoning, and diseases affecting the nervous system, liver and kidneys in humans. According to activists, there are studies showing that the rates of cancer and other ailments are high in the region.

Life Saver or Life Taker ? (Therac-25) – Impact of poor testing (Testing Tragedies #1: Learning from past)

2010-07-13T00:48:00.001+05:30

This blog is for everyone who wants to know how software testing job touches human lives and why defects in applications such as healthcare cant be ignored.

History:

The Therac-25 was a radiation therapy machine produced by Atomic Energy of Canada Limited (AECL) It was involved with at least six accidents between 1985 and 1987, in which patients were given massive overdoses of radiation, approximately 100 times the intended dose.

Loss:

Four of the six patients died as a direct result of poor design, coding and testing

Company’s Response

After careful consideration, we are of the opinion that this damage could not have been produced by any malfunction of the Therac-25 or by any operator error

[raj]: Only if the company had taken very first of these incidents seriously they could have saved 3 precious lives. Every critical issue which customer finds should be given utmost priority before it becomes much worse.

Facts

Only One person did the programming for this system and he largely did all the testing.

Therac-25 was tested as a whole machine rather then in separate modules.

[raj]: Yes, that was my reaction too. We left the lives of so many hundreds in the hands of ‘One/ person only . We are humans and to err is humans. Plus we humans are not so good in finding errors in our own work.

If System and Integration testing is important, so is Unit testing, we can undermine the importance of any of these. They are meant to complement each other. becomes much worse.

Incident Log & My Observations

Severity 1 Production Defect #1:

A 40 year old women was receiving her 24^th Therac-25 treatment. The machine stopped 5 seconds into the treatment with an error. The technician seeing that "No Dose" had been administered (according to the computer) hit the 'P' key thus proceeding with the dose. This was done a total of 5 times giving the patient 13 000 - 17 000 rads. To give an idea of how much of an overdose this is; a regular treatment is around 200 rads and 1000 rads of radiation to the entire body can be fatal. The patient died 3 months after the overdose.

Severity 1 Production Defect #4:

The patient required only a small dose and according to the machine that is all he received. Yet again when the treatment was underway and error paused the machine and the technician hit the 'P' key to proceed. A overdose was administered and the man died just 3 months later.

[raj]: Better testability could have warned the technician that the dose had already been delivered. Misleading information and lack of transparency through the system confused him and he went on repeating the procedure again and again which made it fatal.

Severity 1 Production Defect #2:

A male required radiation treatment on his back. The machine was set to X-ray mode instead of Electron mode so the technician just used the "cursor up" key up and quickly changed this mistake. However, this only made things worse as a software bug had been mistakenly stumbled upon. While administering the first treatment an error "Malfunction 54" flashed up telling the technician an underdose had been administered. The technician hit the 'P' key and a 2nd dose was delivered. The patient had been given an overdose after the first treatment, and he knew something was wrong, due to the burning sensation he felt in his back. As he attempted to get up the 2^nd dose was administered. The technician would have known the man was in pain if the audio and visual equipment was working. his man within weeks, lost the use of both legs and his left arm. Five months later he became the first fatality directly related to the Therac-25 system.

Severity 1 Production Defect #3:

A month later at the same hospital, with the same technician another fatal dosage was given. The technician made the same error of quickly changing the mode from X-ray mode to Electron mode using the 'cursor up' key. This again caused "Malfunction 54". The patient this time was receiving treatment on his face. When the overdose was administered he yelled and then began to moan. The audio equipment was working this time but the initial dose was too much for the man. He received severe neurological damage, fell into a coma and died only 3 weeks later.

[raj]: If system was designed considering that a simple wrong choice can have such adverse effects then a choice made by technician could have warned him and possibly stopped him from making that mistake.

Learning

Learning #1: Never dismiss any failure without reaching the bottom of it. Over confidence about your quality can take you and your customer down

Learning #2: Never depend on just one resource for the entire functionality. It’s dangerous. and it takes two to tango (certain activities can’t be achieved singly like arguing, fighting, dancing, making love :))

Learning #3: Unit, Integration and System testing, they all are equally important and one shouldn’t undermine importance of any of these.

Learning #4: Poor testability is extremely fatal. Lacks of user’s ability to validate the completion of the software operation/task can take lives as we have seen above

Learning #5: Don’t repeat any important function/operation/task without confirming the behaviour of the previous operation. Many times we think that running the software function again is case of failure is perfectly fine but that can be risky if the last operation resulted into corruption or left the machine in inconsistent state

Learning #6: For critical functions in your software, ensure there are provisions to handle silly human errors where we perform an action what we don't intend. Design should consider that humans can make mistakes and for important tasks, there should be a warning/message confirming the change (that can possibly warn him and correct the action as he intended).

Example of such human mistakes.

we want to click on Checkbox Yes but because page scroll happens and we click on NO and we don't even notice it

or

we are not 100 % concentrating and our brain is lost in it thoughts and we humans are sometime unaware of the the action performed.

e.g. I bet you would have felt the same more than once “Have i left the tap open after the bath?” when you would have closed it.

Thoughts for you

If you are thinking,this was a rare scenarios and example of worst engineering and the machine would have got retired for ever then i want to leave you with this fact that this machine is still in use today and there might be someone you know who might be sitting in front of the machine as we speak and that’s why it is important to find defects before a life-saver turns life-taker

Adventures with Testing BI/DW Applications (Business Intelligence / Data Warehoue Testing)

2010-07-05T20:36:00.003+05:30

Adventures With Testing BI/DW Application

Saga of Unsung Heroes (A tribute to all testers in this world)

2010-07-02T18:06:00.000+05:30

Saga of Unsung Heroes (A tribute to all testers in this world)

Correlation between Cyclometric Complexity & Code Coverage while doing White box testing

2009-12-17T16:51:00.001+05:30

If you are among those testers (or developers) who have often wondered how cyclometric complexity can practically help me achieve higher code coverage and uncovering max number of bugs by executing minimum no. of unit test then please read on. I have tried to take a very simple example and have focussed on explaining the very basics with data and images only to make sure you can visualize and understand the differences between statement coverage, decision coverage and path coverage and their relationship with cyclometric complexity for once and all.

Pseudocode

Read a;

Read b;

if(a>=b)

print a

else

print b

Do x;

if (a ==0)

print “Zero”

do nothing;

Flow chart of the above pseudocode:

Cyclometric Complexity:

= E-N + 2P (where E is no. of edges, N is the no. of node and P is the no. of connected components in the above graph)

= 10 – 9 + 2 x 1

= 3

Minimum Test Required for 100 % Statement coverage:

Unit Test #1:

Test data: a = 0, b = –1 (True & True)

Unit Test #2:

Test data = a =0, b = 2 (False & True)

Total test required for 100 % statement coverage = 2

With the above two test cases we are able to execute each line of code at least once.

Minimum Test Required for 100 % Branch Coverage or Decision Coverage

Unit Test #1:

Test data: a = 0, b = –1 (True & True)

Unit Test #2:

Test data: a =1, b = 2 (False & False)

Total test required for 100 % branch coverage = 2

Minimum Test Required for 100 % Path Coverage (Basis Path Coverage)

Unit Test #1:

Test data: a = 0, b = –1 (True & True Input)

< same as scenario #1 of statement coverage>

Unit Test #2:

Test Data : a = 1, b =0 (True & False Input)

Unit Test #3

Test data: a =0, b = 2 (False & True Input)

Unit Test #4:

Test data: a =1, b = 2 (False and False Input)

<same as Scenario #2 of branch coverage>

Total test required for 100 % path coverage = 4 (Which is also 2^2 choices where 2 is n decided by first IF-ELSE)

basically what we have done above can also be shown in the form the grid:

Input\Output	First IF	O/p of First IF	Second IF	O/P of Second IF
a = 0, b = –1	True	Print a	True	Print 0
a = 1, b =0	True	Print a	False	-
a =0, b = 2	False	Print b	True	print a
a =1, b = 2	False	Print b	False	-

Universal Formula:

Branch Coverage <= Cyclometric Complexity <= Path Coverage

Replacing our values:

2 <= 3 <= 4 (holds good :))

Now what do we conclude by doing all this:

Does 100 % statement coverage ensures no bugs?

Answer is NO. 100% statement coverage just ensure every statement is executed at least once BUT it doesn’t guarantee that all the conditions are tested for different combination of data and hence it is not the most efficient method

Does 100 % branch coverage ensures no bugs?

Answer is again NO. 100 % branch coverage is certainly better than statement coverage and more efficient but it only ensures that each branch is executed at least once but again it doesn’t cover all the possible combinations (like two Ifs can be tested with 2 branch test case for TT, FF where as total possible combination will be 4 (TT,TF,FT,FF) which are not getting covered here)

Which is the best way of ensuring 100 % code coverage.

Answer is Path Coverage (a.k.a. basis path testing). This is better than both statement and branch coverage. Though this is practically very impossible to cover all permutations and combinations (For ex when there are 2 IF statements, total paths are 2^2= 4 combination and as no. of Ifs in the program goes to n, no of possible combination goes to 2^n)

Thats where cyclomertic complexity is used to find out the min test to be executed to achieve path coverage and the max test to be executed for branch coverage.

In above example, if the bug was in a condition when one first IF resulted into True (YES) and Second IF resulted into False (NO) then none statement coverage or branch coverage could find that bug and hence cyclometric complexity = 3 shows that adding one more scenario to branch coverage as TF or FT (other than TT,FF) will help to uncover that bug so minimum 3 test are required to have high probability of uncovering bugs but max 4 test are required (max possible combinations) to ensure no bugs, which generally become impractical with the growing code containing multiple nested if-else and loops.

Here cyclometric complexity helps unit tester decides at least what are the min no. of test that should be executed to cover the scenarios which have high probability of finding bugs out of total possible scenarios of path coverage (Which is 2^n)

PS: I want to thank Renu (an enthusiastic tester) to bring this topic in our discussions which triggered this interest in me to link all these pieces and try giving an overall perspective to testers by taking this simple example.

Feel free to write your comments to have a discussion or to clarify a point if you disagree.

Do you have it in you? Technical skills are no more OPTIONAL skills for Software Testing

2009-10-07T15:41:00.001+05:30

Words of Caution: When i say “Technical skills” please don't confuse it with “Test Automation” knowledge only. It more and the less has become a standard in the industry to presume that knowing QTP or RFT is same as having technical skills.

To start with, for a tester to be successful following are the 3 most important components which you should never forget:

In this blog post, we will talk about the second most important component “Technical Skills” which is only next to “Testing Skills & Aptitude” for a tester to be successful in the next decade.

In last decade or so, testing has been tagged and given the title of an easy profession where non-technical candidates (who aren’t capable of writing or understanding the code) can write some test cases and execute them to find issues. It was considered having domain knowledge to understand the business is sufficient to be a successful tester.

In last 5 yrs, automation has become the talk of the town as it has become a successful marketing tool to win more projects by showing huge ROIs benefits of million of dollars to the customers. We have also witnessed the same trend by the emergence of specialized commercial testing tools by big players like IBM, HP & Microsoft in this arena.

Lot of candidates in testing do have technical skills but they don't hone it or have stopped using them because of the popular perception that “its not test‘s job to write/read code”. The max they do is automation (where record/playback is most popular and only a few percentage (<20%) can truly write automation code to validate the application)

Because testing is considered a second fiddle to Development and the best minds get/shift into development, testing community is lagging seriously on candidates with strong technical skills.

Test automation is a subset of a broader category “Technical Skills” and now (more than ever), importance is being given to other technical skills other than test automation which are becoming mandatory to become a successful test professionals at organizations.

Following 10 are the skills / activities / tasks where technical skills of a tester is tested

(you might have read many of these in your testing training / process tutorials but now time has come to start exercising these)

1. Requirement Feasibility:

Don’t get surprised, there is not typo error here. You read it all right that requirement analysis ALSO requires you to have technical skills to understand if the requirements expected by the customer are POSSIBLE or not. Lot of times there are “technology constraints” and unless you understand the underlying technology you run a risk of committing on a requirement which can’t be implemented and you must know that telling customer on the last day wont be such a great idea.

Its not only dev job to call out technical constraints. In fact test can call out if a particular requirement is too unrealistic or ambitious to be implemented if they understand the technology well.

For ex. I can tell the business upfront that a specific multi select tree control is not present in c# and we need to think of an alternative.

2. Scalability Study (Hardware/Software):

Your app used to work fine and then a new functionality was added and then in the next release all of a sudden the app started hanging or the batch job is taking infinite time. If you have had experience something similar to this, you are a victim of “hardware/software constraints” which was not thought in advance.

Lets say the users have increased to million from thousand and you hardware RAM and Processor is not enough to handle so much traffic

Your RDBMS cant handle more than 10000 concurrent connections.

So it is must for you to understand your platform (hardware + software) and their boundary limits so that you can avoid surprised and downtimes in production leading to SLA misses and customer dissatisfaction.

3. Design consideration

When is the last time you were in a design meeting where you contributed to the translation of the requirements into design or more importantly, when you helped avoiding developers to commit a design blunder where let’s say they misunderstood the requirement and the flow was incorrect.

Unless you understand the development models and don't participate in high level and low level designs meetings you wont understand what’s happening “behind-the-scenes”.

It can also help you to write better test cases by understanding the flow of the system and how modules are integrated. Black box testing is not sufficient always when the architecture & the flow becomes complex as not all implementation details are captured in the functional specifications.

Technical design documents should also be evaluated by the testing team where say why Approach A is preferred over other options considered like Approach B or C.

4. Code walkthrough & Inspection

This might give nightmares to many test professions. They just hate looking at the code but believe me you can find lot of bugs by inspecting the code even before the code is dropped to testing. This static testing can save lot of time by avoiding a buggy code to be dropped to test.

You must understand the syntax and semantics of the language being used (C#, java etc) and ask the right questions to find out any memory leak or incorrect loop/condition clause being used in the code

5. Unit Testing & Integration Testing

Okay I wont start a debate by saying test should do unit testing but you will surely come across a situation in your long testing career when you will agree that knowing unit testing help a big deal.

For example, how do you do code coverage now? Unless you know all the inputs/outputs, all the main flows and branches, all the conditions and exception handling code, we cant write good scenarios to exercise critical code paths.

6. Performance tuning

With more and more focus now given on customer satisfaction, performance testing is something now which goes without saying and every customer expects it by default. It is extremely critical now for test team to ensure that the performance is well within customer’s expectations and with high traffic OR volume of data the application would respond within the SLA agreed at the time of signing-off requirements (as part of acceptance criteria).

Unless test teams are familiar with performance counters of the operating system and the the software / technology being used, bottlenecks and potential issues cant be determined.

Also when you want to simulate real-world scenario say to simulate 100000 concurrent users, the knowledge of session management and other technical concepts are must.

Many a times the queries / code are not optimized and it is also important to understand if the code written is efficient or not and also if there is a better way of writing the code which can improve the product’s performance.

8. Security testing

With the increasing number of virus attacks and the risk involved with losing highly confidential customer data or any possible frauds, tester’s contribution has become very crucial.

For example:

How can you test your application for SQL injection if you aren’t good at SQL?

How would you test your application for exposing any confidential data if you aren’t good at encryption/decryption algorithms or understanding different masking techniques?

How do you ensure which port shouldn’t be open/closed unless you understand the networking concepts well.

These are just some high level examples trust me security testing is much more than that and now constitutes of many modern & advanced techniques like fuzzing etc

9. Deployment & Prod Support

We are the gatekeepers of the application. It goes from our hand to Support folks for UAT & Production deployment. If we don't understand the configurations setting or environment pre-requisites to run our applications, chances are there that business would lose lot of their precious time waiting for your dev team to come and rescue you.

I am sure you don't want to end up there and its critical to understand the installation and configuration setting, troubleshooting tips and common FAQs

For example, there was a permission issue which was holding your user from using the application and you just don't know how to give the appropriate permission. Believe me it can be very embarrassing and frustrating at times.

That’s your lucky chance to build credibility with business and Support folks and help them get-going without wasting any amount of time.

10. Automation

And just because I haven’t kept at top of the list doesn't mean its less important than any of the above activities. Its the most important activity only next to manual testing but lot has been written about it and there is much more understanding and awareness about test automation in the industry now and hence I am keeping this short.

With this I am closing this post and i hope this will provide a quick checklist to the testers to become ideal testers-of-future.

Testing:: Chosen-By-Profession turned Chose-the-Profession.

2009-10-01T21:22:00.001+05:30

It was morning of 12th Nov, 2004 when I found myself standing in a classroom and for the first time it was my back facing the black board and a new batch of future software professionals sitting in front of me and most of them were as anxious as me (obviously for different reasons). These young professionals fresh out of top colleges in India were campus hired ELTs (Entry Level Trainees) of Cognizant Technology Solutions.

Since that day and the day when I am typing this I have come across many Software Testers (99.9% of them where chosen by the profession and not otherwise) and the questions I have been asked or I could read it on their face is:

“What am I doing in Software Testing Batch? OR Why am I not present in .Net/Java Batch running in next classroom? Am I not good enough?”
“Don’t I have required technical skills to be in Development? OR Based on my aptitude have I been put into Testing?”
“Do people get lesser raise / bonus in Testing over Development?”
“If I perform well and prove my skills, will they shift me into development?”
“My elder brother/ cousin/parents/uncle etc told me that testing is not a good profession and its just about validating someone’s one. Is that true?”

Anyone who was present in one of those classrooms is reading this blog would know that raising above questions is definitely no sin and in fact I had the same questions myself as i was “chosen-by-the-profession” too. Please make a note here (I said “I was” and not “I am”).

I would have moved into development way back in 2004 but i CHOSE to stay in Software Testing and that was BY-CHOICE. Just to give you a brief background (no boasting here), I used to and still think that i am amazingly good in development (and yes i did all graduation lab & major projects on my own –be it C, C++,Computer Graphics Linux, Oracle, Java and got 3 scholarships for topping the college)

Times have changed since 2003-04 and now testing has turned out to be a lucrative and easy career option for candidates who find it difficult to crack into development but in next few paragraphs you will read “How I fell in love with Software Testing” and “Why Software testing is as good as Building software”

Software Testing is fun

If as a kid you enjoyed playing with a new electronic gadget more than building sand castles you will know what am i talking about?

If you had the curiosity and strong urge to see how watch or a transistor works, you will be able to correlate the itching that you get when you want to understand the new unexplored & unseen thing

so guys, testing is like seeing a product and then letting yourself go with the flow and try to understand how it works and where it doesn’t?

Human mind always wants to find out what you CANT do with any object in hand. and let’s say getting paid for it. Isn’t it a bonus?

Software Testing is an ART

Software Testing is art and so is software development as well but think of it that specifications and requirements say “what needs to be done” and design done by the architects & modellers says “how is it to be done” so development is left with very LITTLE room for innovation. I have known developers whose jobs are so monotonous and life is limited to just convert the plain English into a programming language code.

Now in Software Testing nobody says “HOW IT NEEDS TO BE TESTED?” :)

To an extent test plan and test strategy covers the scope and approaches to be followed but as a tester you can think innovative and challenge the very requirement or a workflow or a specific behaviour. You have absolute power to think of a undocumented alternative path and ultimately discover a MISSING/EXTRA requirement.

Software Testing is Advocacy

You like fighting for a cause ? You like making sure that justice is done?

You are customer’s advocate and most of the time he might not be even knowing it. Its YOU who has to ensure that justice is served and your customer is getting service / experience for the amount paid. Usability / Accessibility / Security / Performance are few non-explicit expectations which often get ignored by Developers, Managers, Business Analyst and that’s something can make a huge difference in a great customer experience.

Software Testing is Progressive

You want to be an entrepreneur? You want to understand your business?

There are lot of customer-facing professionals / business analyst who have been testers in their early career. The obvious reason why they are so successful is that they GET TO SEE A BIGGER PICTURE. They understand business better than developers because:

1. They clarify requirements / bugs directly or indirectly with customers

2. They understand the whole system (multiple modules and their integration)

3. They don’t get distracted by getting too deep in technology

Software Testing is Respectable

You have the key responsibility. You got to to take the call here. GO / NO-GO?

Customer & Management relies on your work to decide if we are ready to go into the wild and compete with the world.

You are the one who drives the Quality. You are the one who ensure processes are adhered to. You are the gate keeper. You are the important one here.

Software Testing is Challenge

If you are the one who don't like life to be boring or predictable, then join Testing. One life is not enough to learn everything in testing. Sometimes reproducing a simple-yet-important bug becomes a nightmare and at times understanding the customer requirement becomes so tough when you are working in an offshore model or when the dev slips and you are the one who has to still deliver quality without compromising on the timelines.

Software Testing is Technical

My favourite. This is exactly opposite to the popular belief that testing don’t need good testing skills. In fact, if you are not technical:

1. You won’t be able to repro many bugs because unless you understand what's-happening-behind-the-scenes you wont understand the problem

2. You wont be able to find the root cause many times. You will just be able to say “there is a problem” but “don’t-know-why”

3. You wont be able to suggest a better solutions / alternative

In the end, software testing is one profession that requires great communication skills, strong technical skills and most importantly a testing mind (passion to learn and understand new things)

Remember, Testing will Test you FOREVER.

Reduced / Selective Test Data Execution to SAVE Time & Effort - Introduction

2009-04-06T23:07:00.004+05:30

Context: Data Warehouse / Business Intelligence (ETL Execution)

Background:

If you have ever involved in DW/BI Testing, waiting for ETL execution to complete won't be an alien subject to you. Its not ONLY frustating to discover ETL fails at Step X after running for say 8 long hours but its ALSO a show stopper for you to continue with your functional testing and directly impacts your efficiency and effectiveness.

If there are performance lags (like low configuration h/w, network bandwidth etc), the ETL job is only going to take longer than expected and often times this adds to higher WAIT Time (a.k.a NON Productive time).

Root Cause:

If dev / test team wants to run the ETL with Production data, which can be really huge (in GBs or sometimes even TBs), due to the volume its obviously takes a long time.
Trade Off is REAL vs SIMULATED data. Testing with REAL data comes with a cost.

Is testing team the only victim?

Answer is NO. Even Dev team has to wait for entire ETL to run to see the changes made in the code while doing integration testing.

Why don't we create Smaller test data ALWAYS?

-> Due to the data integration and transformational complexities, the team has to have very deep domain knowledge of the business rules to create test data which satisfy the ETL conditions (often times lack of this results in ETL failure or ZERO o/p after the run)

-> If the complexity is high (consider data coming from multiple disparate sources) then creating data while maintainig logical relationship can turn out to be quite tedious and time consuming.

Why dont we SELECT REDUCED / SUBSET of Test data from Production?

This is the heart of this blog and we are going to emphasize on this in detail. Though the above sounds simple and the obvious solution but it has been observed that teams dont find it very popular time because of following important reasons:

1. Random sampling of data doesnt work. Many times it will result in ORPHAN RECORDS (where you might end up pulling child records without parents records from the source)

2. ETL might break if the keys are not found.

3. ETL might succeed but can result in ZERO output records as none of the business conditions got satisfied for the the sample data.

4. Business / Domain knowledge is required

What do we recommend to select SUBSET test data with success?

1. Select a pattern:

a) Master Data Driven

b) Transaction Data Driven:

2. Identify Business Conditions

3. Select Sampling Method

4. Copy the SELECTED test data to a test db

5. Modify Test Data to cover Boundary Value conditions

6. Run the ETL

What are the advantages?

1. Save Time

2. Save Effort

3. Save Hardware Cost (Less Data, Less Space and computational resources required)

4. Better Test Coverage (High Requirement Coverage)

5. Better Code Coverage

The subsequent post will focus on the above methods in detail....Please share your feedback and suggestions.

SSAS Cube Testing

2008-12-13T00:50:00.002+05:30

1.1 BVT / Smoke Testing Scenarios:

1. Ensure only the user with the desired permission is able to connect to the cube

2. Validate Data Source Connection string for cube

E.g. Provider=SQLOLEDB.1;Data Source=<Test ETL Server>.redmond.corp.microsoft.com;Integrated Security=SSPI;Initial Catalog=dbCPRMart

3. Validate your are able to successfully process the cube

4. Validate your are able to browse the cube

5. Configure and validating Analysis Services Query Logging

1.2 Data Validation Scenarios:

Approach 1: Browse the cube using BIDS or SQL Server Management Studio and compare the output against Mart using SQL query

If it’s a new development of a Cube, Browse the Cube, drag and drop measures on the Page and Dimension on the Y-axis. You need to write T-SQL that does similar aggregation that brings similar output.

Eg:

OWC Output:

                                 SALES        PROFIT

AP

      CHINA                 200                 60

      INDIA                   250                 50

US

      CHICAGO              500               65

       WASHINGTON      550               65

Equivalent T-SQL:

SELECT REGION, CITY, SUM(SALES ), SUM(PROFIT) FROM GEOG G JOIN SALES S FACT ON G.ID=S.ID

GROUP BY REGION, CITY ORDER BY 2

Approach 2: MDX query to compare data with SQL Query executed against Mart

2.1 For simpler Cube & Dimensional Models

Write your MDX query and compose SQL query for the same conditions and results should match

2.2 Automatically generate MDX using SQL Profiler & Reporting tools and compare it against SQL

SQL Profiler & Reporting Tools like PPS, ProClarity provide the capability of generating MDX on Cube conditions. Use these MDX against the SQL queries formed by you based on the relational / dimensional model on Mart.

Approach 3: Black box testing using sample test data

Insert / Update / Delete test data in your backend and calculate the outcome value based on the desired functionality / requirement without going to cube and ensuring that your expected output value matches the cube output in the final reports.

Approach 4: AMOMD object Compare

Create Automatic Verification Mechanism between OLAP and SQL objects

http://msdn.microsoft.com/en-us/library/cc281460.aspx

http://msdn.microsoft.com/en-us/library/cc280975.aspx

Figure: shows a high-level view of the AMO object model for OLAP objects

Reference:

http://microsoft.apress.com/feature/74/introduction-to-analysis-management-object-amo-programming

1.3 Cube Design Scenarios:

1. Validate all measures

Open Visual Studio analysis services DB

Browse and Open Cube DB

Double Cube and browse to "Measure" pane

Select measure <measure name>

Go to Properties, and check "source" field

     Expected Result:

Measure source should be set to correct table as source table and correct column as source column in Mart

Ensure measure has all the required fields / columns as present in the mart

2. Validate all dimensions & dimension hierarchies

Open Visual Studio analysis services DB

Browse and Open Cube DB

Double click Cube and open data source views and Open <Dimension name> Dimension under "Dimensions"

   Expected Result:

Make sure Table columns in Mart are mapped to Dimension correctly

Make sure Dimension key is correctly mapped to Dimension key column of the dimension Table in Mart

Make sure all the required columns / fields are present in dimension as present in mart

Ensure Hierarchy is correctly defined

Fiscal Year -> Fiscal Month -> Fiscal Week -> Calendar date

3. Validate all calculated measures

Open Visual Studio analysis services DB

Browse and Open Cube and go to "Calculated" tab

Check expression for "Calculated measure name"

    Expected Result

    The MDX expression defined should be accurate as per your requirements.

     Eg. [Measures].[ChangePoint Total Backlog] + [Measures].[Siebel High Pipe] + [Measures].[Siebel Low Pipe]

4. Validate “Data Source Views” of your cube against your design

1.4 Security Testing Scenarios:

1. Ensure each user belonging to a cube role has appropriate access

2. User with read permission should only have access to browse the cube

Read Definition checkbox should be selected

3. User with Process permission should be able to process the cube as well.

Process Database checkbox should be selected

4. User with Admin permission should be able to browse, process, make changes to the cube as well.

Full control (Administrator) checkbox should be selected

5. Cube roles should be mapped to correct users and group

6. Cube roles should be having restricted access or unlimited access to dimension data based on the design and project needs

1.5 Miscellaneous Scenarios:

1. Backup and Restore:

Take the backup of the cube and try restoring. The functionality should remain working as earlier.

1.6 Performance Scenarios:

1. Optimize Cube Aggregations

Before running your stress tests, you’ll want to ensure that you OLAP design has optimized aggregations. To optimize your aggregations, you’ll need to first turn on SQL Server Analysis Services query logging, then run the cube optimization wizard.

2. Using load simulator

Reduce number of invalid defects -> Improve Test Productivity and Efficiency & Let your Developers be happy

2008-11-12T14:05:00.003+05:30

Just the other day we were discussing that how can we (testing team) reduce number of invalid defects. I was in deep thinking that why is it really important to reduce the number of invalid defects.

Isn't it a tester's fundamental job to log every potential defect and let it go through normal defect life cycle and let business and management take call if it is a valid defect or not?
Isn't it right that tester shouldn't assume that it is not a bug and then regret later because of a false assumption made?
Isn't a tester taught to think negative and always be suspicious and uncover what is not seen by someone like dev?

The point here is what is a big fuss if testing team raises 'invalid' defects unknowingly. At least they don't leave anything to assumptions which is far more dangerous. There primary job is to find defects, whether that is a 'valid' defect or not is a secondary question.

One of the tester I am mentoring complained that his testing team had found 108 defects out of which 104 were valid and only 4 were invalid but his management didn't seem to appreciate them for the the number of valid defects found as they were expecting number of invalid defects to be zero.

My take on this:

Yes, it is important to reduce number of invalid defects.

Why?

-> Test Metrics gets screwed (Test Effectiveness or lets say Test Productivity goes down with no. of invalid defects)

Test Effectiveness = No .of Valid defect / (No. of Valid Defects + No of invalid Defects) X 100 %

Example for above: 104 / (104 + 4) X 100 % = ~96.6 %

-> Time lost in tracking and logging invalid bug

When you raise a bug in your reporting tool like Test Director, it has to go through complete bug lifecycle. Say you raised a bug spending effort in recording it and then it turned out to be invalid, you developer rejects it. Finally you have to close it.

-> Management don't like invalid bugs

You bet me that "invalid" bugs doesn't please any manager. It is a human behavior to criticize something that is not right. It sets them off.

-> Developers stop taking you seriously

When they observe that you raise many invalid bugs, they start expecting that every time. They stop paying due attention to valid bugs considering them to be invalid. Quality over Quantity concept.

-> Time lost in triage meeting to discuss invalid bugs

When you log invalid bug, it not only your time getting lost, developers waste their time reading them, then testers and developers waste time arguing on that as it has been officially logged, and most importantly business waste their time in triage meeting to take call.

-> Spoiling terms with development team

Developers are under pressure to reduce number of defect found in their code by the test team. Now if you log it, they go defensive and try their best to prove your bug an invalid bug so that it doesn't spoil their commitments.

Now that we have a problem. Let me propose something which we successfully implemented

Now with this process, every bugs gets verified online by the Development team even before we officially log it. They update the sheet saying that they are okay with so and so bugs and we log only those bugs in the Bug Management tool and hence all "VALID" bugs.

Bugs which they update as "INVALID" or "REJECTED", we update the SHARED SHEET with more information like repro steps etc and they change the status in the sheet accordingly. Now if it was our fault and it was actually INVALID defect, we update the SHEET and close it there itself rather than logging it in bug management tool and going through the entire process.

Now our metrics always say 100 % valid bugs. We don't miss any bugs because we record it anyway in the Shared Sheet and triage it with development team online. Development team feels good as they get a chance to to repro bug and confirm before it is actually logged against them. We don't have to waste time in the triage meetings discussing whether its a bug or not. Now business only take call on functional bugs which are more important to end user.

Butterfly Effect in Software Testing (its happening all the time, did you ever notice)

2008-11-05T22:47:00.002+05:30

"Small variation in the initial condition of a dynamic system may produce large variation in the long term behavior of the system"

Example: The phrase refers to the idea that a butterfly's wings might create tiny changes in the atmosphere that may ultimately alter the path of a tornado or delay, accelerate or even prevent the occurrence of a tornado in a certain location.

and for we guys "Small variation in the initial requirement of a dynamic application / product / system may produce large defects in the long term usage of the system"

Example: A tiny misinterpretation or misrepresentation of a customer expectation in the form of requirement can ultimately impact the analysis, design and code, testing in such a way that it might result in a product failure or in worst case business failure sometime

here tiny change in requirement can be compared with butterfly's flapping of wings and because this change was not detected in the requirement stage and it keeps on passing through analysis, design and coding phase and finally gets so much amplified that the impact becomes a huge loss to the business.

One of my friend had been asked in an interview "what is Domino effect in software testing?" He didn't know and hence couldn't answer.

It is again related to Butterfly effect and Chaos theory that a small change anywhere in the system (requirement, analysis, design, code, testing) can cause a similar change nearby and that will another similar change and so on in a linear sequence.

Example: A wrong change in requirement -> similar change in analysis to accommodate that -> similar change in design for that analysis-> similar change in code for that analysis -> similar change in test cases for that requirement and so on.

Interesting fact: The term is best known as a mechanical effect, and is used as an analogy to a falling row of dominoes.