This document provides 10 tips for getting the most out of automated tests. It recommends choosing a layered architecture with page objects and business logic helpers to make code reusable and easy to understand. Tests should be configurable, data-driven, and able to run in parallel for faster execution. Rich reports with screenshots, logs, and failure details help debug issues. Customized email reports consolidate results. Metrics like code coverage and failure analysis measure quality. Continuous integration with automated deployments and test execution provides faster feedback.
4. Why Automate?
• Manual is slow
• Faster feedback on Continuous Integration
– 24*7 testing
• Repetitive manual regression tests are
boring and make testers lose focus
• Manual testers focus on testing new
features
• Allows for faster upgrades and refactoring
• Consistency and Documentation
• Higher test coverage – multiple data
combinations
2/1/2016
4
5. Let’s see an example of
poorly written automation…
2/1/2016
5
6. Automation Design
• Example of a bad code Hardcoded
Browser
Hardcoded
Environment
Hardcoded
Timeout
2/1/2016
6
7. Automation Design
• Example of a bad code
Hardcoded
Object
Identifiers
Hardcoded
Test Data
Hardcoded
Selenium
Constructs
Repeated
Business
Logic
2/1/2016
7
8. Automation Design
• Non-Actionable Report
– High Failure Investigation Cost
Manually run
again to repro.
Can’t log bugs
from report
No
Screenshot /
Page HTML
Insufficient
Logging
2/1/2016
8
19. Automation Libs
• Selenium, QTP, etc.
• All external third party jars
2/1/2016
19
Framework Core
• Wrapper around various
external libraries
Class Description
Log Log reporting
Helper Various utility methods
DataBase Methods to interact with database
Popup Methods to handle popups, alerts on the page
TestData
Reader
Methods to read from test data sheet
Browser Methods to do operations on your Browser/Page
Element Methods to do operations on Elements of the Page
20. Tip #1:- Right Design- Benefits
• Reduce Maintenance cost
• Business logic at one place
• Reusable code
• Localized changes – Data and Object Repo
• Easy to understand test cases
• Easy to change underlying automation lib
2/1/2016
20
21. Tip #2:- Configurable
• Don’t hardcode – Load at runtime
– Environment URL’s
– Browser Name
– Platform
– ObjectWaitTime
– Database credentials
– Any other param which drives the execution
2/1/2016
21
22. Tip #3:- Data Driven
• Run same test case with different data combinations
Read login
credentials
from Excel
Test Data Excel
Row Number
2/1/2016
22
23. Tip #4:- Parallel Execution
• Write non-interfering atomic test
cases
• Standard Win7 can run 12 parallel
browsers – execution time
reduced to 1/12th
Faster Execution =
Faster Feedback
2/1/2016
23
25. Tip #5:- Rich Reports
Hyperlinks of
Page Screenshot
& HTML
Uniform logging
of TC steps in
Green/Red/Black
logs
Failure Page
URL
Log bugs by just
reading logs,
even manual
tester can do!
TC
Description
2/1/2016
25
28. Tip #6:- Customized email results
• Host Test Results on a web server - Reports
history
• Email Results - Customized Subject
Pass
Percentage
Area Owner
2/1/2016
28
30. 2/1/2016
30
Tip #7:- Measure Automation Accuracy
# of test cases failed due to Dev code issue
Vs
# of test cases failed due to automation code issue
32. Tip #9:- Automated Deployments
• Using Docker - Build production like, testing
environment on the fly
• Jenkins job to:-
– Deploy application code automatically
– Run Automation
• A Step towards Continuous Delivery
2/1/2016
32
33. TestLink - TCM
Test Cases
Test Plans
Test Builds
Reports
Execution Engine
Fetch Git Code & Compile
TestLink Integration Module
Get Test Cases To Run
Launch & Drive Execution
Retrieve
Results/Exceptions
Push Test Results & Email
(ReportNG)
Layered Automation
Suite
Test Cases (TestNG)
Business Logic
Page Objects
Framework Core
Selenium Webdriver
Tip #10:- Continuous Integration
2/1/2016
33
Lets understand why we should automate at all
……
Now we understand why we should automate. But we see a lot of automation projects failing. Instead of solving these problems, they actually become a burden on the team. Why this happens??
If you this bad code, you wont be able to:-
Run your automation on diff browser,
On diff env
On slower env
This is a poorly written test case.
What if dev makes a change in element of the page, you will have to fix all test cases.
What if you want to run the same TC with different data. You can’t
What if you want to remove selenium and use xyz
What if there is change in business logic - Change in one logic, will break all tests
Moreover a poorly written automation framework will give you a non-actionable report
Imagine if you have 2000 cases running daily, out of which 500 fail daily. You will end up spending days investigating such kind of reports - High failure investigation cost due to poor reporting
Over a period of time you will be in a complete mess. Your tests will be quite brittle, and you will have high Maintenance cost.
You should design your automation framework in a layered fashion, starting with the test cases which use…and lastly the automation libs.
For any new feature automation, this will usually be the starting point, where you will add classes for all the new pages introduced in the feature.
Now if developer changes any page, there will be only one place where you need to fix. You don’t need to make any change in test cases.
Your application will have multiple pages, once you have created page objects for all of them, start combining them into business logic.
These are reusable functions, which others can also use when they want to use this feature. Also, these will make your test case body neat and clean.
The logical unit, way PM or manual tester will describe
Extensively use reusable Business Logic Helpers, so that test case body looks short and neat. Just by reading the names of the helper methods being called, the objective of manual test case should become clear.
And if business logic of lets say doing a booking changes, I don’t need any change in my test case.
Wrapper around various external libraries and utility methods, so that writing automation code is easy.
Talk only about Browser & Element
Eg. Here it is reading login credentials from excel
You just add more rows in excel, and your TC will run for those.
No need to write code for every data input.
Very important aim of automation is faster feedback, so it should execute fast.
You must reduce the failure investigation time, when you have large number of tests.
This is a very nice reporting solution by Allure, you can integrate your automation reports with this.
Let’s say you have 2000 TC, out of which 500 fail. This will group your failure reasons and show you the top defects.
- Also you can do cleanup work like – closing browser, closing db connection, recording screenshot, html, firebug– test case writer does not needs to worry about it
Also – you can use softassert, and then assertall in listener. Test case will not stop in between.
Reduce failure investigation time
Instrument dev code and run automation
It will capture the lines of dev code that are executed, and by which test case
It has 3 major parts, TestLink where we keep manual cases, Execution engine to read those cases, and layered automation suite where all test cases are automated.
Jenkins is a Continuous Integration tool, which we use to build & schedule automation. We can create job for the same.