We all know that automation is one of the key enablers for those on the CI-CD journey. Most teams are: • implementing automation • talking about its benefits • up-skilling themselves • talking about tooling • etc. In my experience, unfortunately most of the functional automation that is built is: • not optimal • not fit-for-purpose • does not run fast enough • gives inconsistent feedback, hence unreliable Hence, for the amount of effort invested in implementing automation, are you really getting the value from this activity? In this talk, we will discuss these challenges and why it would lead to poor ROI of automation. More importantly, we will discuss the following techniques to make automation valuable: • know the objective for the automation framework • establish criteria for tests to be automated • design your framework with proper abstraction layers • develop using appropriate design patterns

1. Does your functional automation
really add value?
@BagmarAnand
Anand Bagmar
Software Quality Evangelist

2. @BagmarAnand
About Me

3. The Silver Bullet!
@BagmarAnand

4. Automation!
@BagmarAnand

5. • Faster feedback
• Enabler of CI-CD
• Work with interesting tools / technologies
Value of Automation
@BagmarAnand

6. Be a “developer”
without the pressure of
sending code to production
@BagmarAnand

7. @BagmarAnand

8. How do you setup your test
execution machine?
@BagmarAnand
Manual setup? Scripted setup

9. Do your tests run in CI ?
@BagmarAnand
No Yes

10. If tests run in CI, do they run
automatically?
@BagmarAnand
Manual trigger Auto triggers

11. How long does it take to run
your functional tests?
@BagmarAnand
> 30 min < 30 min

12. For multiple platforms, do
you implement separate
tests?
@BagmarAnand
Duplicated tests Single test

13. What is the test passing
percentage?
@BagmarAnand
Various reasons
of failure
Failures related
to product issues

14. Do you automatically “rerun”
the failing tests?
@BagmarAnand
Yes No

15. How do you debug and get to
the root cause of failures?
@BagmarAnand
Rerun test
All info available
to debug

16. Do you think about Patterns /
Code Quality / DRY principles
/ etc?
@BagmarAnand
What patterns? Yes

17. How do you validate the UI /
UX as part of the e2e tests?
@BagmarAnand
Manual verification Intelligent tooling

18. @BagmarAnand
https://freesvg.org/img/Chaos-Order-Black.png

19. What is the way forward?
@BagmarAnand

20. Bugs still escape
@BagmarAnand
Traditional automated testing frameworks are not built for modern apps

21. Bugs still Escape
@BagmarAnand

22. @BagmarAnand

23. Bugs escape because our
approach to testing is incorrect
@BagmarAnand

24. @BagmarAnand
The Way Forward!

25. Test Automation In the world
of AI & ML
@BagmarAnand

26. @BagmarAnand
Test Automation In the world of AI & ML
https://www.infoq.com/articles/test-automation-ai-ml/

27. Unit (xUnit / JavaScript)
Manual / Exploratory
Web Service
UI / e2e
Business-facing
Tests
Technology-facing
Tests
@BagmarAnand
Test Pyramid
Slow
More integration
Expensive
Fast
More isolation
Cheap

28. Unit (xUnit / JavaScript)
Manual / Exploratory
Web
Service
UI / e2e
Business-facing
Tests
Technology-facing
Tests
@BagmarAnand
Slow
More integration
Expensive
Fast
More isolation
Cheap
Workflow
API
Contract
Consumer
Provider
Test Pyramid

29. Product Quality
@BagmarAnand

30. @BagmarAnand
Design Patterns in Automation

31. • Page-Object
• Business Layer
• Composition
• Factory Method
• Singleton -> DO NOT USE UNLESS YOU ARE VERY SURE
Commonly used Design Patterns
@BagmarAnand

32. • Page-Object
• Business Layer
• Composition
• Factory Method
• Singleton -> DO NOT USE UNLESS YOU ARE VERY SURE
Commonly used Design Patterns
@BagmarAnand

33. Architecture of e2e
Automation Framework
@BagmarAnand

34. gradle / groovy
Manage Devices
Wifi Management
Optimize Device
Utilization
Appium Server
Management
Test Specifications
Business Rules (cucumber-jvm)
Business Flows
Implement Domain
functionality
(actions, verifications, rules)
Resources
Entities / Models
Pages / Screens
Trigger e2e Tests in
Jenkins Server
- Jenkins File
- Node Management
Archive artefacts
- logs, screenshots, videos, logcat, Appium logs, etc
- Cucumber reports
‘x’ Jenkins Agents on ‘y’ Mac Minis
Helpers / Utilities
(Android / iOS) Driver utilities
Loggers
File utilities
Custom Reporter
Screenshot utilities
ADB utilities Video capture
Analytics
GPU Profiling
Command Executor
Log files from Devices
Enrich Test Execution
Report
Manage Failing Tests
Upload to TTA

35. How do you measure the
quality of your product?
@BagmarAnand

36. • Test Case Effectiveness = (Number of defects detected / Number of test cases run) x 100
• Test Case Productivity = (Number of Test Cases / Efforts Spent for Test Case Preparation)
• Test Coverage = Number of detected faults/number of predicted defects.
• Test Code Coverage = Produce code coverage by e2e / UI tests
• Requirement Coverage = (Number of requirements covered / Total number of requirements) x 100
• Test Design Coverage = (Total number of requirements mapped to test cases / Total number of requirements) x 100
• Test Execution Coverage = (Total number of executed test cases or scripts / Total number of test cases or scripts
planned to be executed) x 100
• %ge of work completed & yet to be completed
• Time to complete the remaining work
@BagmarAnand
(Traditional) Test Metrics

37. USE METRICS THAT ADD VALUE,
NOT JUST BECAUSE SOMEONE IS
ASKING FOR IT
@BagmarAnand

38. • Does your functional automation really add value?
• https://essenceoftesting.blogspot.com/2020/07/does-your-functional-
automation-really.html
• Tracking functional coverage from your api / functional UI (e2e) tests
• https://essenceoftesting.blogspot.com/2020/03/tracking-functional-
coverage.html
• Test Automation in the World of AI & ML
• https://www.infoq.com/articles/test-automation-ai-ml/
Resources
@BagmarAnand

39. @BagmarAnand
Thank you