This presentation dives into the role of test automation in modern software delivery pipelines and how the expectations on the test automation have changes over time with complex business demands and faster release cycles. And the dive in to details on how Test Automation needs to evolve into Continuous Testing approach to handle the complexities and the speed required in a modern business context.

1. Role of Test Automation In Modern
Software Delivery Pipelines
Kasun Kodagoda
Technical Lead | 99X Technology
https://kasunkodagoda.com

2. Coming Up…
• Why Test Automation is Important?
• What Should Be Automated?
• Expectation of Test Automation
• Continuous Testing
• Why Continuous Testing is Important?
• Continuous Testing vs Test Automation
• Benefits of Continuous Testing
• Key Best Practices

3. Why Test Automation is Important?
• Automated repetitive test scenarios
• More time to focus on new business cases and exploratory testing
• Can be used to assess the application load and
performance
• Executing end-to-end scenarios with many virtual users
• Can be integrated into CI/CD Pipelines
• Run automated smoke tests, load tests on the different stages of development
• Increase productivity and quality of the software product.

4. What Should be Automated?
• Unit Tests
• Smoke Tests
• API Tests
• Integration Tests
• UI Tests
• Regression Tests
• Performance Tests
• Security Tests

5. Expectation of Test Automation
• The expectation being put on Test Automation has
changed
• Business demands are more complex and require fast release cycles
• Legacy Test Automation can not keep up
• Legacy Test Automation Frameworks
• Brittle tests
• Slow End-to-End execution time
• Increased levels of False Positives
• This leads to more reliance on Manual Testing

6. The Reality
Source: Tricentis

7. Continuous Testing
The Next Evolution of Test Automation

8. Continuous Testing
Continuous testing is the process of executing automated tests
as part of the software delivery pipeline to obtain feedback on
the business risks associated with a software release candidate
as rapidly as possible.

9. Continuous Testing
• Executing automated tests as part of a software
delivery pipeline
• Get feedback on;
• Business risk associated with a release
• Fast feedback to mitigate risk
• Evolution of Test Automation
• To handle the increasing complexity of software products
• Fast delivery cycles

10. Why Continuous Testing Is Important?
• Application Architecture is getting
increasingly complex
• Microservices, Cloud, APIs creates endless combinations in a
single business scenarios
• Application Releases have become much
faster
• Applications are getting deployed multiple times per day,
even multiple times per hour
• Software is an Interface for the Business
• A software failure is a business failure. And the impact is
huge.

11. Continuous Testing vs Test Automation
• Risk
• Risk is different from Code Complete vs Analyzing Risk in
earlier stages of development
• Detect risk at the end of the development
• Risk is much higher
• Impact is much greater
• Time to react is much less
• Continuous Testing detects risk early as possible
• More time to react
• The risk and the Impact associated is much less and more
manageable

12. Continuous Testing vs Test Automation
• Breadth
• The boundaries between applications are getting blurred
• You are no longer doing your end-to-end scenarios in a
single application
• Span across multiple applications and services
• Using different technologies
• Scope of end-to-end scenarios are much wider
• Advanced Automation techniques are required
• To handle these complexities

13. Continuous Testing vs Test Automation
• Time
• Time-to-market is what matters
• Deliver value faster to the customer
• Beat the competition to become the market leader
• Development can always release software much faster
than we can test it.
• Traditional testing methods can’t keep up
• Predictor of Success
• Time-to-Market
• The Risk Associated with it.
• It’s no longer
• When the Code/Product is Ready, it’s When the Business is Ready

14. Benefits of Continuous Testing
• Assess the business risk coverage
• Instant insights into the viability of a release
candidate
• Is it too risky to progress to the next stage of the software
delivery pipeline?
• Helps preserving the quality and user
experience in an accelerated development
process
• Testing is embedded in the development
process, Not an afterthought

15. Benefits of Continuous Testing
• Seamlessly integrates with Software Delivery and
DevOps pipelines
• A shift-left approach to software quality
• Fast feedback at each stage of delivery

16. Who Is Responsible?
• The entire Engineering Team is responsible
• Shift-Left Approach to Quality is a must
• Cultural & Mindset Change is necessary

17. Key Best Practices
• Incorporate Exploratory Testing
• Identify usability issues, missing functionality
• Address functionality that automation can’t handle or too
expensive
• Risk Based Test Prioritization and Test Case Design
• Continues testing is risk based
• So identifying the business risks and prioritizing with risk in mind is
important
• Risk Coverage over Test Coverage

18. Key Best Practices
• UI and API Based Test Automation
• UI tests should complement the lower level automation
• Test Data Management Process
• Proper Test Data Management is important
• If the test data is correct, you can prevent false positives
• CI/CD Integration
• Continuous testing needs to be integrated into CI/CD
• Define acceptable baselines
• Create thresholds that automated pipelines can identify and break
the CI/CD

19. Key Best Practices
• Planning your Test Frameworks
• Reuse code, create common packages to share
• Reduce duplication, decrease maintenance
• Allows you to focus on writing tests that add value
• Continuous Learning and Improvements
• Identify key technologies that can be used to
• Reduce execution time
• Reduce scripts (or even script-less automation)
• Reduce maintenance

20. Thank You :)
Any Questions? ;)