Diving into the World of Test Automation The Approach and the Technologies
•
3 likes•1,556 views
This presentation was originally given at Quality Jam London. Elise covered test automation and the progression for test automation that you might encounter. The session agenda included: The stages of the test team Why are we automating? What are we automating? How are we automating? What languages should we use? What frameworks and libraries should we use? Open source or proprietary? Learn more at www.qualityjam.com
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Diving into the World of Test Automation The Approach and the Technologies
Similar to Diving into the World of Test Automation The Approach and the Technologies (20)
More from QASymphony
More from QASymphony (16)
Recently uploaded
Recently uploaded (20)
Diving into the World of Test Automation The Approach and the Technologies
- 1. Diving into the World of Test Automation The Approach and the Technologies ELISE CARMICHAEL | OCTOBER 11, 2017
- 2. Agenda The stages of the test team Why are we automating? What are we automating? How are we automating? • What languages should we use? • What frameworks and libraries should we use? • Open source or proprietary?
- 3. The stages of testing
- 4. Stage 1 – Scrappy Startup There are no separate testers Dev do all the testing along with any stakeholders or product owners No automation of any kind exists
- 5. Stage 2 – Real Product Developers (hopefully) start writing unit tests Unit tests run as part of CI and the build fails if the tests fail No separate testers, yet
- 6. Stage 3 – All The Potential Testers are hired! Testers write a test plan and start writing out manual test cases
- 7. Stage 4 – Wow, lots of users! Purchase a test case management tool – spreadsheets don’t give me traceability and they’re a pain to update Test Engineers automate functional UI and API tests Testing augmented with exploratory testing Load testing becomes standard
- 8. Stage 5 – Professional/Enterprise All smoke tests and regression tests are automated Automated, integrated code coverage tests Automated code quality scans Testers spend their time performing risk analysis, determining what to automate, and performing exploratory tests or employing BDD for new feature testing Repeatable processes for load testing, performance testing, stress testing
- 9. Stage 6 – Industry Leaders Production monitoring for all layers including the application layer Testing blue/green deployment and release processes with automation Automated testing of third party integration points
- 10. Why are we automating?
- 11. But Why? It’s less expensive to find bugs earlier in the release cycle Push towards DevOps - Continuous or more frequent Delivery Gives the test team the time to perform more intelligent testing
- 12. Bug found in testing Bug found in Production Bug found in development Bug found in UAT Cost of Failures
- 14. What are we testing?
- 15. Infrastructure Deployment Application Production Monitoring Load & Performance & Stress Static Code Analysis API Functionality & Security Client App Functionality Unit Tests Application Security Tests to Automate
- 16. Infrastructure Deployment Application Production Monitoring Load & Performance & Stress Static Code Analysis API Functionality & Security Client App Functionality Unit Tests Application Security Starting Point
- 18. Application Testing - Options Proprietary or open source? Programming language Test frameworks & design pattern Assertion library Where will you execute the tests Reporting Infrastructur e Deployment Application Production Monitoring Load & Performance & Stress Static Code Analysis API Functionality & Security Client App Functionality Unit Tests Application Security
- 19. Proprietary or Open Source? Open Source Free Community for support Transparency Flexibility Better security?! Proprietary Ownership Stability Innovative Usability Enterprise Support with SLAs
- 20. Programming Language Options Use what makes sense for the programming language the code was written in Use what makes sense for the test team Use the technology the devs know so they can help out with technology challenges Use an intelligent point & click recorder and record something that can be run later
- 21. Testing Frameworks Homegrown Modular/Object Oriented – usually based on a common unit test library BDD style – supports given/when/then format Record & playback – Free or Proprietary
- 22. Design Patterns Page object model Data driven tests Keyword driven tests Selenium BDD Data Driven
- 23. Assertion Libraries / Styles Basic • Assert(x == y) • Assert.equal(x, y) Chainable (usually BDD style) • Expect(foo).to.have.lengthOf(3) • Expect(bar).to.equal(“some string”) • Should.not.exist(foo)
- 24. Where to execute the tests Local test machine • Only use this for testing the tests! Dedicated test machines • Simulate your different user environments that you want to regularly run your smoke tests on • Integrate with CI or run on a separate test schedule 3rd party • Great, but expensive!
- 25. QTest Automation Hub - coming soon! Manage all your test machine automation agents from one centralized location Schedule automation executions from qTest Manager against all of your test configurations More native integrations with popular automation frameworks
- 26. Reporting Document what you want to test, first Automate as much as possible Provide visibility into what’s automated, what’s not, and what should be Correlate that data to number of production bugs or other KPIs that prove your team’s effectiveness • Speed to market increase • # of bugs caught in test vs production • # bugs caught vs # of tests written
- 27. API Testing
- 28. Type of APIs Your own public API Your web services that provide content to your website Microservices that act as middle layers 3rd party APIs that you consume in your application Infrastructur e Deployment Application Production Monitoring Load & Performance & Stress Static Code Analysis API Functionality & Security Client App Functionality Unit Tests Application Security
- 29. API Automation Automate ALL the API tests! • These are the fastest tests you can automate as well as the highest risk. • Continue to do exploratory testing and really test some edge cases. Execute these as part of your continuous integration suite. • When a dev modifies the services code, rerun all the API tests. Automate tests against 3rd parties. • Make sure the endpoints you rely on are working as you expect.
- 30. API Tooling Soap API or REST API? • The tools can to be specific to one. Some do both. We use Postman, Newman, and Jenkins for our API testing tool suite at QA Symphony • I recently ran a webinar about how to use these together if you want to go to the QASymphony website, you can view it there • Use the Jenkins plugin or the API to send the results back to qTest
- 32. Static Code Analysis ~20 open source code analysis tools • Many specific to a language • SonarQube is one of the most extensive ones ~15 proprietary static analysis tools • Checkmarx is a great tool that has additional emphasis on security Infrastructur e Deployment Application Production Monitoring Load & Performance & Stress Static Code Analysis API Functionality & Security Client App Functionality Unit Tests Application Security
- 33. Testing Approach
- 34. Get home and make a plan! 1. Document your tests and mark the ones you want to automate so that you can track progress 2. Automate the API tests & include them in your CI 3. Choose your application testing language/framework/libs 4. Automate the smoke test suite 5. Include smoke tests in a regularly scheduled suite 6. Introduce static analysis & security testing regularly 7. Automate regression tests 8. Continue to improve – don’t get left behind and feel vulnerable!
Editor's Notes
- Hello! SLOW & bring up our products & the hub. Get excited. My name is Elise Carmichael. I’m the VP of Quality at QA Symphony. I wanted to provide a little background on myself so you know who you are going to be listening to for the next half hour! I studied computer science and spent many years writing code and running projects in all kinds of industries from blood banking to trucking. In fact, my first program I wrote on a Commodore 64 that just printed something out out about my older brother. I joined QA symphony about 5 months ago and I help manage our development and testing processes internally and help guide our product direction which is based on industry trends and the needs of you all! Many of you are users and are already familiar with our products! If you’re not familiar with our existing automation offerings, we actually have three core ways to integrate any type of test automation with qTest Manager. I’ll mention these as we go.
- (at end) Show of hands – who has test automation in their software in any way? In general, we’re going to talk about test automation and the progression for test automation that you might encounter. Test automation can be intimidating and requires endless decision making. Even if you’re already automated testing some of your software, the methodologies are constantly changing just like software. Review Agenda: Let’s review what I consider the stages of testing at an organization Why are we automating What are we automating How are we automating Review languages/frameworks/libraries
- I came up with 6 stages to a software company and what the testing may look like at each stage. Think about where you and your team belong! I’ll ask you to raise your hand at the end.
- Ok that was the last stage. Where do you fit in? Raise your hand if your company is this stage, regardless of the testing.. Who is in “Stage 1 – scrappy startup?” Who is in “Stage 2 – a real product?” Who is in “Stage 3 – All the potential!” Who is in stage 4 – wow, lots of users!” Who is in stage 5 – professional or enterprise Who is in stage 6 – industry leaders? Great, how many of you feel like need more test automation at your company?
- Let’s talk about the technology first -
- (click through slide) Less expensive: You can find bugs when a code change is made, when the testers get their hands on the code, in UAT, or in production (or of course somewhere in the middle) If you haven’t already calculated the cost difference of a bug found in each stage, I highly recommend you do this for your business. Devops: We’re hearing a lot about Devops lately and using it as the catalyst for the push to release faster. This of course leaves testing in a bad spot because we *already* don’t have enough time to test! This paradigm shift requires automation. Test engineers, if they are separate teams, need to acquire a wider range of skills. PULSE Intelligent Testing: By automating our tests, our test team has the time to do more intelligent testing. There is SO much that can be tested and automated that being a tester is not a straight forward job. Automate what will be repeated, and let your testers have the freedom to really get deep into their role. Sessions and Explorer Google, for example, has several different testing roles and they even created the role for themselves: Site Reliability Engineers Test Engineers (think domain knowledge) Software Engineers in Test Software Engineer, Tools & Infrastructure (SETI) (in addition to SET, expanded IDEs, release testing, etc)
- If you don’t know the cost differential for finding bugs go find out. It will help when you want to make a case for hiring more test engineers! This is also a great metric to use with your KPIs – you’re finding fewer production bugs with all your automation, so how much money are you saving by having test engineers write automated tests?
- I’ve heard this at few testing talks and it stuck with me. Checking vs testing. Let me give you some dictionary definitions: checking = examine (something) in order to determine if its accurate testing = take the business logic into consideration and look at the big picture. This is where exploratory testing is useful. Is this what the user is looking for. We don’t want our testers just doing the same things over and over again each release cycle. Something about insanity… We’ll call that just checking. We should automate that stuff as much as possible. Give the testers more time to actually test with their domain knowledge and more thorough understanding of the application than anyone else!
- Where do we start automating our tests? We have quite a few options. Ideally, in a larger organization with widely used products, all of this will be included in some sort of automated way. For today, I’m going to focus on what I consider to be the foundational automated tests as well as technologies that you need to investigate for them (in the next slide). Infrastructure as code – using code to define what production and the machines look like – needs to be tested.
- I highlighted three types of testing. Client App Functionality – This is probably what most people think of when you say test automation. These are your selenium or appium type tests. We’re testing what the user experiences and cares about. Static Code Analysis – I added this because it is easy to set up in an automated way and gives tons of feedback regarding code quality immediately. This is a great bang for your buck, especially when some of the tools are free. How many of you scan your code base regularly? API Functionality & Security – I would start here. These tests are quick to write and they are straight forward. They are essentially functions – put in some input, expect the same output generally. Furthermore, this is where you will be testing your application security. You can do a lot on the client side testing to pass in bad data through the UI, but your hackers aren’t doing that; they’re testing your API for you. RELATE TO QTEST PRODUCTS
- There are the most options with application testing, so we’ll start here.
- Review slide
- Let’s do a quick pro and con between open source an proprietary: Open Source Free – can you can update it Community – fast answers or already answered items if it’s a widely used product Transparency – you know how their software is tested. You can look at the code, run your own scans. Use third parties to validate such as blackduck software. Flexibility – it doesn’t do what you want? Just add it. Security – Debatable; lots of eyes on it, no cutting corners, almost always has automated tests. I think you’d be surprised (or not) at how many companies produce software that is…less than good. Proprietary Ownership – can call someone up/submit support requests Stability – you can count on releases Innovative – that’s where people are spending money on some new idea Usability – they hire designers (Jmeter!?) Support – pay enough, and get more in return
- Use what the code was written in: - If you have angular code, protractor is the defacto testing tool therefore you’re using javascript Do what the test team wants: - If all your testers really want to use ruby, use ruby. You want passionate people Do what the devs know: - If all development is in java, use java so they can get free internal training and help with some of the difficult tech items that come up Something new: - If the team doesn’t feel comfortable with software, a modern, record play back tool might be a good choice. What you’ll want to verify is that it gives you the code, uses good design patterns and reusable code principles so that you can version the software. Also, you’ll want this to use/write software that is open source so you’re not stuck with a tool in case it becomes deprecated.
- You’re probably going to use something behind the scenes like selenium or Appium or some other executor behind the scenes, but on top, you need an engine that runs your test. This is the test framework that is the engine that executes your tests. There are dozens – many in every language. Open: https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks (Click through) – 4 main options Homegrown: - Your own proprietary way to run tests and to view results. This is pretty self explanatory. Modular/object oriented: - Page Object Model - This is very popular and endlessly extensible. As a technical person, this is my preference. You can abstract details away so testers perhaps just use the high level functions to write tests. BDD – behavior driven development – more of a process: - This allows tests to be written at a high level by your BA or less technical testers. Behind the scenes, there are many BDD libraries and frameworks. A developer or test engineer will still need to write code but this allows someone less technical to write requirements as high level tests allowing you to have a better flow of communication between your BAs and testers. Record and playback - These can be great especially if your testing team is less technical. However it can also be limited and can make you reliant on a tool to keep up with new browser standards. Proprietary tools will give you the ability to have a company release updates regularly, but then you are also dependent on that company. There are fewer open source versions of this, but there are some that create open source code for you so you’re not reliant on the tool if you don’t want to use the tool anymore.
- Page Object The ideas is you use some standard development designs for organizing your application Each screen would be a class with all of it’s own functionality The screen object (or pageobject) will know where all of it’s own elements are and what the text should be (LOGIN PAGE EXAMPLE) It can also perform actions on it’s screen – like clicking a button All the tests call these functions within the screen This allows your tests to be extremely robust. You’ll always need to update the scripts when new features are added or the screens shift around; now you can just update this in one place. Data Driven Tests Test data is not hard coded – it needs to be in a separate file or database and pulled into the tests. Keyword driven tests Keyword driven tests allow the tester to write/execute parts of code without knowing how to write code. This is a layer on top of the code. This abstracts more of the testing away. All these can be used in conjunction with each other – look at what outcome you want and design it based on the way you want the tests to be written.
- Your assertions are the actual testing line in an automated test! You’re verifying a value is something, is not something, is greater than something, or exists. Like frameworks, there are many assertion libraries for each language, and often a built in one. Let your test engineer select what they want to use here as both are fairly straight forward.
- Local test machines Great for writing and testing that the tests work. It’s not a controlled environment so I don’t recommend this e a great place unless you’re in the first stage of writing test automation (the Wow, lots of users phase!) Dedicated test machines This is generally the most cost effective unless you have a LOT of configurations that you don’t use often 3rd party - These are fantastic choices because the reporting is often built in. The only drawbacks are the price generally. They tend to bill by the minute or credit and it adds up quickly. My recommendation here is to actually script the test machines so they can be stood up or torn down just when you use them. Of course this can take a lot of setup effort and may require maintenance. It will likely be the most inexpensive solution over time.
- This is not the final name, but our internal code name I’m sharing with you today. You’ve all used the automation agent software, I’m sure. If you haven’t, and you’re automating tests, you may want to consider it! Current state The current software allows your to install this software on your test machine or local machine. You can schedule test automation jobs against your test machine and voila, they run and results are sent back to qTest. First version looking for a release at the end of the yaer. Native framework integration – not just selenium tests, but things like API tests or load tests as well. Your full test suite from functional to could be executed from one place. Use reporting form qTest and Insights to get the full picture of the health of your application.
- Let’s review what APIs you may be using or producing
- API tests are fast to run so run them every build – use Jenkins to put the results back into qTest. JENKINS IMPROVEMENTS 3rd party: sometimes other people make mistakes. Usually an interface change will be announced and your dev team should stay on top of this, but I’ve seen it happen where something wasn’t outline in the spec specifically so there can be modifications you’re just not expecting.
- Look for native handling of Postman tests in the Automation Agent soon
- https://www.owasp.org/index.php/Static_Code_Analysis Static Code analysis tools can help highlight defects or potential issues in safety, security, portability, and reliability without having to lift a finger once a scan is run. These can be included in your CI jobs depending on the tool and is a setup once and run. This is particularly good when you get it included at the start of the project, but can be invaluable later.
- When I don’t have the majority of that pyramid covered, I have a hard time sleeping at night! Leave the conference and make a plan if you don’t already have one