TWISummit 2019 - Take the Pain out of Browser Automation!
•
5 recomendaciones•970 vistas
Are you ready to challenge the existing beliefs centered around browser automation? Explore Taiko - a free and open source library that has a lean learning curve, with Easy APIs to quickly automate web applications and reduce test flakiness with a prerequisite of NodeJS.
Recomendados
Recomendados
Más contenido relacionado
Similar a TWISummit 2019 - Take the Pain out of Browser Automation!
Similar a TWISummit 2019 - Take the Pain out of Browser Automation! (20)
Más de Thoughtworks
Más de Thoughtworks (20)
Último
Último (20)
TWISummit 2019 - Take the Pain out of Browser Automation!
- 2. “Software testing is a process, or a series of processes, designed to make sure computer code does what it was designed to do and that it does not do anything unintended.” - Glenford J Myers ©ThoughtWorks 2019
- 4. “Are you ready to challenge your existing beliefs about test automation?” ©ThoughtWorks 2019
- 5. taiko.gauge.org End to end browser tests Problems associated with UI tests: ● Cost of maintenance is high ● Long time to run. (around 4-12 hours) ● Considered Flaky ● Long feedback cycles ©ThoughtWorks 2019
- 6. UI Tests Service Tests Unit Tests Unit tests are important but they aren’t tests, they are a part of the development process! ©ThoughtWorks 2019
- 8. taiko.gauge.org Our findings! Maintenance Test flakiness Learning curve ©ThoughtWorks 2019
- 9. TODO App System under test ©ThoughtWorks 2019
- 14. taiko.gauge.org Lets run some tests with Selenium To Understand the problems with Maintenance ©ThoughtWorks 2019
- 16. taiko.gauge.org { await driver.get('http://todomvc.com/examples/angularjs/#/'); ... var newElements = await driver.findElements(By.xpath('//ul[@class="todo-list"]/li[@class=""]')); assert.ok(newElements.length == 1); ... } What went wrong? Did the Functionality change? Error: AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value: assert.ok(newElements.length==1) Did the DOM change?Test can’t find the required css ©ThoughtWorks 2019
- 17. taiko.gauge.org Pros and Cons of using Xpath or Css style selectors in tests We have always been doing that isn’t it? ©ThoughtWorks 2019
- 18. “Maintaining locators must be calculated as part of test maintenance cost.” Zac Campbell (Mozilla Test Engineering Blog) https://blog.mozilla.org/fxtesteng/2013/09/26/writing-reliable-locators-for-selenium-and-webdriver-tests/ But then, what about Modern Web frameworks? ©ThoughtWorks 2019
- 19. taiko.gauge.org Lets Understand how flakiness affects your Tests! ©ThoughtWorks 2019
- 21. taiko.gauge.org What went wrong? ()=> { ... await driver.get('http://todomvc.com/examples/react/#/'); await driver.sleep(300); await driver.findElement(By.css('input[placeholder="What needs to be done?")) .sendKeys('Automate web page'); ... } Error: NoSuchElementError: no such element: Unable to locate element: {"method":"css selector","selector":"input[placeholder="What needs to be done?"]"} Test can’t find the elementExplicit Wait... in test code ©ThoughtWorks 2019
- 22. taiko.gauge.org Does this solve the issue? ()=> { ... await driver.sleep(300); await driver.wait(until.elementLocated( By.css('input[placeholder="What needs to be done?"]'))); await driver.wait(until.elementIsVisible(driver.findElement( By.css('input[placeholder="What needs to be done?"]')))); await driver.findElement(By.css('input[placeholder="What needs to be done?")) .sendKeys('Automate web page'); ... } ©ThoughtWorks 2019
- 23. “Almost 16% of the our tests have some flakiness associated with them! This is a staggering number; it means that more than 1 in 7 of the tests written by our world-class engineers occasionally fail in a way not caused by changes to the code or tests” - John Micco (Google Testing Blog https://testing.googleblog.com/2016/05/flaky- tests-at-google-and-how-we.html?m=1) ©ThoughtWorks 2019
- 24. taiko.gauge.org Page Object Pattern Advanced Page Object Pattern Facade Design Pattern Singleton Design Pattern Fluent Page Object Pattern IoC Container and Page Object Strategy Design Pattern Advanced Strategy Design Pattern Observer Design Pattern Observer Design Pattern via Events and Delegate Observer Design Pattern via IObservable and IObserver Learning curves ©ThoughtWorks 2019
- 25. “Testers spend 50% of their time designing and maintaining test frameworks” ©ThoughtWorks 2019
- 26. taiko.gauge.org Understanding the problem! Auto generated code using recorders ©ThoughtWorks 2019
- 27. It’s time we stopped looking for workarounds! Our take on these pain points ©ThoughtWorks 2019
- 28. taiko.gauge.org The different approach! Low learning curve Eases the complex of the required test code Allows to automate modern Web applications quickly and Reduces test flakiness with Nifty APIs What is needed is a tool which has ©ThoughtWorks 2019
- 30. taiko.gauge.org Smart Selectors async()=> { ... await click("Active"); ... } ©ThoughtWorks 2019
- 31. taiko.gauge.org Smart Selectors async()=> { ... await write("Tech Radar"))); ... } ©ThoughtWorks 2019
- 32. taiko.gauge.org Proximity Selectors async()=> { ... await click(checkBox(near("Tech Radar"))); ... } ©ThoughtWorks 2019
- 33. taiko.gauge.org Ease of use Taiko REPL ©ThoughtWorks 2019
- 35. taiko.gauge.org Lets try to automate the same tests with Taiko! ©ThoughtWorks 2019
- 37. taiko.gauge.org Implicit waits async()=> { await openBrowser({headless:false}); await goto('http://todomvc.com/examples/react/#/'); await write('Tech Radar',into(textBox({placeholder:"What needs to be done?"}))); ... await closeBrowser(); } ©ThoughtWorks 2019
- 38. taiko.gauge.org Putting all together async()=> { await openBrowser({headless:false}); await goto('http://todomvc.com/examples/react/#/'); await write('Automate web page', into(inputField({ placeholder: "What needs to be done?" }))); await press('Enter'); assert.ok(await text('1 item left').exists()); await click(checkBox(near('Automate web page'))); assert.ok(await text('0 items left').exists()); await closeBrowser(); } With smart selectors and implicit waits ©ThoughtWorks 2019
- 39. taiko.gauge.org Other Features of Taiko Network Interceptors are built in eg: intercept("https://fetchdata.com", {"test": data }) Mocking and stubbing Extensible using Plugins Taiko Diagnostics plugin Taiko Screencast plugin Taiko Android plugin ©ThoughtWorks 2019
- 40. taiko.gauge.org Integration with Test Frameworks + ©ThoughtWorks 2019
- 42. taiko.gauge.org Summary The different approach! At ThoughtWorks testing plays a big role while delivering projects. We keep learning from our projects to improve and create better tools. We write a lot about it, create a lot of open source testing tools to make testing easier. Automate tests quickly Reduce associated maintenance cost by using smart selectors Keep test code clean and reduce flakiness by letting taiko handle waits for you A NodeJS library offers all you need to write End-end tests. Explore more and let us know your feedback! ©ThoughtWorks 2019
- 43. taiko.gauge.org Maintaining locators must be calculated as part of the cost of test maintenance. - https://blog.mozilla.org/fxtesteng/2013/09/26/writing-reliable-locators-for-selenium- and-webdriver-tests/ Almost 16% of our tests have some level of flakiness associated with them! - https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html?m=1 A whooping 44% automation checks are done using Java - https://applitools.com/blog/language-software-test-automation Unit tests aren’t tests - https://gauge.org/2019/03/15/unit-tests-arent-tests/ Try Gauge and Taiko - https://gauge.org/gauge-taiko Gauge and Taiko Blogs - https://gauge.org/blog/ Gauge and Taiko Gitter Channel - https://gitter.im/getgauge/chat Gauge and Taiko Google group - getgauge@googlegroups.com References ©ThoughtWorks 2019
- 44. THANK YOU! #TWISummit Please share your feedback with nehashrl@thoughtworks.com sswaroop@thoughtworks.com ©ThoughtWorks 2019
Notas del editor
- [Soumya] So basically it’s a process or a series of processes to ensure code works as intended and NOT DO ANYTHING UNINTENDED
- [SOUMYA] At ThoughtWorks; testing, plays a very big role while delivering projects. Infact, we keep learning from our projects. It helps us improve and create better tools to make testing easier. ONE SUCH PRODUCT IS GAUGE. [Soumya] - I am the Lead QA on it [Neha] - and I am the Tech Lead on it Our goal is to provide Reliable Test Automation tools. Before we begin
- [Neha] ARE YOU READY TO CHALLENGE YOUR EXISTING BELIEFS ABOUT TEST AUTOMATION! We have seen that the Test Pyramid has been a common reference point for teams to structure their test suites
- [Neha] It is believed that there should be fewer UI tests and more of unit and service tests. Even the test pyramid clearly shows us that UI tests are costly and to have long running times. It’s become a common practice not invest in UI Tests and concentrate on Unit tests instead. This is because UI tests are often considered difficult to write and even more difficult to maintain.
- [NEHA] But, unit tests aren’t really tests! They are part of development process. After all, the developers are the ones who write these tests.
- [Neha] We have also faced a lot of problems while writing UI tests! We conducted a small survey. And here are our findings
- [Neha] The common pain points were Maintenance, Flakiness in tests and learning curve associated with the UI test tools
- [Soumya] Since we want to understand problems of test automation with examples. We will need a system under test, For that, we chose a simple TODO app! This is used to manage todo tasks. But today for the Demo we will only add a task and mark it as complete
- [Soumya] As you see, the App here is written in React JS.
- [Soumya] Here u can write the name of the task and press Enter to add it
- [Soumya] And then click on the checkbox to mark it as complete. Simple enough?
- [Soumya] The reason we chose this APP is because it is written in a bunch of UI frameworks with exactly the same functionality. For this demo, we will be using the React and AngularJS flavours of the APP.
- [Soumya] To understand the problems with Maintenance, lets automate todo App with Selenium. WHY SELENIUM: ‘Cause it's been around for ages and also been popular for a long time too. How many of you have used selenium before. Great! So, you will be able to relate to the examples we will be giving now!
- [SOUMYA] Let’s begin, this is the URL of our todo APP, the React flavour. Of Course SELENIUM selectors are dependent on the HTML structure When we run against the React flavour of our todo app. The tests pass. Obviously it was written specifically for it. I mean mainly the selectors. But now, lets see what happens when the same test is run against the AngularJS flavour of the application. The test failed.
- [SOUMYA] What went wrong? [CLICK] Why are the tests failing? Did the functionality change? No, [CLICK] Did the DOM change Yes [CLICK] And now! the test doesn’t seem to have the required CSS class.
- [SOUMYA] Let’s talk about some pros and cons of using selectors Xpath or Css selectors in tests. Yes, these are very powerful. They can select any element on your HTML page. But if you are going to use it in tests, we have to get used to updating them every time the HTML structure changes. Be it design change, streamlining of HTML or fix bugs. And this will be ongoing as long as your web app is evolving. [CLICK] We have been doing that isn’t it. We have got used to it also. So much so that
- [Soumya] We account for Maintaining locators in the test maintenance cost. As quoted in a Mozilla Test blog. [CLICK] But then, what about Modern Web frameworks? They auto-generate the HTML structure. It will perhaps take more effort to locate elements with Css or XPath selectors! [Next - Neha]
- [Neha] What else adds to your cost? Flakiness? Lets understand how flakiness affects your tests.
- [Neha] Here, the we have successfully run tests once. Does rerunning the test without any change give the same result? The tests have failed.
- [NEHA] What went wrong? Is the element to interact with available after a page load or asynchronous call without any waits? Perhaps Not [CLICK] Will introducing delays with magic numbers help? We saw otherwise [CLICK] Shall try using wait for conditions instead? [CLICK]
- [NEHA] Like so This looks complicated isn’t, what should be your focus be; testing the application or focus on the nitty gritties of the tool being used for automation!
- [Neha] Google Testing blogs claim that more than 1 in 7 of their tests occasionally fail in a way not caused by changes to the code or tests. Flaky tests are at least twice as harmful as we think. Why? Because - We not only waste time re-running flaky tests but also real test failures which we assume to flake out. This causes the team to lose their confidence in their test suite.
- [Soumya] After seeing maintenance and flakiness issues, you may think that learning curve is the least of your problems. But, having to write waits described earlier; or having to structure your test code with all the DOM selectors is overwhelming! You not only have to design your production code, but also your test code.
- [Soumya] Our observation with test projects has been that Tester’s spend close to 50% of the time designing and maintaining test frameworks. What if there were recorders to generate the required code! Then would we be spending less time and effort to maintain tests?.
- [NEHA] Not really! There are a lot tools which try to make writing tests easier by recording flow and generating the code. But, Be it auto generated code or handwritten, you see; the problems associated with DOM selectors doesn’t change. So, It’s but natural that there is a general complaint with code generated using such recorders are brittle and difficult to maintain. [Next Soumya]
- [Soumya] We have spoken so much about the problems. You know what this shows? It shows that over the period of time we have learned to live with the associated pain points and just been looking for work arounds for it. [CLICK] Sometimes all it takes is a different approach!
- [Soumya] What is needed today is a tool which has a low learning curve, Something that eases the complexity of the required test code Allows you to automate modern web applications quickly Takes care of reducing test flakiness With some nifty APIs
- [Neha] Yes, that’s where Taiko adds value! A tool that can be used to automate modern web applications easily. It is the tools aim to reduce the flakiness and help write more maintainable code! Let me give you a little insight about how Taiko was built. It is built over Chrome DevTools Protocol, the same Protocol used by the chrome console. So what does this mean for a team developing Taiko. It gives us a handle over the low level protocols used by the Chrome Browser. This means, better and direct control over the browser. So, how does this help you? Taiko is able to figure out low level details like when page loads are complete. So you don’t have to handle these in your tests anymore.
- [Soumya] In Taiko, the selectors are smart. It has simple APIs to automate what is visible on the screen. For example `click(“Active”)`. Clicks Active. The HTML tag associated with it doesn’t have to be specified
- [Soumya] If a textbox has focus, you can write into it using this simple API
- [Soumya] But; not all elements have text. What we can do in such cases is select elements based on their proximity to other elements. Here is an example where the checkbox near TechRadar is clicked. [Next - Neha]
- [Neha] As you have seen before, the selectors API’s are simple and can be used easily to give instructions for you to perform the action on the browser. You can continue to give instructions to complete the workflow. You can then create a script out of the successful actions performed. So you can think like a user to automate your workflow. It helps you in doing black box testing.
- [Neha] As you have seen before, the selectors API’s are simple and can be used easily to give instructions for the user to perform the action on the browser. The user can continue to give instructions to complete the workflow. The user can then create a script out of the successful actions performed. This along with our smart selectors, Taiko helps you think like a user to automate your workflow. It helps you in doing black box testing.
- [Soumya] Let’s take the same examples we used earlier and run the tests with Taiko. Let’s see what happens.
- [Soumya] First we run the test with the ReactJS flavor of the APP. As you can see, the smart selectors are not dependent on the HTML structure. Yes, the tests passed. But let’s re-run it. To check if this was a false positive. It seems to work fine even without explicit waits defined in test code. Now, let’s modify the application URL to point to the angularjs flavor of the app. Here the tests passed in both cases. In this case, with no changes in the workflow test code didn’t change. Mention fall back options - DOM selector [Next - Neha]
- [Neha] Taiko has reliable inbuilt wait mechanisms. This means, Starting from opening a browser to performing other tasks, in most cases there are no explicit waits needed.
- [Neha] Putting it all together, Taiko lets you focus on writing reliable tests and reduce the cost associated with test maintenance.
- [Neha] Taiko comes with a lot of other features. Like Network interceptors are built in. No more using a proxy server to stub that annoying 3rd party payment gateway service or the like. You can also record a video of your entire workflow using the Screencast plugin.
- [Neha] Since Taiko is a simple NodeJS library, you can integrate it with any test framework like Gauge or Mocha or Jest easily. However, you can find your testing experience simplified using Gauge and Taiko. TESTING EXPERIENCE SIMPLIFIED USING GAUGE + TAIKO
- [Soumya] There are a lot of free and open source tools to automate the browser and write end to end tests. Last year the Gauge team released Taiko. You may think it is yet another one in the market. But, in this blog, we will discuss what’s different about it. Here we compared Taiko with some popular browser testing tools. There will be always pros and cons with each tool you use. If you are looking for something with cross browser support, for instance, selenium would be a good fit. If you are looking to reliably automate modern web applications with low maintenance cost, it's worth considering Taiko.
- [Neha] ----- If you have some questions about Taiko you can always reach us out on Gitter for a quick chat, Google groups for discussion or the Stack overflow.
- Some references here