5 Memorable Presentations from Agile Testing Days 2017

This year I had the opportunity to visit Agile Testing Days 2017 in Potsdam, Germany which is considered as one of the greatest testing conferences in Europe intended for Agile Software Testing Community (basically for anyone who works in Agile team and is responsible for the Testing and Quality of the Software). This was the 9th Agile Testing Days and seems like the community grows each year (this year there were more than 700 attendees).

It was a four full day conference, and tutorial day on day one, with more than 160 speakers. Does it sound like much? It was. There were up to eight parallel track sessions, so you can imagine the challenge deciding which talks and workshops to attend. There were great experience stories and talks about testing, test automation and agile future as well as best practices and common mistakes to avoid. I would like to share some interesting presentations as they provide some new viewpoints on software testing and new thoughts about how to improve the quality of the software in agile development teams.

Regression Testing or No More Regression Testing (Carlo van Driel & Marc van den Berg)

Carlo van Driel has been working in IT since 1991 and he is the CEO of “De Agile Testers” consultancy that specialises in agile testing. His colleague Marc is an agile coach, Scrum master and agile tester with more than 10 years of experience.

In this presentation they talked about a well known testing approach – regression testing. Every tester in an agile team would agree that regression tests are very time consuming and expensive, especially if automated. Authors mentioned that by crossing out regression tests of their software testing life cycles they save time and still can deliver quality software . They shared some principles to get confidence that the newly created features and bug fixes are well tested and will not cause any new issue:

• Unit test everything. Each unit of the code should be tested and validated, so testers can perform the tests on well-written code. Unit tests can be executed in seconds, so it also gives fast feedback to developer.
• Look at the code and know what you test. Tester should completely understand what they test. Dig into the code or ask the developer what has changed and see how the changes affect the software.
• Exploratory testing. Well performed exploratory tests by experienced tester who knows which areas should be covered, can be the key to finding the most bugs.

A lot of teams would be happy to remove regression tests, but it might not be so easy in every project especially when confidence of software quality is based on regression tests. Being aware of it. There were some suggestions to take a half-step toward no regression testing strategy:

• Create small test suits aimed to cover scenarios that are critical to the company. Find a new name for these suits (I would prefer Functional test, Feature or Component test) and don’t call them regression tests. Consider the risks and perform tests when some feature could be actually affected by the changes. Think before testing and decide what tests to perform.
• Automate these tests and use variable test data – change the data when performing tests, do not rely on the same input over and over. This way you can spend more time on exploratory testing.

The Pothole of Automating Too Much (Paul Holland)

Paul Holland is Senior Director of Test Engineering at “Medidata Solutions”, in New York City. With over 20 years experience in software testing Paul now is a software testing consultant, teacher and one of four instructors for Rapid Software Testing course.

As Paul reveals in the title of his keynote, there might be some potholes when deciding to automate tests. In his presentation he talked about challenges of UI test automation. It’s important to do the automation the right way so maintenance of automation doesn’t become ineffective and expensive. Paul came up with some solutions to this problem so automation can be done in a smart way:

• Automate critical paths and paths with highest use. Trying to cover all areas,  easily can become slow and ineffective. Take into account how many paths (tests) are in your software, there will always be more checks to automate than tests that can be written. Cover the scenarios that are sensitive for the product to keep working.
• Do not write automation for all bugs found. If the bug is reproducible, I would consider automating tests for this feature. It’s important to always keep your automated tests simple and easy to maintain. Automate what really makes sense.
• Consider the cost of automation vs. benefit. By costs I mean difficulty to create, maintain and analyse failures. You have to consider how often the changes can take place in specific area and the difficulty to automate specific scenarios. It’s important to always investigate failed tests, if they are not investigated, then why run them at all? It can  show you software issue that occurs early, or there could be issue with unstable tests.

No More Flaky Tests (Emanuil Slavov)

Emanuil is Head of QA at Komfo. With more than 15 years of experience in software quality, he directs cross-functional teams of technical experts to analyse systems and processes, creating and mentoring on applications development in large-scale infrastructures.

Emanuil revealed system and practical techniques on how to fix unreliable automation tests that can help to considerably improve test stability so that they can be real part of CI/CD pipeline. Note that Komfo’s flaky test rate is 0,15% comparing to Google’s 1,5%.

Most tests are flaky when written the first time and unstable tests are worse than no tests at all. So it’s important to detect them fast enough and immediately investigate. That should be the first task of the day for the tester – investigate failed tests, fix them and consider deleting the test if it cannot be fixed.

There are too many moving parts that can affect tests as UI is made for human consumption.

Some basic fixes to do:

• Don’t use record and playback tools.
• Don’t use XPath locators
• Use dynamic wait for UI elements and wait for all calls completion.
• Support UI tests with API calls to have minimal interactions in the UI.
• Avoid of navigation or setup in UI, instead insert test data directly with API or in DB.
• Skipping login will also save a lot of time, use tokens in the URL, cookie, browser plugins.

Usually unstable tests are in UI or API level but not in unit tests. Lack of API documentation can force testing only through the UI, but most of the business logic is not in the UI, so it’s important to test at the right level and every part independently. Well written API tests can cover most of the critical parts, and if you still need UI tests, they will be more stable and you can rely on already performed system tests.

A Practical Guide to Testing in DevOps (Katrina Clokie)

Katrina Clokie is a test practice manager to a team of around 100 testers located in New Zealand. Katrina is author of a book called A Practical Guide to Testing in DevOps, founder and editor of a magazine called Testing Trapeze, co-founder of the testing community WeTest New Zealand, as well as international conference speaker, frequent blogger and tweeter.

In this session Katrina shared a few thoughts from her book, such as awareness of common DevOps practices for testing and understanding how and why DevOps might alter test strategy. In most of DevOps literature, testing is not covered, so it might be hard to understand tester’s role in the world of DevOps.

As it turns out, not only DevOps has changed the way we are delivering software, but also testing has evolved and tends to be continuous. Katrina uses Dan Ashby’s illustration showing that testing can happen at every step of production.

Testing occurs everywhere though you don’t see it, so it makes sense that it could be challenging to identify a tester’s role. Here Katrina shows test pyramid by Noah Sussman.

In her presentation she flipped the pyramid and described it  as a bug filter where each level is separate sieve that catches different sizes of bugs. Although a unit test is the largest level, each filter can catch different kind of bugs as the coverage differs between levels. Left side is covered by bug filter, the other side – operations. When software has been released, testing in production takes place. Supplying bug filter with 3 more testing levels – Alerting, Monitoring and Logging. It’s important to see what’s going on with released code and have several methods to detect any warnings or issues.

DevOps makes handovers more visible, especially when proper testing has been done, and it will reduce headache of detecting and solving particular issues.

Owning Our Narrative (Angie Jones)

Angie Jones is an experienced automation engineer who advises multiple agile teams on automation strategies and has developed automation frameworks for countless software products. She is creator of more than 20 patented inventions in the areas of collaboration software, social networking, virtual worlds, smarter planet, and software development processes. International public speaker who teaches workshops and gives conference talks on agile automation strategies.

Angie started her keynote with music tech history. As it turns out musicians and their history has a lot in common with current day testers. Each time new technologies (radio, jukebox, Walkman) were introduced in music, musicians were afraid to lose revenue. Any resistance of these changes failed, so they used all these new technologies instead and learned how to earn using them. In the end a musician’s role has remained the same despite of all the challenges.

Similarly with testers and the wide set of different technology and automation tools available. Angie says: “Our tools can’t always shift with us.” Especially in testing, you have to be clever enough to continuously learn new techniques and familiarize with new tools. Don’t stagnate and resist new changes, instead gather the information and embrace them, so you can learn how to use all of that in your favour.

That was a short insight what brought me new thoughts and good emotions after this conference, there were several truly inspired keynotes and useful workshops all around testing and agile teams. Unfortunately I couldn’t attend each talk or workshop I wanted, but organisers promised that all slides will be available in one place shortly after the conference. Just check conference’s web page for presentations.

Looking forward to the 10th “Agile Testing Days”

This was my first time visiting Agile Testing Days and now I know why it’s also called as testing festival. The organisers did a great work there so attendees can fully enjoy the day. Starting with some early morning events like yoga, morning run to the nearby park or some discussion sessions at the coffee to get your mind in the right mood. Although the days where already fulfilled with talks and workshops, there was that Agile atmosphere where attendees gathered between talks and discussed presented topics a lot rather than just grabbed a coffee and ran to the next session. Even at the end of each day organisers made sure that we don’t have time to go to sleep early. Instead they welcomed us to participate in Super Hero Party, Networking and Collaboration events, which lead to meeting new people. I totally enjoyed every day of this conference!

Next year “Agile Testing Days” is going to have the 10th anniversary, so this conference should become a must visit for anyone involved in software development and agile community, especially Testers, QA Managers, Project Manager and Automation engineers. I am looking forward to the next year’s conference! As “Agile Testing Days” motto says – Always be yourself, unless you can be unicorn. Then always be unicorn!