In software development, testing is an integral part of the development process. Tests are used to verify that software behaves as expected, and they ensure that changes made to the code do not break existing functionality. However, there is a problem that often arises when running automated tests: Flaky tests.
Flaky tests are tests that produce different results each time they are run, even when the code being tested has not changed. This can happen for a variety of reasons, such as race conditions, non-deterministic behaviour, or environmental factors. Flaky tests can be frustrating and time-consuming to deal with, and they can undermine confidence in the testing process.
The goal of bringing flaky test management in-product is to reduce developer pain cause by flaky tests and cater to the whole workflow. Azure DevOps provides 2 types of flaky test detection settings: System detection and Custom detection. Read more about them in our official documentation.
Issue with previous experience:
Users can mark or unmark a test as flaky based on analysis or context and this tagging is stored in our database. Users can also change the setting from System detection to Custom detection and vice-versa. However, all the tests that had been previously marked as flaky under system detection automatically get marked as flaky under custom detection as well. This can lead to issues as incorrect test cases have been marked as flaky.
New experience:
Going forward, a setting change in Flaky test detection removes the list of test cases that had been previously marked as flaky. Hence, if a user changes Flaky test management setting, they will have to mark all test cases as flaky again. A confirmation alert has also been added to inform users of the impact of the setting change.
Conclusion:
Azure DevOps now provides the ability to delete past history of flaky test cases on changing the settings of it’s Flaky Test management feature so that developers can improve the reliability of their automated tests
Quick demo of feature:
Step 1: Goto Project Settings -> Test Management. Step 2: Make sure ‘Flaky Test Detection’ is turned on. Step 3: Select between ‘System Detection’ or ‘Custom Detection’. Step 4: The confirmation dialog alerts users regarding deletion on existing tests that have been marked as flaky.
