One of the important factors to increase your test results reliability is to reduce the test flakiness, so let's start in the next few lines show you how can do that simply with an examples.

For sure the reasons behind each test failure is context depends, so I will talk here about the most general reasons and solutions which cause the flaky test results

  • Use the correct assertion command
  • Find the correct element to be asserted
  • Retry Failed Scenarios

1- Use the correct assertion command

We will discuss here two test scenarios have different assertions command


Scenario 1: Verify that Slogan is removed from website

In this case, we should assert on slogan is removed from website, so when assert using be.not.visible the test assertion failed as Cypress verifies that element is hidden via css property like display: none or visibility: hidden. While in this case, the element is not even present in DOM. That is why the assertion fails.

Now, when assert using not.exist instead of visibility assertion, the test assertion passed as the target element is not exist in DOM

Scenario 2: Verify that API doesn't return data in response body

In this case, we should ensure that response.body doesn't have data, so when assert using to.not.be.null the test assertion pass although the array is empty as Cypress verifies that response.body not equal null as string

Now, when assert using not.empty the test assertion fail as expected because response.body return empty array

2- Find the correct element to be asserted

To make sure that you get the right test assertion, you need to check that you are passing the correct element, let's see this example

In this example, I am trying to upload file to a certain input, so when check the element exist it's pass but when try to click on it, it's fail as there are 2 inputs for this filed

So, if we try to pass .click({force: true}) will not change anything, as shown here

What we need to do is identify the correct element to click on and upload the file successfully as shown in the second trial below

3- Retry failed scenarios

Sometimes a lot of test scenarios fails due to unpredictable conditions (eg., temporary outages in external dependencies, Animations, API calls, random network errors, etc.) which make the test results unreliable, so to reduce test flakiness and continuous integration (CI) build failures, we need to add retries{"runMode": 1, "openMode": 1} to Cypress.json file


In this example, the test assertion fail in the first attempt due to Timed out after waiting 60000ms for your remote page to load. Although in the second attempt test assertion pass as the page was fully loaded.


References