Bug: retryOnFailedStep works only for the first test in the same run

[smalia2001]

I have the following test file:

Feature("github testing")

Scenario("test", async ({ I }) => {
  await I.amOnPage('https://github.com');
  await I.click('thistextdoesnotexist');
})

Scenario("test2", async ({ I }) => {
  await I.amOnPage('https://github.com');
  await I.click('thistextdoesnotexisttoo');
})

and I enabled the retryFailedStep plugin (this is the only plugin, also using playwright helper)

I noticed that the plugin doesn't retry the click step in test2

I patched two functions in the plugin (codeceptjs v4.0.3):

99   const when = err => {
100     if (!enableRetry) return
101     if (store.debugMode) return false
102     console.log("store.autoRetries", store.autoRetries); // <-- this is new
103     if (!store.autoRetries) return false
104     if (err && err.isTerminal) return false
105     if (err && err.message && (err.message.includes('ERR_ABORTED') || err.message.includes('frame was detached') || err.message.includes('Target page, context or browser has been closed'))) return false
106     return true
107   }

and

140     if (scenarioRetries > 0 && config.deferToScenarioRetries !== false) {
141       store.autoRetries = false
142       return
143     }
144 
145     const hasManualRetries = recorder.retries.some(retry => retry !== config) // in the second test, recorder.retries.length === 3
146     console.log("hasManualRetries", hasManualRetries); // <-- this is new
147     if (hasManualRetries) {
148       store.autoRetries = false
149       return
150     }
151

I ran it and this is the output:

github testing --
hasManualRetries false
store.autoRetries true
store.autoRetries true
store.autoRetries true
store.autoRetries true
  ✖ test in 7886ms
hasManualRetries true
store.autoRetries false
store.autoRetries false
  ✖ test2 in 5760ms

Thanks!

[kobenguyent]

@smalia2001 could you try this locally if this resolves your issue? #5601

[smalia2001]

@kobenguyent Thank you for the quick response!
I tried it, but it didn’t work - I’m still getting the same output.

I also tried to see what happens if the first test passes - the bug remains.

[smalia2001]

@kobenguyent what is the purpose of the hasManualRetries variable?
I saw that lib/helper/Playwright.js adds another retry which makes this variable be true.
also it looks like there is a leak in recorder.retries over time (I added the isFromLibHelperPlaywright):

github testing --
{
  hasManualRetries: false,
  retries: [
    {
      retries: 3,
      defaultIgnoredSteps: [Array],
      minTimeout: 150,
      maxTimeout: 10000,
      factor: 1.5,
      randomize: false,
      ignoredSteps: [Array],
      deferToScenarioRetries: true,
      enabled: true,
      when: [Function: when]
    }
  ]
}
playwright helper will add a retry now:
recorder.retries after playwright helper added its retry: [
  {
    retries: 3,
    defaultIgnoredSteps: [ 'amOnPage', 'wait*', 'send*', 'execute*', 'run*', 'have*' ],
    minTimeout: 150,
    maxTimeout: 10000,
    factor: 1.5,
    randomize: false,
    ignoredSteps: [ 'amOnPage', 'wait*', 'send*', 'execute*', 'run*', 'have*' ],
    deferToScenarioRetries: true,
    enabled: true,
    when: [Function: when]
  },
  {
    retries: 3,
    defaultIgnoredSteps: [ 'amOnPage', 'wait*', 'send*', 'execute*', 'run*', 'have*' ],
    minTimeout: 150,
    maxTimeout: 10000,
    factor: 1.5,
    randomize: false,
    ignoredSteps: [ 'amOnPage', 'wait*', 'send*', 'execute*', 'run*', 'have*' ],
    deferToScenarioRetries: true,
    enabled: true,
    when: [Function: when]
  },
  {
    isFromLibHelperPlaywright: true,
    retries: 3,
    when: [Function: when]
  }
]
  ✖ test in 8493ms
{
  hasManualRetries: true,
  retries: [
    {
      retries: 3,
      defaultIgnoredSteps: [Array],
      minTimeout: 150,
      maxTimeout: 10000,
      factor: 1.5,
      randomize: false,
      ignoredSteps: [Array],
      deferToScenarioRetries: true,
      enabled: true,
      when: [Function: when]
    },
    {
      retries: 3,
      defaultIgnoredSteps: [Array],
      minTimeout: 150,
      maxTimeout: 10000,
      factor: 1.5,
      randomize: false,
      ignoredSteps: [Array],
      deferToScenarioRetries: true,
      enabled: true,
      when: [Function: when]
    },
    {
      isFromLibHelperPlaywright: true,
      retries: 3,
      when: [Function: when]
    }
  ]
}
playwright helper will add a retry now:
recorder.retries after playwright helper added its retry: [
  {
    retries: 3,
    defaultIgnoredSteps: [ 'amOnPage', 'wait*', 'send*', 'execute*', 'run*', 'have*' ],
    minTimeout: 150,
    maxTimeout: 10000,
    factor: 1.5,
    randomize: false,
    ignoredSteps: [ 'amOnPage', 'wait*', 'send*', 'execute*', 'run*', 'have*' ],
    deferToScenarioRetries: true,
    enabled: true,
    when: [Function: when]
  },
  {
    retries: 3,
    defaultIgnoredSteps: [ 'amOnPage', 'wait*', 'send*', 'execute*', 'run*', 'have*' ],
    minTimeout: 150,
    maxTimeout: 10000,
    factor: 1.5,
    randomize: false,
    ignoredSteps: [ 'amOnPage', 'wait*', 'send*', 'execute*', 'run*', 'have*' ],
    deferToScenarioRetries: true,
    enabled: true,
    when: [Function: when]
  },
  {
    isFromLibHelperPlaywright: true,
    retries: 3,
    when: [Function: when]
  },
  {
    isFromLibHelperPlaywright: true,
    retries: 3,
    when: [Function: when]
  }
]
  ✖ test2 in 2654ms

[kobenguyent]

maybe @DavertMik could explain this better than myself.

[smalia2001]

The reset of the retries array was commented out here: #3768
I understand the reason for commenting it out.
I think that each plugin/helper should be responsible of managing its own retries configuration by adding and removing it from the recorder.retries array.