A flaky test is a disagreement at one commit, not a failure rate

The code did not change, and the test result did. That is the whole definition.

A failure rate — "this test fails 8% of the time" — sounds more scientific, and it is worse. It confounds a test that is nondeterministic with a test that was broken on Tuesday and fixed on Thursday. It changes as history ages. And it invites the argument every team has had: is 8% flaky, or is 8% a real bug that only shows up under load?

A disagreement at one commit is not arguable. Two runs of the same test, on the same commit, with the same inputs, produced different results. Either the test is nondeterministic or the environment is, and in both cases the pull request author did nothing wrong.

The matrix leg is part of the identity

A test that flakes only on arm64 is a different fact from a test that flakes everywhere, so a leg is part of a test's name. test_upload (ubuntu-latest, node 22) and test_upload (pristine-4core, node 22) are two tests. One of them may be fine.

A skip is no evidence either way. A test that was skipped on one run and passed on the next has not disagreed with itself.

One warning, never a second failure

When Apex sees a disagreement, the pull request gets one warning naming the test. The run stays green if the test eventually passed and red if it did not; the flake never adds a failure of its own. A second push does not produce a second warning for the same test — the author already knows.

Where the reports come from

Apex reads the JUnit, TAP and CTRF reports your job already writes and discovers them in the workspace. There is no upload step, no action to add, and nothing changes in the workflow file. If a job writes no report, it gets no flaky-test detection and nothing else changes.