Bright Labs

Is a failing GitHub Actions check flaky or real? How to read the run history before you re-run

Your pull request has a red check on a job you didn't touch, and you're deciding whether to hit re-run or open the log. You can usually tell from the run history. If that job failed and then passed on the same commit, that points to a flake. If it's also failing on your default branch, your change probably didn't cause it. If neither is true, treat it as a real failure and read the log.

This post covers how to do those checks by hand in the web UI, with the gh CLI and with the REST API. It also covers what each answer means and the limits you'll run into.

Why the same commit is the test that matters

A re-run doesn't pick up new code. GitHub's docs say a re-run uses the same commit SHA and ref of the original event that triggered the workflow run. So if attempt one failed and a later attempt passed, the job ran on the same commit both times.

That's why "failed, then passed on the same commit" is the signal to look for. A job that fails on one commit and passes on the next tells you much less, because the code changed in between. Here's the catch: one fail-then-pass is a single data point. A job that does it again and again across many commits is a much stronger signal than one that did it once.

Check earlier attempts in the web UI

Open the failing check's workflow run. If anyone has re-run it, the run has more than one attempt. Per GitHub's docs, to the right of the run name you select the Latest dropdown menu and click a previous run attempt.

Check attempts and logs with the gh CLI

The CLI is handy when you want to look at several runs. gh run list has a --commit flag, which the manual describes as "Filter runs by the SHA of the commit", so you can list runs for your PR's head commit. It also has --branch and --status, so you can filter to failures on main.

To look at one attempt, gh run view takes --attempt, described as "The attempt number of the workflow run". The manual gives this example:

From the gh run view manual
gh run view 12345 --attempt 3

gh run view also has --log-failed. The manual describes it as "View the log for any failed steps in a run or specific job". Comparing the failed step's log across attempts is often the quickest way to see whether the error is the same each time or different each time.

Check history at scale with the REST API

For counts across many runs, use the REST API. The endpoint /repos/{owner}/{repo}/actions/runs lists workflow runs. Its head_sha parameter only returns workflow runs that are associated with the specified head_sha, and its branch parameter returns runs associated with a branch.

For each run, list the jobs per attempt. /repos/{owner}/{repo}/actions/runs/{run_id}/attempts/{attempt_number}/jobs lists jobs for a specific workflow run attempt. The plain jobs endpoint, /repos/{owner}/{repo}/actions/runs/{run_id}/jobs, also takes filter=all, which returns all jobs for a workflow run, including from old executions of the workflow run. A job that failed in one attempt and succeeded in a later attempt on the same run is your fail-then-pass.

One limit to plan around: the list endpoint will return up to 1,000 results for each search when using certain filter parameters, so check the reference for which ones. Narrow the date range with created if you're scanning a busy repository.

Check the default branch before you blame your PR

This one is easy to miss. A job can fail every time without being caused by your change, because it's already broken on main. Before you dig into your diff, list runs on the default branch with gh run list --branch and --status, or call the runs endpoint with branch.

What to do with each answer

If it looks like a flake, a re-run of only the failed jobs is usually the cheapest next step. GitHub's docs say that to re-run failed jobs in a workflow run, you use the run rerun subcommand with the --failed flag:

From GitHub's re-run docs
gh run rerun RUN_ID --failed

Keep in mind that when you re-run failed jobs or a specific job, GitHub also re-runs the jobs that depend on them. If you want more detail from the next attempt, GitHub's docs say that to enable runner diagnostic logging and step debug logging for the re-run, you use the --debug flag.

There are hard limits too. You can re-run a workflow run up to 30 days after its initial run, and a workflow run can be re-run a maximum of 50 times. If you keep re-running the same flaky job, write down which test it is and fix or quarantine it rather than re-running forever.

If it looks real, read the failed step's log before anything else. Even when the history says flake, a known flake isn't safe to ignore. It means the job has failed and then passed on the same commit before. That's evidence, not proof that this particular failure is harmless.

Getting the label on the pull request itself

If you'd rather not run these checks by hand on every red PR, we make a free Chrome extension, FlakeRadar, that does this lookup on the pull request page. It reads the repository's own GitHub Actions history and gives each failing Actions job one label: Known flake, New failure, Also failing on the default branch, or Not enough history. A job counts as flaky when it failed and then passed on the same commit. Each label shows counts, a time window and a date, for example: failed then passed on the same commit on 4 of 120 commits on main in the last 30 days.

It runs in your browser with a GitHub token you create and keep on your device, so there's nothing to add to your CI. It only analyses GitHub Actions checks. FlakeRadar is not affiliated with or endorsed by GitHub.

Sources

Questions

Does a passing re-run prove the failure was flaky?

No. It shows the job failed and then passed on the same commit, since a re-run uses the same commit SHA as the original run. That points to a flake, but read the failed log if the failure could matter.

How do I see the logs of an earlier attempt?

In the web UI, open the run, then use the Latest dropdown next to the run name and pick a previous attempt. With the CLI, use gh run view with --attempt, and use --log-failed to view the log for failed steps.

How long can I re-run a failed workflow?

GitHub's docs say you can re-run a workflow run up to 30 days after its initial run, and a workflow run can be re-run a maximum of 50 times.

Related