A certificate expired at 23:05 UTC and GitHub Actions started stalling
On the night of July 19, 2026, at 23:05 UTC, jobs on GitHub Actions began backing up. Not all of them. GitHub-hosted Ubuntu and macOS runners kept working fine. But self-hosted runners and the larger hosted runners stopped picking up work, and the queue grew. At the worst point 21.4% of all workflow runs were delayed at the start. Almost 79% of larger-hosted jobs sat waiting more than five minutes to begin. It ran until 03:55 UTC. Close to six hours.
GitHub's own availability report gives the cause in one flat sentence: a certificate lifecycle management failure in a subset of internal services, an expired SSL certificate that broke runner connectivity.
So the certificate that took down a chunk of the busiest CI platform on the planet was not the one on github.com. It lived somewhere inside, on a service most people who use GitHub have never heard of. The report is specific about which one: it handled minimum runner version enforcement. A runner checks in, an internal service decides whether it is new enough to be allowed to work, and that handshake runs over TLS. The cert on that handshake expired. Runners couldn't be told they were fine, so they didn't run.
The certificate that gets you is never the one on your homepage
Everyone monitors the public cert. It is on the marketing site, it is in the SSL Labs report someone ran in 2023, it renews on its own through your CDN or your ACME client, and it is the one nobody has ever forgotten. That is exactly why it doesn't take you down.
The one that takes you down is internal. mTLS between two services. A private CA that signs your service mesh. The cert on an admin API, a metrics endpoint, a message broker, an internal registry, a gRPC backend that only ever talks to other machines. Nobody types its hostname into a browser. It got issued once, by hand or by a script someone left running, with a one-year or two-year life because short-lived felt like more work at the time. Then the person who set it up changed teams.
GitHub is not a shop that forgets about certificates. If an internal cert can expire there and cost six hours, the equivalent cert in your stack is not safer. It is just less watched.
Why your public monitoring never had a chance
Two common monitoring approaches both miss this class of certificate, and it is worth being clear about why.
Certificate Transparency monitoring watches the public logs. Every cert a public CA issues gets logged, so you can watch CT for every name you own and catch a cert before it even reaches a server. The gap: internal certs from a private CA or a self-signed root never touch a CT log. There is nothing to see. A CT feed will happily report a clean bill of health while your mesh CA quietly counts down to zero.
Uptime and endpoint checks watch what is reachable. Your synthetic monitor hits https://app.example.com every minute and screams if the handshake breaks. But it only knows about endpoints you pointed it at, and you point it at public ones. The runner-version service behind GitHub's outage was not on anyone's status page. You can't alert on a URL you never registered.
So the internal cert falls into a hole. CT doesn't log it, your uptime checks don't reach it, and the team that issued it is trusting a calendar reminder that stopped mattering the day the reminder's owner left.
"Despite alerts" is the part that should keep you up
Here is the line from GitHub's report that I keep coming back to. The certificate relied on manual deployment without automatic updates, despite alerts.
Despite alerts. The monitoring worked. Something fired. And the cert expired anyway.
This is the failure that inventory and dashboards don't fix, because it is not an information problem. It is an ownership problem. An alert that lands in a channel nobody owns is decoration. When every expiry warning looks the same and most of them are for certs that renew themselves ten days later, your brain learns to file the whole category under "handled." Then the one alert that actually needed hands gets filed there too.
An alert has to do one of two things or it is noise. It either triggers automation that fixes the problem without a human, or it pages a specific person who is on the hook to act. "It emailed a distribution list" is neither. GitHub's own remediation says as much between the lines: they are adding independent certificate expiration monitoring and improving renewal automation. They had alerts. What they are building is alerts that force an outcome.
Go find the certificates nobody is watching
Before you buy or build anything, go look. Most teams have never made a real list of the certs their internal services depend on, and the inventory you think you have is wrong in the direction that hurts.
Start with what a service actually presents on the wire, because the file on disk and the cert in memory are not always the same thing:
echo | openssl s_client -connect runner-gw.internal:8443 -servername runner-gw.internal 2>/dev/null \
| openssl x509 -noout -subject -enddate
Walk your internal endpoints in a loop and print the expiry next to each one. Unreachable or no cert is a finding too, not a blank you skip over:
for hostport in api-gw.internal:8443 mesh-ca.internal:443 registry.internal:5000 broker.internal:9093; do
end=$(echo | openssl s_client -connect "$hostport" -servername "${hostport%%:*}" 2>/dev/null \
| openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
printf '%-28s %s\n' "$hostport" "${end:-UNREACHABLE / NO CERT}"
done
If you run Kubernetes, the certs your ingress and your mesh hold live in secrets, and they can drift from whatever issued them. Read the actual bytes:
kubectl get secret internal-mtls -n mesh \
-o jsonpath='{.data.tls\.crt}' | base64 -d \
| openssl x509 -noout -subject -enddate
For a cert sitting in a file, -checkend is the one flag worth memorizing. It exits non-zero when the cert will expire within the number of seconds you give it, so it drops straight into a script:
openssl x509 -in /etc/ssl/private/service.pem -noout -checkend $((30*86400)) \
|| echo "service.pem expires within 30 days"
Check what the endpoint serves, not what the file says
There is a subtle trap in checking cert files, and GitHub's phrasing hints at it: manual deployment. Renewal and deployment are two different events. A cron job renewed the cert into /etc/ssl last night, everyone assumed it was live, and the service was still holding the old one in memory because nobody reloaded it. The file said fine. The socket said expired.
So verify the thing clients actually reach. The s_client check above is the honest one, because it does the handshake the way a real caller does. Run it against the port the service listens on, from a host that can reach it the way callers do, and read the date off the cert that comes back. That is the only cert that can take you down. The one on disk is a hope, not a fact.
This bites hardest with mutual TLS in production, where both ends carry certs and a rotation one side never picked up fails in ways that look like anything except a cert problem.
Automate the renewal, then prove it happened
The fix GitHub named is the right one. Automatic updates. If a human has to remember to rotate a cert, the cert will eventually outlive the human's attention, and the calendar reminder is the root cause more often than any protocol detail.
But automation you don't verify is just a slower way to be surprised, and renewal cron jobs fail silently all the time: an expired token, a DNS hiccup, a full disk, a service that renewed but never reloaded. The renewal ran, exit code zero, and the served cert never changed. So close the loop. After every renewal, have something reconnect to the live endpoint and confirm the served cert now expires far in the future. If it doesn't, that is your page.
For public ACME certs, ARI helps. ACME Renewal Information lets the CA tell your client when to renew instead of you guessing at a fraction of the lifetime, and it doubles as an early warning when a CA needs to push a mass renewal ahead of a revocation. It does nothing for your private CA, though. Those certs are yours to watch, start to finish.
What GitHub meant by independent monitoring
The word doing the work in GitHub's remediation is independent. Independent certificate expiration monitoring.
Independent means the thing watching the cert does not share fate with the thing that issues or serves it. If your expiry checks run inside the same cluster, signed by the same internal CA, behind the same expired cert, then the outage that kills the service kills the monitoring in the same breath and you hear about it from customers. The watcher has to sit outside, on a different system and a different trust path, so that when the inside is on fire the alarm still rings somewhere you can hear it.
That is the lesson from a six-hour outage at a company that is genuinely good at this. The public cert was never the risk. The internal one was, nobody watched it independently, and the alert that did exist landed somewhere soft. Go run the loop above against your own internal hosts this afternoon. You will find at least one cert you forgot you owned, and you would rather find it now than at 23:05 on a Saturday.
Frequently asked questions
Would Certificate Transparency monitoring have caught the GitHub cert? No. CT logs only contain certificates issued by publicly trusted CAs. An internal service cert from a private or self-signed CA never appears in a CT log, so CT monitoring has nothing to match against and reports no problem while the cert quietly expires.
How short should internal certificate lifetimes be? Short enough that renewal has to be automated, which is the real goal. A cert with a one-year life invites manual handling and a forgotten calendar reminder. A cert measured in days or weeks forces you to build automation, and automation that runs constantly is far less likely to be silently broken than a script that fires once a year.
We already get expiry email alerts. Why did GitHub still go down with alerts in place? Because an alert that nobody owns is noise. GitHub's report says the certificate expired despite alerts. The fix is not more alerts, it is routing each one to either automation that renews the cert or a specific on-call person who is accountable for acting, so the warning produces an outcome instead of a read receipt.
What is the fastest way to check a live internal endpoint's expiry?
Pipe openssl s_client into openssl x509 -noout -enddate against the host and port the service listens on. That reads the certificate actually being served on the wire, which is the only one that can cause an outage, rather than a file on disk that may not have been reloaded into the running process.
Does ARI help with internal certificates? No. ACME Renewal Information is part of the ACME protocol and applies to certs issued by an ACME-capable CA. Your private CA and any self-signed certs are outside that flow entirely, so you are responsible for scheduling their renewal and verifying it took effect.
