Your probes didn't fail. They stopped existing.
You bump the blackbox exporter from 0.28.0 to 0.29.0. Routine. The pod comes up, prints one line, and dies:
fatal error: removed GODEBUG "tlsrsakex" set to old value "1" in environment (https://go.dev/doc/godebug#go-127)
CrashLoopBackOff. You added that environment variable in 2024 for two legacy endpoints. Now every TLS probe that exporter ran is gone, certificate expiry checks included. And because a dead exporter produces no probe_success 0, the alerts that watch for failed probes have nothing to fire on, unless you also alert on the scrape itself.
That's not a hypothetical. I reproduced it. blackbox_exporter 0.29.0, released October 7, 2026, is compiled with Go 1.27.1. Set GODEBUG=tlsrsakex=1 on it and it refuses to start. Same for lego v5.5.2. The cause is a line in the Go 1.27 release notes that almost nobody in ops reads, because why would you read compiler release notes for a tool you didn't write.
Five escape hatches, removed permanently
Go 1.27.0 shipped on August 19, 2026. Under crypto/tls, the release notes say that the tlsunsafeekm, tlsrsakex, tls3des, tls10server and x509keypairleaf GODEBUG settings "have been removed permanently."
Each of those was a compatibility switch for a security default Go tightened two or three releases ago:
tlsrsakex=1brought back RSA key exchange cipher suites, dropped from the default list in Go 1.22.tls10server=1let servers accept TLS 1.0 and 1.1 again by default after Go 1.22 raised the server minimum to 1.2.tls3des=1restored 3DES suites, removed from the defaults in Go 1.23.tlsunsafeekm=1allowedExportKeyingMaterialon TLS 1.2 connections without Extended Master Secret.x509keypairleaf=0stoppedtls.X509KeyPairfrom populating theLeaffield.
None of this is a surprise to the Go team. Filippo Valsorda opened the removal proposal on September 8, 2025, and the GODEBUG policy has always said compatibility settings live for "a minimum of two years (four Go releases)." These hit that floor. One got a reprieve: x509negativeserial stays because too many people still need it.
So the deprecation was on schedule and documented. The trouble is who reads that documentation. In the 262 comments on the Hacker News thread for the Go 1.27 release, the crypto talk was about ML-DSA and post-quantum timelines. I couldn't find anyone mentioning the GODEBUG removals. The Go team wrote it for Go developers. The people who set GODEBUG=tlsrsakex=1 were mostly operators keeping a vendor binary talking to some appliance, and nobody tells them anything.
Setting the old value now kills the process
I tested this with the Go 1.27.2 and 1.26.6 toolchains against an OpenSSL server that only speaks AES128-SHA over TLS 1.2. The behaviour is more specific than "the setting is ignored."
Set the removed key to its old value and the runtime aborts at startup with exit status 2, before your main even runs. That goes for every one of the five. Tuck it inside a longer list (GODEBUG=http2client=0,tls10server=1) and the result is the same.
Set it to the new value (tlsrsakex=0) and nothing happens. Unknown keys get ignored, same as before. Only the old value is fatal, which makes sense: it's the one setting that asks for behaviour the binary can't give you any more.
At build time it's a hard error too:
./main.go:1:1: invalid //go:debug: removed GODEBUG "tlsrsakex" set to old value "1" (https://go.dev/doc/godebug#go-127)
go.mod:5: removed GODEBUG "tlsrsakex" set to old value "1" (https://go.dev/doc/godebug#go-127)
Honestly, I prefer this to silent failure. A crash at startup is loud and it's obvious what caused it. Just make sure it happens in staging and not on a Friday.
The breakage nobody configured
The env var case at least leaves a trail. This one doesn't.
GODEBUG defaults aren't fixed per toolchain. Go amends them to match the go line in the main module's go.mod. A tool whose go.mod still said go 1.21 got the old TLS behaviour automatically, even on a Go 1.26 build. You can see it in the binary:
$ go version -m ./client-built-with-1.26 | grep DefaultGODEBUG
build DefaultGODEBUG=...,tls10server=1,tls3des=1,tlsrsakex=1,...,tlsunsafeekm=1,x509keypairleaf=0,...
Build the same source, same go 1.21 line, with Go 1.27 and those entries are gone. The client that connected fine to the RSA-only server the day before now gets remote error: tls: handshake failure. Nobody set a variable or touched a config, and the tool's own changelog probably doesn't say a word about it, because from the maintainers' side all they did was bump the compiler.
This is already turning up in the wild. An issue filed on September 29, 2026 against plat, a Go RDAP client, describes it exactly: lookups against rdap.nic.cat and rdap.nic.eus failed because those servers only offer TLS_RSA_WITH_AES_128_CBC_SHA. The fix described there is a retry with the RSA suites listed explicitly. Registry infrastructure, in 2026, RSA key exchange only. Your legacy peers are out there too.
Which of your tools are already on 1.27
go version reads the toolchain out of any Go binary, so you don't have to trust a changelog. I pulled the latest releases of a few tools you probably run:
| Tool | Release | Date | Built with |
|---|---|---|---|
| blackbox_exporter | v0.29.0 | 2026-10-07 | go1.27.1 |
| lego | v5.5.2 | 2026-09-23 | go1.27.1 |
| Traefik | v3.7.14 | 2026-10-06 | go1.26.8 |
| Caddy | v2.11.7 | 2026-10-03 | go1.26.8 |
Traefik and Caddy aren't there yet. Don't take that as breathing room. Go supports a major release until two newer ones exist, so 1.26 falls out of security support when 1.28 lands, and everyone moves before then. Check your own fleet:
# Works on any Go binary, including one copied out of a container image
go version ./traefik ./caddy ./lego ./blackbox_exporter
# Full build info, including the effective DefaultGODEBUG
go version -m ./blackbox_exporter | grep -E '^\S|DefaultGODEBUG'
Find the GODEBUG lines before the upgrade does
Before you roll anything, grep for the five keys wherever environment gets set:
# Kubernetes: workloads plus ConfigMaps, since envFrom hides values there
kubectl get deploy,statefulset,daemonset,cronjob,configmap -A -o yaml \
| grep -nE 'GODEBUG|tlsrsakex|tls10server|tls3des|tlsunsafeekm|x509keypairleaf'
# Docker hosts
docker ps -q | xargs docker inspect --format '{{.Name}} {{.Config.Env}}' | grep GODEBUG
# systemd units and the usual env files
grep -rnE 'GODEBUG' /etc/systemd /etc/default /etc/sysconfig /etc/environment 2>/dev/null
Check Helm values files and CI runner config too. If anyone ever dropped GODEBUG=tlsrsakex=1 into a global env "just to be safe," every Go binary in that job will die on start once it is rebuilt with 1.27, including the ones that never needed it.
Find the peers that needed them
Every hit is a pointer to an endpoint that only speaks something old. Find out which one before you delete the variable:
# Does the peer do ECDHE at all on TLS 1.2?
openssl s_client -connect legacy.example.com:443 -tls1_2 -cipher 'ECDHE' </dev/null 2>&1 | grep -E 'Cipher is|alert'
# Does it do TLS 1.3?
openssl s_client -connect legacy.example.com:443 -tls1_3 </dev/null 2>&1 | grep -E 'Cipher is|alert'
If both come back with an alert, that peer depends on RSA key exchange and a Go 1.27 client with default settings won't talk to it. That's the same set of suites RFC 10015 put a MUST NOT on this summer; our post on RFC 10015 covers how to remove them from servers you control. If you can fix the peer, fix the peer. Everything below is the fallback.
The fix in code you own
The GODEBUG settings only ever changed defaults. An explicit tls.Config still works in Go 1.27, and I confirmed this one connects to the RSA-only test server:
cfg := &tls.Config{
CipherSuites: []uint16{
tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
tls.TLS_RSA_WITH_AES_128_CBC_SHA, // legacy peer only
},
}
The CipherSuites field doesn't affect TLS 1.3, so modern peers keep negotiating 1.3 as before. Keep it on the one client that talks to the legacy peer. Put it on a shared transport and every connection in the process can fall back to RSA key exchange. In Go 1.27 every TLS_RSA_* suite is listed in tls.InsecureCipherSuites(), and a code reviewer should see that word next to your change.
For servers that still have to accept TLS 1.0 clients, set MinVersion: tls.VersionTLS10 explicitly. With that set, a Go 1.27 server accepted an openssl s_client -tls1 handshake on ECDHE-RSA-AES128-SHA. Without it, the client got protocol version alert 70.
The fix in tools you don't own
This is the hard case, and the blackbox exporter is the clearest example. Its client tls_config comes from prometheus/common and has min_version and max_version, but no cipher suite field. GODEBUG was the only lever you had. On 0.28.0, built with Go 1.25.5, GODEBUG=tlsrsakex=1 turned a failed probe against the RSA-only server into probe_success 1. On 0.29.0 there's no equivalent.
If you have to keep probing that endpoint, run a second exporter pinned to 0.28.0 with the variable set, serving only the legacy module, and point every other probe at a clean 0.29.0 without it. If that one breaks, you lose one module and the rest of your monitoring keeps running. Put an expiry date on that pin, because it stops getting fixes, and a monitoring stack you can't patch is a liability in itself. If your monitoring is already noisy, this is a good moment to ask whether that legacy endpoint still needs probing at all.
Vendor agents and proxies get the same treatment. Check the toolchain with go version and the environment for the five keys. Then get the peer upgraded before the vendor's next release makes the call for you. When the handshake errors start anyway, our handshake debugging guide walks through telling a cipher mismatch apart from everything else that throws the same alert.
The Go team gave two years of notice. They gave it to the people who write Go. You run it, which is why you're hearing about it from a crashed pod.
Frequently asked questions
Does Go 1.27 remove support for RSA key exchange entirely?
No. The TLS_RSA_* cipher suites still exist in crypto/tls and still work when you list them in tls.Config.CipherSuites, which I confirmed against an RSA-only OpenSSL server. What's gone is the tlsrsakex GODEBUG switch that put them back into the default list without a code change, and Go 1.27 lists those suites in tls.InsecureCipherSuites().
Why does my Go binary crash instead of ignoring the old setting?
Go 1.27 treats a removed GODEBUG set to its old value as a fatal error at startup, because the binary can no longer provide the behaviour you asked for. Unknown keys and removed keys set to the new value are still ignored, so tlsrsakex=0 is harmless while tlsrsakex=1 stops the process with exit status 2.
How can I tell which Go version a third-party binary was built with?
Run go version /path/to/binary from any machine with a Go toolchain installed, including on a binary copied out of a container image. Adding -m prints the full build info, including the DefaultGODEBUG line that shows which compatibility defaults the binary was compiled with.
Are Caddy and Traefik affected yet? Not as of their early October 2026 releases: Caddy v2.11.7 and Traefik v3.7.14 are both built with go1.26.8. They'll move to 1.27 or later before Go 1.26 leaves security support when 1.28 ships, so remove any of the five GODEBUG keys from their environment now. Doing it mid-upgrade is how you end up debugging a crash loop on your ingress.
Which other GODEBUG settings did Go 1.27 keep?
x509negativeserial was deliberately left out of the removal proposal because too many deployments still depend on it, and Go 1.27 added new ones such as x509sslcertoverrideplatform. Each compatibility setting gets at least two years, so check the GODEBUG history page whenever you upgrade a Go toolchain.