The bug report that makes no sense
Your API is working fine. You tested it in Chrome, Firefox, Postman. All green. Then a user opens a ticket: "The app crashes when I try to log in." You check the logs and see TLS handshake failures from their iPhone. Meanwhile, your Android QA team says everything works.
What's happening?
You forgot to install the intermediate certificate. Chrome didn't care because it cached it from another site. The iPhone's trust store doesn't have it yet. Android might have it, might not, depends on what other apps the user has installed recently. Welcome to certificate chain validation hell.
Why chains exist in the first place
Certificate Authorities don't sign your certificate directly with their root certificate. That would require exposing the root private key constantly, which is a terrible idea. Instead, CAs use intermediate certificates.
The chain looks like this:
Root CA (in client trust store)
└── Intermediate CA (you must send this)
└── Your server certificate (obviously sent)When a client connects, your server sends your certificate. The client needs to verify it's signed by a trusted root. But if you only send your certificate and the client doesn't already have the intermediate, the chain is broken. Connection fails.
Desktop browsers have been quietly fixing this for years through a mechanism called AIA fetching. If they're missing an intermediate, they download it automatically using the Authority Information Access extension in your certificate. Mobile clients? Way less forgiving. curl? Nope. Java applications? Inconsistent at best.
How to check if you have a problem
OpenSSL makes this easy to verify:
# Good: shows the full chain
openssl s_client -connect yourdomain.com:443 -showcerts
# If you see "Verify return code: 0 (ok)" you're fine.
# If you see "Verify return code: 21 (unable to verify the first certificate)"
# you're missing intermediates.
# More aggressive test - disable AIA fetching (simulates strict clients)
openssl s_client -connect yourdomain.com:443 -CAfile /etc/ssl/certs/ca-certificates.crt -verify_return_error -no_ssl3 -no_tls1 -no_tls1_1Better yet, use SSL Labs. They'll tell you exactly what's missing and which clients will fail: https://www.ssllabs.com/ssltest/analyze.html?d=yourdomain.com
The right way to install certificates
When your CA issues a certificate, they give you two files: your certificate and a bundle (or chain file) containing the intermediates. Your job is to configure your server to send both.
Nginx does this cleanly:
# nginx.conf
server {
listen 443 ssl;
server_name yourdomain.com;
# This file should contain YOUR cert followed by the intermediates
ssl_certificate /etc/nginx/ssl/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;
# Modern TLS config
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
}The trick is that fullchain.pem must be in the right order. Your certificate first, then intermediate(s), then optionally the root (though sending the root is unnecessary since clients already trust it).
If you're using Let's Encrypt with certbot, it creates this file for you automatically. If you bought a commercial certificate, you need to concatenate them manually:
# Create the full chain yourself
cat your-certificate.crt intermediate.crt > fullchain.pem
# Verify the order is correct (should show 2+ certificates)
openssl crl2pkcs7 -nocrl -certfile fullchain.pem | openssl pkcs7 -print_certs -nooutApache does it differently
Apache splits the config into two directives:
# httpd.conf or your vhost config
SSLEngine on
SSLCertificateFile /etc/ssl/certs/your-cert.crt
SSLCertificateKeyFile /etc/ssl/private/your-key.key
# This is the critical line most people forget
SSLCertificateChainFile /etc/ssl/certs/intermediate.crtIf you omit SSLCertificateChainFile, Apache will happily start and serve traffic. Chrome will work because of AIA fetching. Then your mobile app breaks and you'll spend three hours debugging before you find this config line.
On newer Apache versions (2.4.8+), they unified this:
# Modern Apache - just use one file like nginx
SSLCertificateFile /etc/ssl/certs/fullchain.pem
SSLCertificateKeyFile /etc/ssl/private/your-key.keyCloud load balancers and their quirks
AWS ALB handles this well. When you upload a certificate to ACM (AWS Certificate Manager), it validates the chain and stores everything. You don't have to think about intermediates. When you import a certificate manually (not through ACM), you explicitly provide the chain in a separate field. Hard to mess up.
GCP's Certificate Manager is similar. Good validation, clear separation of cert and chain.
Where things get weird is older infrastructure or legacy setups. I've seen HAProxy configs where someone copy-pasted just the leaf certificate, and it worked for months until someone tested from an old Android device. HAProxy's config combines everything into one PEM file, same as nginx:
# haproxy.cfg
bind :443 ssl crt /etc/haproxy/certs/combined.pem
# combined.pem must contain:
# 1. Private key
# 2. Your certificate
# 3. Intermediate(s)
# All in one file, that exact order.Why cross-signed intermediates complicate this
Some CAs issue cross-signed intermediates to maintain compatibility with older clients that don't trust their newer roots yet. Let's Encrypt did this famously when they transitioned from the IdenTrust cross-sign to their own ISRG Root X1.
You might get two intermediate certificate options: one that chains to the old trusted root (works on ancient Android devices) and one that chains to the new root (smaller, faster, but requires recent trust stores).
Choosing the wrong one means you either break old devices or send unnecessary bytes on every handshake. Most teams pick "whatever certbot gave me" and hope. Which is fine until you need to support a corporate Android tablet running a 2019 OS.
Certificate renewal is when chains break
You set up your server correctly two years ago. Intermediates were in place. Everything worked. Then you renew the certificate.
The new certificate might chain through a different intermediate. Maybe your CA rotated their intermediate certificates for security reasons. If your renewal process just replaces the leaf certificate and doesn't update the chain file, you now have a mismatch. Your server is sending an old intermediate that doesn't sign your new certificate.
This is why automation matters. ACME clients like certbot handle this by always fetching the current chain. Manual renewal workflows forget this step constantly.
# When renewing manually, always get a fresh chain
# Don't just re-use your old intermediate.crt file.
# For Let's Encrypt, grab the current intermediate
curl -o intermediate.pem https://letsencrypt.org/certs/lets-encrypt-r3.pem
# For commercial CAs, download from their repository
# They publish current intermediates at a stable URLMonitoring this before users complain
SSL Labs is great for ad-hoc checks but you need automated monitoring. The validation you want is: connect without AIA fetching and verify the chain completes to a trusted root.
#!/bin/bash
# Strict chain validation (simulates unforgiving clients)
DOMAIN="${1}"
TRUSTED_ROOTS="/etc/ssl/certs/ca-certificates.crt"
# Fetch the chain
CHAIN=$(openssl s_client -connect "${DOMAIN}:443" -servername "${DOMAIN}" -showcerts 2>/dev/null | awk '/BEGIN CERT/,/END CERT/ {print}')
# Verify without AIA fetching
echo "${CHAIN}" | openssl verify -CAfile "${TRUSTED_ROOTS}" -untrusted /dev/stdin
if [ $? -eq 0 ]; then
echo "Chain is valid"
exit 0
else
echo "Chain validation failed - missing intermediates?"
exit 1
fiRun this daily against your production endpoints. Alert if it fails. Much better than waiting for user reports.
The Java ecosystem's special hell
Java applications bring their own trust store (the JRE's cacerts file), which is often outdated. Even if your server is configured perfectly, a Java client might fail because their local trust store doesn't have your CA's root.
Android apps are Java-based. If your mobile app works on iOS but fails on older Android devices, this is probably why. The device's trust store is out of date.
You can't fix their trust store. What you can do is make absolutely certain you're sending a complete chain, so even if they're missing some intermediates, they only need the root (which is more likely to be present).
Testing across real devices matters
Emulators and simulators are useful. They're also liars. An iPhone simulator running on your Mac uses macOS's trust store, not iOS's. An Android emulator might have Google Play services updating its trust store constantly, while a real device from 2020 hasn't been updated in years.
If you support mobile clients, you need at least one physical test device running an older OS. When you renew certificates, test on that device before you push to production. Yes, this is annoying. It's less annoying than emergency debugging on a Friday night because the mobile app suddenly breaks for 30% of your users.
When certificate pinning makes everything worse
Certificate pinning is when your mobile app hard-codes which certificates or public keys it will trust. This is supposed to prevent man-in-the-middle attacks. In practice, it creates deployment nightmares during certificate renewals.
If you pin to the leaf certificate and you renew it, every app version with the old pin immediately breaks. If you pin to the intermediate, you're safe during leaf renewals but screwed when the CA rotates intermediates. If you pin to the root, you're mostly safe but you've lost most of the security benefit of pinning.
The only viable pinning strategy is to include multiple backup pins (current + future certificates) and to have a kill-switch that disables pinning via a config update if things go sideways. Most teams discover this the hard way.
What good lifecycle management looks like
Here's the workflow that actually prevents chain issues:
1. When issuing or renewing, always download the full chain from the CA.
2. Verify the chain locally before deploying (use openssl verify).
3. Deploy the leaf + intermediates together, atomically.
4. Test from a strict client immediately after deployment (curl with --cacert, or from an older mobile device).
5. Monitor chain validation continuously, not just cert expiry.
If your certificate management process doesn't include all five steps, you'll have incidents. Maybe not today, maybe not on desktop browsers, but eventually on some client that doesn't do AIA fetching for you.
Certificate chains are one of those things that seem to "just work" right up until they don't. When they break, they break silently for some clients while working perfectly for others, which makes debugging a nightmare. Build your process assuming nothing is forgiving. That way, when you do hit an unforgiving client, you're already prepared.