Fixing “Certbot: Some challenges have failed” on a Cloudflare-Proxied Site (Apache + Let’s Encrypt)

My Let’s Encrypt renewal started failing on an Apache server behind Cloudflare, and the error gave almost no clues. This post covers what the failure looked like, how I diagnosed it, and the fix that worked, so I can solve it faster next time.

The symptom

Running sudo certbot renew --dry-run ended with a generic message:

Failed to renew certificate example.com with error: Some challenges have failed.

That line is only a summary. The useful details are a few lines above it, or in the log:

sudo certbot renew --dry-run --cert-name example.com 2>&1 | grep -iE "detail|type:|domain|error|timeout"
sudo tail -n 80 /var/log/letsencrypt/letsencrypt.log

This showed:

Type:   unauthorized
Detail: 2606:4700:3032::...: Invalid response from http://example.com/.well-known/acme-challenge/...: 522

Step 1: Read the error code and the IP

Two details in that error matter.

The IP address belongs to Cloudflare, not your server. Addresses starting with 2606:4700, 104., 172.64–172.71 and 162.158 are Cloudflare’s. Let’s Encrypt was talking to Cloudflare’s proxy, not to my Apache.

Codes in the 520s are Cloudflare errors, not Apache errors:

Code	Meaning
520	Origin returned an empty or invalid response
522	Connection to origin timed out
521	Origin refused the connection
526	Invalid SSL certificate on origin (Full strict mode)

So the question isn’t “why is certbot broken?” but “why can’t Cloudflare reach my origin during the challenge?”

Step 2: Rule out the obvious causes

I checked the server first.

sudo systemctl status apache2 --no-pager | head -n 5
sudo ss -tlnp | grep -E ':80|:443'
curl -I http://localhost
sudo ufw status
curl -4 ifconfig.me

Everything looked healthy: Apache running, listening on 80 and 443, firewall open, and a normal 301 redirect to HTTPS.

Then I checked what a hosted-server setup can hide:

• Cloud firewall: On AWS Lightsail or EC2 there’s a firewall separate from ufw. Ports 80 and 443 must be open in the console (IPv4 and IPv6 tabs).
• Cloudflare DNS: The A records for the apex and www must point at the server’s current public IP. If the instance was stopped and started without a static IP, the address may have changed.
• Cloudflare SSL/TLS mode: Flexible combined with an HTTP→HTTPS redirect on Apache causes loops and 52x errors. Use Full (strict).

A note on testing: curl run from the server itself doesn’t prove the outside world can reach it. Test from a laptop as well.

Step 3: See whether Cloudflare’s requests reach Apache

I tailed the access log while the dry run executed:

sudo tail -f /var/log/apache2/access.log | grep --line-buffered acme-challenge

In my case, normal traffic from Cloudflare IPs was reaching Apache and getting 200 responses. The origin was healthy, and the failures were intermittent. The apex domain failed with a 522 in one run and passed in the next, while www failed repeatedly. I never pinned down the exact cause of those intermittent errors, and I had guessed at server overload, which turned out not to be supported by the evidence. When a Cloudflare 52x is intermittent and the origin looks fine, chasing it can eat a lot of time.

Step 4: The fix, switch to DNS-01 validation

The default HTTP-01 challenge needs Let’s Encrypt to fetch a file over port 80, through Cloudflare, from Apache, while certbot temporarily rewrites the Apache config. That’s a lot of moving parts.

DNS-01 proves domain ownership by creating a TXT record through the Cloudflare API. It ignores port 80, the proxy, and Apache.

1. Create a Cloudflare API token

In Cloudflare go to My Profile → API Tokens → Create Token, use the “Edit zone DNS” template, and limit it to the single zone.

2. Install the plugin and store the token

sudo apt install python3-certbot-dns-cloudflare
sudo mkdir -p /root/.secrets
sudo nano /root/.secrets/cloudflare.ini

File contents:

dns_cloudflare_api_token = YOUR_TOKEN_HERE

Lock it down:

sudo chmod 600 /root/.secrets/cloudflare.ini

3. Reconfigure the existing certificate

certbot reconfigure (certbot 2.3 or newer) switches an existing certificate to a new method without reissuing it:

sudo certbot reconfigure --cert-name example.com \
  --authenticator dns-cloudflare \
  --dns-cloudflare-credentials /root/.secrets/cloudflare.ini \
  --dns-cloudflare-propagation-seconds 60

The gotcha: propagation time

My first attempt used the default 10-second wait. The apex domain passed, but www failed with:

DNS problem: NXDOMAIN looking up TXT for _acme-challenge.www.example.com

That didn’t mean a broken record. The TXT record just hadn’t propagated yet. Setting --dns-cloudflare-propagation-seconds 60 fixed it.

Also, reconfigure only saves the change if its test renewal succeeds. When it fails, the old settings stay in place, even though the output may say other things were updated. Always verify the config afterward.

4. Verify

sudo certbot renew --dry-run
grep -E "authenticator|installer" /etc/letsencrypt/renewal/example.com.conf

Expected result:

Congratulations, all simulated renewals succeeded
authenticator = dns-cloudflare
installer = apache

installer = apache means Apache reloads automatically after renewal. If it’s missing, add --deploy-hook "systemctl reload apache2".

Step 5: Confirm renewal really happens

Certbot renews certificates within 30 days of expiry. Before that, timer runs do nothing, so a dry run passing is the best check until the window opens.

systemctl list-timers | grep certbot     # timer active?
sudo certbot certificates                # expiry date
sudo journalctl -u certbot.service --since "3 days ago" --no-pager | tail -n 30

After the renewal window opens, the expiry should jump about 90 days ahead.

Quick checklist for next time

1. Read the Type: and Detail: lines, not just the summary.
2. Is the IP in the error Cloudflare’s? Is the code a Cloudflare 52x?
3. Check Apache, ports 80/443, both firewalls (server and cloud), and the A/AAAA records.
4. Check Cloudflare SSL mode (Full strict) and security events for blocked /.well-known/.
5. If it’s still flaky, switch to DNS-01 with the Cloudflare plugin and a 60-second propagation wait.
6. Verify with certbot renew --dry-run and the authenticator line in the renewal config.
7. Keep the API token in a chmod 600 file, scoped to one zone, and never paste it anywhere public.