homey/docs/caddy-cloudflare-tls.org

Caddy, Cloudflare Tunnel & TLS Setup

• Overview
  • Traffic flow
• Components
  • Caddy (modules/caddy.nix)
  • Cloudflare Tunnel (modules/cloudflared.nix)
• Problems Encountered & How They Were Resolved
  • 1. caddy-dns/cloudflare rejected cfut_ token format
  • 2. cloudflared tls: internal error (SNI mismatch)
  • 3. Cloudflare edge TLS handshake failure (*.home.zakobar.com)
  • 4. ERR_TOO_MANY_REDIRECTS via tunnel
• Adding a New Service
• Certificate Details
  • Let's Encrypt cert (LAN access)
  • Cloudflare Universal SSL cert (tunnel / remote access)
• Quick Reference: Debugging Checklist

Overview

This document describes the TLS and reverse-proxy architecture for the homey self-hosted stack, the problems encountered while getting it working, and the final configuration that resolved them. It is intended as a reference for future debugging and for adding new services.

Traffic flow

Browser
  │  HTTPS (TLS terminated by Cloudflare edge, *.zakobar.com cert)
  ▼
Cloudflare edge (anycast IP)
  │  QUIC/HTTP2 tunnel (outbound from Pi, no open inbound ports)
  ▼
cloudflared daemon on Pi  (systemd: cloudflared-tunnel.service)
  │  plain HTTP on loopback  http://localhost:80
  ▼
Caddy reverse proxy  (systemd: caddy.service, port 80 + 443)
  │  proxies to backend by Host header
  ▼
Service container  (podman, port on 127.0.0.1)

Key points:

• TLS to the browser is provided entirely by Cloudflare's Universal SSL cert (*.zakobar.com), not by the Pi's Let's Encrypt cert.
• The Pi's Let's Encrypt cert (*.zakobar.com via DNS-01) is used only for direct LAN access (bypassing the tunnel).
• The tunnel leg (cloudflared → Caddy) is plain HTTP on loopback — this is safe because both endpoints are the same machine.

Components

Caddy (modules/caddy.nix)

Caddy runs as a NixOS service (services.caddy) using a custom build that includes the caddy-dns/cloudflare plugin for DNS-01 ACME challenges.

Custom build

The nixpkgs caddy package does not include the Cloudflare DNS plugin by default. It is built using the withPlugins passthru function (backed by xcaddy):

caddyWithCloudflare = pkgs.caddy.withPlugins {
  plugins = [
    "github.com/caddy-dns/cloudflare@v0.2.4"
  ];
  hash = "sha256-...";
};

The hash is a fixed-output derivation hash that must be updated whenever the plugin version changes. Use lib.fakeHash to trigger a build failure that prints the correct hash, then substitute it.

API token injection

The Cloudflare API token is stored in sops (cloudflare/api_token) and injected into the Caddy process via systemd LoadCredential:

serviceConfig.LoadCredential =
  "cloudflare_api_token:${config.sops.secrets."cloudflare/api_token".path}";
ExecStart = lib.mkForce [
  ""
  (pkgs.writeShellScript "caddy-start" ''
    export CLOUDFLARE_API_TOKEN=$(cat "$CREDENTIALS_DIRECTORY/cloudflare_api_token")
    exec caddy run --environ --config /etc/caddy/caddy_config --adapter caddyfile
  '')
];

Virtual hosts — dual HTTP/HTTPS entries

Each service has two Caddyfile vhost entries:

Entry	Purpose
git.zakobar.com	HTTPS — for direct LAN access; Caddy handles TLS
http://git.zakobar.com	HTTP — for cloudflared on loopback; no redirect

Caddy's default behaviour is to automatically redirect HTTP → HTTPS for any hostname that has a matching HTTPS vhost. By explicitly defining an http:// vhost, that redirect is suppressed and cloudflared gets a direct 200 response instead of a redirect loop.

Without the http:// vhost, accessing via the tunnel produces: ERR_TOO_MANY_REDIRECTS in the browser because cloudflared follows the 308 back to HTTP indefinitely.

Global config

{
  email admin@zakobar.com
  acme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}
}

The acme_dns directive in the global block tells Caddy to use DNS-01 challenges for all HTTPS vhosts. This allows wildcard and multi-level subdomain certs to be issued without any inbound port 80 requirement.

Cloudflare Tunnel (modules/cloudflared.nix)

cloudflared runs as a plain systemd service using the token-based tunnel approach (cloudflared tunnel run --token). No local credentials file or config file is needed — just the tunnel token from the Zero Trust dashboard.

Tunnel configuration (Zero Trust dashboard)

One wildcard public hostname entry covers all services:

Field	Value
Hostname	*.zakobar.com
Service	http://localhost:80
No TLS Verify	off (not needed for HTTP)
HTTP Host Header	(empty — cloudflared forwards the real Host header)
Origin Server Name	(empty — not needed for HTTP)

cloudflared automatically forwards the incoming Host header (e.g. git.zakobar.com) to Caddy, which uses it to select the correct vhost and backend.

DNS records

A single wildcard CNAME record in Cloudflare DNS covers all subdomains:

*.zakobar.com  CNAME  <tunnel-id>.cfargotunnel.com  (proxied, orange cloud)

This means new services require no DNS changes — only a new Caddy vhost.

Cloudflare SSL/TLS mode

Set to Full (strict) in the Cloudflare dashboard (SSL/TLS → Overview).

Mode	Meaning
Off	No HTTPS to browser
Flexible	HTTPS to browser, HTTP to origin
Full	HTTPS to browser, HTTPS to origin (cert not validated)
Full (strict)	HTTPS to browser, HTTPS to origin (cert must be valid)

Full (strict) works here because Cloudflare terminates TLS at its own edge using its Universal cert, and the origin (cloudflared → Caddy) uses plain HTTP which Cloudflare does not validate in this tunnel architecture.

Problems Encountered & How They Were Resolved

1. caddy-dns/cloudflare rejected cfut_ token format

Symptom:

provision dns.providers.cloudflare: API token 'cfut_...' appears invalid;
ensure it's correctly entered and not wrapped in braces nor quotes

Cause: Cloudflare introduced new token formats with a cfut_ (user token) or cfat_ (account token) prefix. These tokens are 54 characters long. The caddy-dns/cloudflare plugin had a validation regex {35,50} that rejected tokens longer than 50 characters, failing before even making an API call.

Fix: The fix was merged into the plugin's master branch as commit a8737d0 and included in the v0.2.4 tag (despite the tag previously being associated with an older tree — the proxy confirmed v0.2.4 resolves to a8737d0).

Updating the hash in caddy.nix to the value produced by lib.fakeHash forced a fresh fetch of the corrected v0.2.4 tree:

plugins = [ "github.com/caddy-dns/cloudflare@v0.2.4" ];
hash = lib.fakeHash;  # replace with hash from build error output

Run nix build .#nixosConfigurations.pi-main.config.system.build.toplevel, copy the got: hash from the error, substitute it, and rebuild.

2. cloudflared tls: internal error (SNI mismatch)

Symptom:

Unable to reach the origin service: remote error: tls: internal error
originService=https://localhost:443

Cause: cloudflared connected to https://localhost:443 without sending an SNI (Server Name Indication) hostname in the TLS ClientHello. Caddy could not match any vhost, had no certificate for localhost, and aborted the handshake with a TLS internal error.

Setting the HTTP Host Header override in the dashboard fixes the HTTP layer but does not affect the TLS SNI, which is negotiated before HTTP headers are exchanged.

Setting the Origin Server Name field does set the SNI, but for a wildcard rule (*.zakobar.com) the dashboard only accepts a static value, not a dynamic placeholder — so it cannot be used for a catch-all.

Fix: Switch the tunnel service from https://localhost:443 to http://localhost:80. The internal leg does not need TLS (loopback interface, same machine). Caddy's HTTP vhosts handle the requests directly.

3. Cloudflare edge TLS handshake failure (*.home.zakobar.com)

Symptom:

TLS connect error: error:0A000410:SSL routines::ssl/tls alert handshake failure

Cause: The domain was originally configured as home.zakobar.com (base domain), making all services two levels deep: git.home.zakobar.com, auth.home.zakobar.com, etc. Cloudflare's free Universal SSL certificate covers only one level of wildcard: *.zakobar.com. It does not cover *.home.zakobar.com (two levels). The Cloudflare edge had no certificate to present to browsers for these hostnames, causing a TLS handshake failure before the request ever reached the tunnel.

Fix: Move all services to single-level subdomains under zakobar.com (git.zakobar.com, auth.zakobar.com, etc.). In the NixOS config this required only one line change — the domain field in flake.nix:

domain = "zakobar.com";   # was "home.zakobar.com"

All modules reference homeyConfig.domain and updated automatically on rebuild. Tunnel hostnames and DNS records in the Cloudflare dashboard were updated to match.

4. ERR_TOO_MANY_REDIRECTS via tunnel

Symptom: Browser shows ERR_TOO_MANY_REDIRECTS when accessing any service through the Cloudflare tunnel.

Cause: cloudflared was talking to Caddy over plain HTTP (http://localhost:80). Caddy's default behaviour is to issue a 308 permanent redirect from HTTP to HTTPS for any hostname that has a matching HTTPS vhost. cloudflared followed the redirect back to http://localhost:80, which redirected again, indefinitely.

Fix: Add explicit http:// vhost entries in caddy.nix for every service. When Caddy has an explicit HTTP vhost for a hostname, it serves it directly without redirecting:

"git.${domain}" = {
  extraConfig = "reverse_proxy localhost:3000";
};
"http://git.${domain}" = {       # ← suppresses HTTP→HTTPS redirect
  extraConfig = "reverse_proxy localhost:3000";
};

Adding a New Service

To expose a new service through the tunnel:

1. Create modules/services/<name>.nix following the module pattern.

2. Add both a plain and http:// vhost in modules/caddy.nix:

   "<name>.${domain}" = {
     extraConfig = "reverse_proxy localhost:<port>";
   };
   "http://<name>.${domain}" = {
     extraConfig = "reverse_proxy localhost:<port>";
   };

3. No DNS or tunnel changes needed — the wildcard CNAME and wildcard tunnel rule (*.zakobar.com) cover new subdomains automatically.
4. Rebuild and switch: sudo nixos-rebuild switch --flake .#pi-main

Certificate Details

Let's Encrypt cert (LAN access)

• Issued per-hostname by Caddy via DNS-01 ACME using the Cloudflare API.
• Covers each hostname individually (e.g. git.zakobar.com).
• Stored in /var/lib/caddy/.local/share/caddy/certificates/.
• Used only when accessing services directly on the LAN (bypassing tunnel).
• Auto-renewed by Caddy.

Cloudflare Universal SSL cert (tunnel / remote access)

• Issued by Google Trust Services for *.zakobar.com.
• Managed entirely by Cloudflare — no action required on the Pi.
• Covers all single-level subdomains (git.zakobar.com, auth.zakobar.com, etc.).
• Does not cover two-level subdomains (git.home.zakobar.com) — this was the root cause of problem #3 above.

Quick Reference: Debugging Checklist

Symptom	Where to look	Command
502 Bad Gateway	cloudflared logs	journalctl -u cloudflared-tunnel -n 50
502 Bad Gateway	Caddy → backend	curl http://localhost:<port>/
TLS internal error	SNI / cert issue	curl -sv --resolve host:443:127.0.0.1 https://host/
Too many redirects	HTTP vhost missing	check http:// entries in caddy.nix
Handshake failure at edge	Cloudflare cert scope	check SSL/TLS → Edge Certificates
Token appears invalid	plugin version	check caddy-dns/cloudflare version vs token format
Caddy won't start	token / config error	journalctl -u caddy --since "5 min ago"