kongkxandClaude Opus 5 6379e8c147 feat: route project instances over the edge network
Project preview instances (gig-platform, gig-app) publish no host ports; they
join an external `edge` Docker network under instance-qualified aliases.

- Caddy joins `edge` and loads per-project sites from sites/*.caddy. It is the
  LAN entry: split DNS → Caddy → instance, by hostname label.
- New preview tunnel, `cloudflared-preview`, on `edge` only. It routes public
  instance hostnames straight to their aliases, so public previews do not
  depend on Caddy, and the connector cannot reach Caddy, the host or LAN
  services. The infra tunnel keeps fronting the host's own services via Caddy
  and has no route to instances.
- sites/gig-platform.caddy and sites/gig-app.caddy mirror gig-platform's
  docs/deploy/caddy/.
- The old host-port gig blocks move to sites/legacy-gig.caddy, to delete at
  gig-platform's main cutover.
- Global client-IP trust for requests arriving through cloudflared.

Validated with the custom Cloudflare-DNS build (`caddy validate`).

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MTExEkPpPMV4gSdfHpztra
2026-09-15 07:43:44 +08:00
2026-09-15 06:59:44 +08:00
2026-09-15 06:59:44 +08:00

Caddy Dual TLS + Cloudflare Tunnel Setup

This architecture provides End-to-End Dual TLS Encryption:

  1. Edge TLS (Cloudflare Edge Certificate): Cloudflare terminates HTTPS for public visitors on the internet.
  2. Origin TLS (Let's Encrypt / ACME Certificate): Caddy automatically requests and renews valid SSL/TLS certificates using the Cloudflare DNS-01 ACME challenge plugin.
  3. Tunnel Transport: Cloudflare Tunnel (cloudflared) connects to Caddy over HTTPS://caddy:443 with strict origin TLS verification.

That describes the infra tunnel, for this host's own services. Project preview instances use a second, preview tunnel that bypasses Caddy:

public, infra:    Client → Cloudflare → cloudflared         → Caddy :443 → service
public, preview:  Client → Cloudflare → cloudflared-preview → (edge) instance container
LAN:              Client → split DNS  → Caddy :443 → (edge) instance container

📁 File Structure

  • Dockerfile: Builds Caddy with github.com/caddy-dns/cloudflare using xcaddy.
  • docker-compose.yml: Runs caddy, the infra tunnel cloudflared, and the preview tunnel cloudflared-preview. Caddy and cloudflared-preview join the external edge network shared with project instances; cloudflared-preview joins nothing else.
  • Caddyfile: Global options (ACME email, client-IP trust), the cloudflare_tls snippet, standalone sites, and import sites/*.caddy.
  • sites/: One file per project, mounted read-only at /etc/caddy/sites.
    • gig-platform.caddy: gig-platform instances — main at gig-platform.dev.kucial.com, e2e / pr-<N> by hostname label, and main's permission gates on studio. / minio. / horizon..
    • gig-app.caddy: gig-app web instances — main at gig-app.dev.kucial.com, pr-<N> by label.
    • legacy-gig.caddy: the old gig preview on host ports. Delete it at gig-platform's main cutover.
  • .env: Stores CLOUDFLARE_API_TOKEN, CLOUDFLARE_TUNNEL_TOKEN and CLOUDFLARE_PREVIEW_TUNNEL_TOKEN.
  • data/: Persistent volume storing Caddy's certificates.

sites/gig-platform.caddy and sites/gig-app.caddy are copies of gig-platform's docs/deploy/caddy/; change them there and copy them over, so the two stay identical.


🧩 Project instances (edge network)

Projects deployed as instances (gig-platform, gig-app) publish no host ports. Each instance's entry container joins the external edge network under an instance-qualified alias — gig-platform-<instance>-nginx, gig-app-<instance>-web. Two things reach those aliases: Caddy, for LAN clients, through a site file:

*.gig-platform.dev.kucial.com {
    reverse_proxy gig-platform-{http.request.host.labels.4}-nginx:80
}
  • The apex needs its own site: labels count from the right, so labels.4 is empty there.
  • A hostname whose instance does not exist returns 502.
  • Placeholder upstreams share health state across instances, so do not add health checks to those sites.

…and cloudflared-preview, for public clients, through the preview tunnel's public hostnames (below). It only ever reaches the named hostnames listed there, so an instance is public exactly when one of its hostnames is on that list.

Create the network once, before the first docker compose up:

docker network create edge

Split DNS on the router must resolve both the apex and the wildcard of each project domain to this host, e.g. dnsmasq address=/gig-platform.dev.kucial.com/<mac-mini-ip>.


🚀 Setup & Cloudflare Dashboard Configuration

1. In .env

  • CLOUDFLARE_API_TOKEN: Token with Zone:DNS:Edit permission (allows Caddy to complete the DNS-01 challenge).
  • CLOUDFLARE_TUNNEL_TOKEN: Token of the infra tunnel.
  • CLOUDFLARE_PREVIEW_TUNNEL_TOKEN: Token of the preview tunnel. Create it as a separate tunnel in Zero Trust.

2. In Cloudflare Zero Trust Dashboard (Public Hostnames)

Infra tunnel — the host's own services, through Caddy:

Public Hostname Service Type URL Origin Server Name (under Additional Settings -> TLS)
synotrip.kucial.com HTTPS caddy:443 synotrip.kucial.com
blog.kucial.com HTTPS caddy:443 blog.kucial.com

Note

Setting Origin Server Name tells cloudflared which SNI hostname to send to Caddy during the TLS handshake, allowing Caddy to serve the matching Let's Encrypt certificate.

Preview tunnel — project instances, straight to their edge aliases. Plain HTTP: the hop is inside the Docker network, and Cloudflare terminates TLS.

Public Hostname Service Type URL
gig-platform.dev.kucial.com HTTP gig-platform-main-nginx:80
studio.gig-platform.dev.kucial.com HTTP gig-platform-main-db-studio-gate:80
minio.gig-platform.dev.kucial.com HTTP gig-platform-main-minio-console-gate:80
horizon.gig-platform.dev.kucial.com HTTP gig-platform-main-horizon-gate:80
gig-app.dev.kucial.com HTTP gig-app-main-web:80
  • Route the internal surfaces only to their gates, never to db-studio, minio:9001 or anything else. The gates enforce the platform permission on every request.
  • Never publish *.gig-platform.dev.kucial.com or *.gig-app.dev.kucial.com: PR and e2e instances are LAN-only, reached through split DNS.
  • Enable WebSockets for the zone; gig-platform serves them at /app/ on the instance hostname.
  • A public hostname must not exist on both tunnels, or traffic goes to whichever connector wins.

3. Start Services

docker compose up -d --build

4. Zero-Downtime Reload

docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile

5. gig-platform main cutover

When gig-platform's main instance is migrated (gig-platform docs/deploy/instances.md):

rm sites/legacy-gig.caddy
docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile

Then remove the gig-api / gig-static / gig-db / gig-web .kucial.com public hostnames from the infra tunnel.

A new sites/*.caddy file only needs a reload. Changing the sites mount or the networks in docker-compose.yml needs docker compose up -d caddy. The preview tunnel's hostnames change in the dashboard only; cloudflared-preview picks them up without a restart.

S
Description
No description provided
Readme
46 KiB
Languages
Dockerfile 100%