feat: route project instances over the edge network

Project preview instances (gig-platform, gig-app) publish no host ports; they join an external `edge` Docker network under instance-qualified aliases. - Caddy joins `edge` and loads per-project sites from sites/*.caddy. It is the LAN entry: split DNS → Caddy → instance, by hostname label. - New preview tunnel, `cloudflared-preview`, on `edge` only. It routes public instance hostnames straight to their aliases, so public previews do not depend on Caddy, and the connector cannot reach Caddy, the host or LAN services. The infra tunnel keeps fronting the host's own services via Caddy and has no route to instances. - sites/gig-platform.caddy and sites/gig-app.caddy mirror gig-platform's docs/deploy/caddy/. - The old host-port gig blocks move to sites/legacy-gig.caddy, to delete at gig-platform's main cutover. - Global client-IP trust for requests arriving through cloudflared. Validated with the custom Cloudflare-DNS build (`caddy validate`). Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MTExEkPpPMV4gSdfHpztra
2026-09-15 07:43:44 +08:00
parent 64c194a5d8
commit 6379e8c147
7 changed files with 217 additions and 28 deletions
@@ -2,6 +2,9 @@
 CLOUDFLARE_API_TOKEN=your_cloudflare_api_token_here
 ACME_EMAIL=[email protected]

-# Cloudflare Tunnel Token (From Cloudflare Zero Trust Dashboard -> Networks -> Tunnels)
+# Cloudflare Tunnel tokens (Zero Trust Dashboard -> Networks -> Tunnels).
+# Infra tunnel: the host's own services, through Caddy.
 CLOUDFLARE_TUNNEL_TOKEN=your_cloudflare_tunnel_token_here
+# Preview tunnel: project instances on the `edge` network, without Caddy.
+CLOUDFLARE_PREVIEW_TUNNEL_TOKEN=your_cloudflare_preview_tunnel_token_here

@@ -1,5 +1,13 @@
 {
    email {$ACME_EMAIL:[email protected]}
+
+    servers {
+        # cloudflared and Docker's gateways sit on private ranges. Take the real
+        # client IP from their headers instead of reporting their own address;
+        # upstreams receive it in X-Forwarded-For.
+        trusted_proxies static private_ranges
+        client_ip_headers Cf-Connecting-Ip X-Forwarded-For
+    }
 }

 # Reusable TLS block using Cloudflare DNS-01 ACME challenge (Let's Encrypt / ZeroSSL)
@@ -26,26 +34,6 @@ blog.kucial.com {
    reverse_proxy host.docker.internal:3201
 }

-gig-api.kucial.com {
-    import cloudflare_tls
-    reverse_proxy host.docker.internal:8088
-}
-
-gig-static.kucial.com {
-    import cloudflare_tls
-    reverse_proxy host.docker.internal:8088
-}
-
-gig-db.kucial.com {
-    import cloudflare_tls
-    reverse_proxy host.docker.internal:3100
-}
-
-gig-web.kucial.com {
-    import cloudflare_tls
-    reverse_proxy host.docker.internal:3101
-}
-

 # Gitea on Z2Pro
 gitea.kucial.com {
@@ -91,3 +79,8 @@ gitea.kucial.com {

    respond 404
 }
+
+# Per-project sites, one file each (sites/*.caddy). Project instances are
+# reached over the shared `edge` Docker network by container alias rather than
+# host ports — see sites/gig-platform.caddy.
+import sites/*.caddy
@@ -5,15 +5,55 @@ This architecture provides **End-to-End Dual TLS Encryption**:
 2. **Origin TLS (Let's Encrypt / ACME Certificate):** Caddy automatically requests and renews valid SSL/TLS certificates using the Cloudflare DNS-01 ACME challenge plugin.
 3. **Tunnel Transport:** Cloudflare Tunnel (`cloudflared`) connects to Caddy over **`HTTPS://caddy:443`** with strict origin TLS verification.

+That describes the **infra tunnel**, for this host's own services. Project preview instances use a second, **preview tunnel** that bypasses Caddy:
+
+```text
+public, infra:    Client → Cloudflare → cloudflared         → Caddy :443 → service
+public, preview:  Client → Cloudflare → cloudflared-preview → (edge) instance container
+LAN:              Client → split DNS  → Caddy :443 → (edge) instance container
+```
+
 ---

 ## 📁 File Structure

- [`Dockerfile`](file:///Users/kongkx/services/caddy/Dockerfile): Builds Caddy with `github.com/caddy-dns/cloudflare` using `xcaddy`.
- [`docker-compose.yml`](file:///Users/kongkx/services/caddy/docker-compose.yml): Runs `caddy` and `cloudflared` on the shared Docker bridge network.
- [`Caddyfile`](file:///Users/kongkx/services/caddy/Caddyfile): Configures `synotrip.kucial.com` & `blog.kucial.com` with DNS-01 TLS challenge.
- [`.env`](file:///Users/kongkx/services/caddy/.env): Stores `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_TUNNEL_TOKEN`.
- [`data/`](file:///Users/kongkx/services/caddy/data): Persistent volume storing Caddy's Let's Encrypt certificates.
+- `Dockerfile`: Builds Caddy with `github.com/caddy-dns/cloudflare` using `xcaddy`.
+- `docker-compose.yml`: Runs `caddy`, the infra tunnel `cloudflared`, and the preview tunnel `cloudflared-preview`. Caddy and `cloudflared-preview` join the external `edge` network shared with project instances; `cloudflared-preview` joins nothing else.
+- `Caddyfile`: Global options (ACME email, client-IP trust), the `cloudflare_tls` snippet, standalone sites, and `import sites/*.caddy`.
+- `sites/`: One file per project, mounted read-only at `/etc/caddy/sites`.
+  - `gig-platform.caddy`: gig-platform instances — `main` at `gig-platform.dev.kucial.com`, `e2e` / `pr-<N>` by hostname label, and main's permission gates on `studio.` / `minio.` / `horizon.`.
+  - `gig-app.caddy`: gig-app web instances — `main` at `gig-app.dev.kucial.com`, `pr-<N>` by label.
+  - `legacy-gig.caddy`: the old gig preview on host ports. **Delete it at gig-platform's main cutover.**
+- `.env`: Stores `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_TUNNEL_TOKEN` and `CLOUDFLARE_PREVIEW_TUNNEL_TOKEN`.
+- `data/`: Persistent volume storing Caddy's certificates.
+
+`sites/gig-platform.caddy` and `sites/gig-app.caddy` are copies of gig-platform's `docs/deploy/caddy/`; change them there and copy them over, so the two stay identical.
+
+---
+
+## 🧩 Project instances (`edge` network)
+
+Projects deployed as instances (gig-platform, gig-app) publish **no host ports**. Each instance's entry container joins the external `edge` network under an instance-qualified alias — `gig-platform-<instance>-nginx`, `gig-app-<instance>-web`. Two things reach those aliases: Caddy, for LAN clients, through a site file:
+
+```caddy
+*.gig-platform.dev.kucial.com {
+    reverse_proxy gig-platform-{http.request.host.labels.4}-nginx:80
+}
+```
+
+- The apex needs its own site: labels count from the right, so `labels.4` is empty there.
+- A hostname whose instance does not exist returns 502.
+- Placeholder upstreams share health state across instances, so do not add health checks to those sites.
+
+…and `cloudflared-preview`, for public clients, through the preview tunnel's public hostnames (below). It only ever reaches the named hostnames listed there, so an instance is public exactly when one of its hostnames is on that list.
+
+Create the network once, before the first `docker compose up`:
+
+```bash
+docker network create edge
+```
+
+Split DNS on the router must resolve both the apex and the wildcard of each project domain to this host, e.g. dnsmasq `address=/gig-platform.dev.kucial.com/<mac-mini-ip>`.

 ---

@@ -21,10 +61,12 @@ This architecture provides **End-to-End Dual TLS Encryption**:

 ### 1. In [`.env`](file:///Users/kongkx/services/caddy/.env)
 - `CLOUDFLARE_API_TOKEN`: Token with `Zone:DNS:Edit` permission (allows Caddy to complete the DNS-01 challenge).
- `CLOUDFLARE_TUNNEL_TOKEN`: Token from Cloudflare Zero Trust Tunnels.
+- `CLOUDFLARE_TUNNEL_TOKEN`: Token of the **infra** tunnel.
+- `CLOUDFLARE_PREVIEW_TUNNEL_TOKEN`: Token of the **preview** tunnel. Create it as a separate tunnel in Zero Trust.

 ### 2. In Cloudflare Zero Trust Dashboard (Public Hostnames)
-Add your subdomains under your Tunnel's **Public Hostnames**:
+
+**Infra tunnel** — the host's own services, through Caddy:

 | Public Hostname | Service Type | URL | Origin Server Name (under Additional Settings -> TLS) |
 | :--- | :--- | :--- | :--- |
@@ -34,6 +76,21 @@ Add your subdomains under your Tunnel's **Public Hostnames**:
 > [!NOTE]
 > Setting **Origin Server Name** tells `cloudflared` which SNI hostname to send to Caddy during the TLS handshake, allowing Caddy to serve the matching Let's Encrypt certificate.

+**Preview tunnel** — project instances, straight to their `edge` aliases. Plain `HTTP`: the hop is inside the Docker network, and Cloudflare terminates TLS.
+
+| Public Hostname | Service Type | URL |
+| :--- | :--- | :--- |
+| `gig-platform.dev.kucial.com` | `HTTP` | `gig-platform-main-nginx:80` |
+| `studio.gig-platform.dev.kucial.com` | `HTTP` | `gig-platform-main-db-studio-gate:80` |
+| `minio.gig-platform.dev.kucial.com` | `HTTP` | `gig-platform-main-minio-console-gate:80` |
+| `horizon.gig-platform.dev.kucial.com` | `HTTP` | `gig-platform-main-horizon-gate:80` |
+| `gig-app.dev.kucial.com` | `HTTP` | `gig-app-main-web:80` |
+
+- Route the internal surfaces only to their **gates**, never to `db-studio`, `minio:9001` or anything else. The gates enforce the platform permission on every request.
+- Never publish `*.gig-platform.dev.kucial.com` or `*.gig-app.dev.kucial.com`: PR and e2e instances are LAN-only, reached through split DNS.
+- Enable WebSockets for the zone; gig-platform serves them at `/app/` on the instance hostname.
+- A public hostname must not exist on both tunnels, or traffic goes to whichever connector wins.
+
 ### 3. Start Services

 ```bash
@@ -46,4 +103,15 @@ docker compose up -d --build
 docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile
 ```

+### 5. gig-platform main cutover

+When gig-platform's main instance is migrated (gig-platform `docs/deploy/instances.md`):
+
+```bash
+rm sites/legacy-gig.caddy
+docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile
+```
+
+Then remove the `gig-api` / `gig-static` / `gig-db` / `gig-web` `.kucial.com` public hostnames from the infra tunnel.
+
+A new `sites/*.caddy` file only needs a reload. Changing the `sites` mount or the networks in `docker-compose.yml` needs `docker compose up -d caddy`. The preview tunnel's hostnames change in the dashboard only; `cloudflared-preview` picks them up without a restart.
@@ -13,12 +13,21 @@ services:
      - CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN}
    extra_hosts:
      - "host.docker.internal:host-gateway"
+    networks:
+      - default
+      # The LAN entry to project instances (split DNS → Caddy → instance).
+      # Instances join `edge` under instance-qualified aliases and publish no
+      # host ports.
+      - edge
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
+      - ./sites:/etc/caddy/sites:ro
      - ./data:/data
      - ./config:/config
      - ./site:/srv

+  # Infra tunnel: public hostnames of the host's own services, all through
+  # Caddy (https://caddy:443). Cannot reach project instances.
  cloudflared:
    image: cloudflare/cloudflared:latest
    container_name: cloudflared
@@ -31,3 +40,30 @@ services:
    depends_on:
      - caddy

+  # Preview tunnel: public hostnames of project instances (gig-platform main,
+  # its gates, gig-app main), routed straight to their aliases on `edge` —
+  # e.g. http://gig-platform-main-nginx:80. Cloudflare terminates TLS.
+  #
+  # A separate tunnel and connector on purpose:
+  #   - public previews do not depend on Caddy, and restarting or
+  #     reconfiguring one tunnel never touches the other;
+  #   - `edge` is its only network, with no host-gateway, so this connector
+  #     cannot reach Caddy, the host or LAN services, and a dashboard mistake
+  #     on this tunnel can only ever expose something already on `edge`;
+  #   - the infra tunnel, in turn, has no route to instances.
+  # PR and e2e instances stay LAN-only: never publish a wildcard on it.
+  cloudflared-preview:
+    image: cloudflare/cloudflared:latest
+    container_name: cloudflared-preview
+    restart: unless-stopped
+    command: tunnel --no-autoupdate run
+    environment:
+      - TUNNEL_TOKEN=${CLOUDFLARE_PREVIEW_TUNNEL_TOKEN}
+    networks:
+      - edge
+
+networks:
+  # Created once, outside any project, because every project attaches to it:
+  #   docker network create edge
+  edge:
+    external: true
@@ -0,0 +1,14 @@
+# gig-app web SPA instances (gig-app .gitea/workflows/deploy-web.yml). Copy to
+# the infra Caddy's sites/ directory. Same shape as gig-platform.caddy: the
+# apex is main, the wildcard routes pr-<N> by label.
+
+gig-app.dev.kucial.com {
+	import cloudflare_tls
+	reverse_proxy gig-app-main-web:80
+}
+
+*.gig-app.dev.kucial.com {
+	import cloudflare_tls
+	# No health checks: placeholder upstreams share health state.
+	reverse_proxy gig-app-{http.request.host.labels.4}-web:80
+}
@@ -0,0 +1,42 @@
+# gig-platform instances. Copy to the infra Caddy's sites/ directory.
+#
+# Every instance's nginx joins the external `edge` network as
+# gig-platform-<instance>-nginx; so do main's permission gates. See
+# docs/deploy/instances.md.
+
+# main, at the apex. Its own site, not a case of the wildcard below: Caddy
+# indexes host labels from the right, so {http.request.host.labels.4} is
+# `pr-57` for pr-57.gig-platform.dev.kucial.com but empty for the apex, which
+# would proxy to `gig-platform--nginx`.
+gig-platform.dev.kucial.com {
+	import cloudflare_tls
+	reverse_proxy gig-platform-main-nginx:80
+}
+
+*.gig-platform.dev.kucial.com {
+	import cloudflare_tls
+
+	# main's internal surfaces. These labels are reserved: instance names are
+	# restricted to main, e2e and pr-<N>, so none can collide with them.
+	@studio host studio.gig-platform.dev.kucial.com
+	@minio host minio.gig-platform.dev.kucial.com
+	@horizon host horizon.gig-platform.dev.kucial.com
+	handle @studio {
+		reverse_proxy gig-platform-main-db-studio-gate:80
+	}
+	handle @minio {
+		reverse_proxy gig-platform-main-minio-console-gate:80
+	}
+	handle @horizon {
+		reverse_proxy gig-platform-main-horizon-gate:80
+	}
+
+	handle {
+		# Per-request placeholder upstream: e2e and pr-<N> instances. A name
+		# that resolves to nothing — a closed PR — returns 502.
+		#
+		# Placeholder upstreams share one upstream's health and load-balancing
+		# state across every instance, so never add health checks here.
+		reverse_proxy gig-platform-{http.request.host.labels.4}-nginx:80
+	}
+}
@@ -0,0 +1,33 @@
+# The old single gig-platform preview stack and gig-app web, on host ports.
+#
+# DELETE THIS FILE when gig-platform's main instance is migrated
+# (gig-platform docs/deploy/instances.md). The instance compose publishes no
+# 8088 or 3101, so after cutover these upstreams no longer exist; their
+# replacements are gig-platform.caddy and gig-app.caddy.
+#
+# Kept separate so the current preview keeps working until then, and cutover
+# is one `rm` plus a reload.
+
+gig-api.kucial.com {
+	import cloudflare_tls
+	reverse_proxy host.docker.internal:8088
+}
+
+# Replaced by $APP_URL/objects/<bucket> on each instance.
+gig-static.kucial.com {
+	import cloudflare_tls
+	reverse_proxy host.docker.internal:8088
+}
+
+# Proxies straight to Studio's break-glass port, bypassing db-studio-gate.
+# Replaced by studio.gig-platform.dev.kucial.com, which goes through the gate.
+gig-db.kucial.com {
+	import cloudflare_tls
+	reverse_proxy host.docker.internal:3100
+}
+
+# Replaced by gig-app.dev.kucial.com.
+gig-web.kucial.com {
+	import cloudflare_tls
+	reverse_proxy host.docker.internal:3101
+}