---
name: thermograph-ops
description: Deploying, monitoring and debugging the Thermograph estate — release flow, reading the fleet, and what to do when something breaks.
---

# Thermograph operations

## Release flow

Code stages by branch, and the branch decides the environment:

```
PR into dev  →(required checks)→  dev  →  LAN dev box
                                  dev  →(promotion PR)→ main → beta
                                                        main →(promotion PR)→ release → prod
```

Promotion is continuous and per-decision — there is no release calendar. Size a
promotion to whatever has actually soaked on the environment below.

Branch protection blocks direct pushes to `dev`/`main`/`release` for everyone.
Forgejo has no auto-merge-on-green, so merges are explicit.

**Image tags are keyed to the last commit that touched a domain**
(`git log -1 -- backend/`), not the branch tip. Both build and deploy compute
the same key, so an infra-only commit at the tip cannot send a deploy chasing an
image no build produced. This is why `deployed_version` can show a tag that is
not the head of the branch — that is correct, not drift.

Rollback is redeploying the previous image tag. Tags are per-commit and GC keeps
the running set.

## Reading the fleet

Start wide, then narrow:

1. `fleet_status` — HTTP reachability plus what is running on each host.
2. `logs_overview since="1h"` — where the volume is, and what has gone quiet.
3. `logs_query service=… host=…` — the actual lines.

**The prod trap.** Prod runs a Swarm stack and its `job="docker"` streams carry
**no `service` label**. A raw `{host="prod", service="backend"}` query matches
nothing and reads as "no logs" when it means "wrong selector". Use the
`logs_query` tool's `service` argument, which rewrites it to a container
name-regex with the churning task suffix wildcarded. Prod service names are
`thermograph_web`, `_worker`, `_frontend`, `_db`, `_autoscaler`, `_lake`.

Retention is 30 days. `thermograph-test_*` containers are rehearsal leftovers —
ignore them.

## Databases

No database is exposed over TCP. Each listens only on its private docker
network, and prod's is a Swarm **overlay the prod host itself cannot route to** —
so `ssh -L` works for beta and is *impossible* for prod. `sql_query` execs into
the container instead, which works identically everywhere.

Reads run as `thermograph_ro` (NOSUPERUSER, `pg_read_all_data`). Read-only is
enforced by Postgres, not by convention:

```
sql_query(env="prod", sql="create table t(i int)")
→ ERROR: permission denied for schema public
```

`write:true` escalates to the app's superuser role and takes effect immediately
with no confirmation. On prod, know what you are doing.

## The lake

`lake_query` runs DuckDB over the ERA5 Iceberg warehouse in Contabo object
storage — one shared warehouse readable from every environment, so `env` only
picks where the engine runs. Filter on partition columns (tile/year/month): the
scan is partition-pruned, and a predicate on them is the difference between a
fast query and reading the whole warehouse.

## When something breaks

1. `fleet_status` — is it up at all?
2. `logs_query host=prod contains="(?i)error|traceback"` since the incident.
3. `deployed_version env=prod` — did something ship just before it started?
4. `forge_prs` / `forge_ci` — what landed recently, and did CI actually pass?
5. Write down what you found with `notes_write` **before** fixing it. The
   reconstruction is worth more than the fix, and it will be gone by tomorrow.

For anything the typed tools do not cover, `run_on_host` gives a shell as the
`agent` user (passwordless sudo on the VPSes). Every call is audit-logged.

## Things that will bite you

- **Prod container names change on every redeploy** (Swarm task suffix). Never
  hardcode one; resolve via `docker ps --filter name=` at call time.
- **Contabo object storage is path-style only**, `region=default`. Virtual-host
  addressing fails outright.
- The **`backups/` prefix** in that bucket belongs to the nightly backup jobs.
  Do not write outside the lake's own prefix.
- **Terraform has no state anywhere.** It is executable documentation; a casual
  `terraform apply` would try to re-provision live prod from scratch.
- **Grafana dashboards are provisioned from repo JSON.** An edit through the UI
  or API is silently overwritten on the next provision — change them with
  `dashboard_write`, which opens a PR.
- Backups are a single copy on the same box as the database, with no offsite
  yet, and restores must handle TimescaleDB's `continuous_agg` circular-FK
  warning (`--disable-triggers`).