---
name: thermograph-ops
description: Deploying, monitoring and debugging the Thermograph estate — release flow, reading the fleet, and what to do when something breaks.
---

# Thermograph operations

## Release flow

Code stages by branch, and the branch decides the environment:

```
PR into dev  →(required checks)→  dev  →  no live target (dev's environment retired 2026-08-31)
                                  dev  →(promotion PR)→ main → testing (central1)
                                                        main →(promotion PR)→ release → prod (prod1)
```

`dev` the branch is unaffected by the environment's retirement — it is still
where feature PRs land first, same as always. It just no longer deploys
anywhere on its own; only the promotion to `main` reaches a live host.

Promotion is continuous and per-decision — there is no release calendar. Size a
promotion to whatever has actually soaked on the environment below.

Branch protection blocks direct pushes to `dev`/`main`/`release` for everyone.
OneDev, like Forgejo before it, has no auto-merge-on-green, so merges are
explicit.

**Doing the git work.** OneDev at `code.jinemi.com` is the forge of record —
Forgejo and its `tea` CLI were retired outright on 2026-08-31, and neither
resolves any more. Project paths are hierarchical (`Jinemi/thermograph`, never
a bare `thermograph`), and the REST API is under `/~api`. Use Centralis's native
`onedev_*` tools for reads and ordinary PR work: `onedev_prs`, `onedev_pr_open`,
`onedev_ci`, `onedev_status`, `onedev_pr_await` (wait on CI), `onedev_pr_update`
(replay a PR onto a moved base), `onedev_branch_protection` (rule options).
`onedev_pr_merge` merges ordinary OneDev PRs, but deliberately carries no
promotion-aware merge verb — it does not know about the promotion guard,
fast-forward-only enforcement, or the owner-only `main` → `release` rule below,
so it is not a shortcut around any of them.

### Who may merge what

Everything is a PR; that is about how a change travels, not who lands it. Who
lands it is three tiers, and they are not symmetric:

| Hop | Whose call | What to do |
|---|---|---|
| anything → `dev` | yours | Merge it. Do not leave outstanding PRs — an open PR into `dev` is unfinished work, not caution. |
| `dev` → `main` (testing) | yours, batched | Promote when a batch is coherent enough to be worth *testing*. Testing is the test environment; not promoting is not testing. |
| `main` → `release` (prod) | **the owner's** | Prepare it, recommend for or against, and ask. Never unilaterally, and never by another route. |

There is no `promote` tool any more — it died with Forgejo along with the rest
of the TypeScript server, and nothing OneDev-aware replaced it. The three-tier
rule above is now held by discipline, not enforced by a tool: `onedev_pr_open`
and `onedev_pr_merge` will happily act on any PR that OneDev's own permissions
allow, so nothing in Centralis's tool surface stops a `main` → `release` merge
that should have gone to the owner first. Preparing a promotion — what would
ship, the CI evidence from `onedev_ci` / `onedev_pr_await`, how long it soaked
on testing — and recommending for or against is still useful groundwork;
landing the `dev` → `main` and `main` → `release` hops themselves is a manual,
operator-performed step outside Centralis's own tool surface. "Ready, but I
would wait, because X" is still a legitimate and useful answer to hand back.

**Commit counts are not deliverable content.** There is no tree-compare tool any
more — `branch_diff` died with `promote` and nothing OneDev-aware replaced it —
so checking whether two tips' *trees* actually differ, not just how many
commits separate them, means a manual `git diff` from a checkout. Identical
trees mean the same changes are already on the target under different SHAs — a
promotion there merges nothing while still firing the target's deploy
workflows, including `infra-sync` re-rendering the rendered env file on prod
and testing (one file per environment — `/etc/thermograph.env` and
`/etc/thermograph-testing.env`). Check before promoting; the ahead/behind
numbers will look like real work.

**Promotions should stay fast-forward-only, but nothing enforces that any
more.** `dev`, `main` and `release` were reconciled on 2026-08-02 so that each
is a strict ancestor of the next; that discipline is still the goal. What
changed is the guard: `promote` and `forge_pr_merge`, which used to refuse a
squash or a rebase on a promotion, are both gone, and OneDev's merge API takes
no per-call style — a project carries one configured default merge strategy
instead. Nothing stops a squash or a merge commit from being picked for a
promotion by hand; choosing `fast-forward-only` deliberately, every time, is
now the only thing keeping the chain from re-diverging the way it did before
the 2026-08-02 reconciliation.

Verify rather than assume, before and after a promotion:

```sh
git merge-base --is-ancestor origin/main origin/dev      # dev -> main ff-able
git merge-base --is-ancestor origin/release origin/main  # main -> release ff-able
```

If a promotion is refused as not fast-forwardable, something moved the target
out of line — a hotfix landed straight on it, most likely. The fix is a
down-merge of the target back into the source, never a rebase or a force-push of
a protected branch.

**Image tags are keyed to the last commit that touched a domain**
(`git log -1 -- backend/`), not the branch tip. Both build and deploy compute
the same key, so an infra-only commit at the tip cannot send a deploy chasing an
image no build produced. This is why `deployed_version` can show a tag that is
not the head of the branch — that is correct, not drift.

Rollback is redeploying the previous image tag. Tags are per-commit and GC keeps
the running set.

## Reading the fleet

Start wide, then narrow:

1. `fleet_status` — HTTP reachability plus what is running on each host.
2. `logs_overview since="1h"` — where the volume is, and what has gone quiet.
3. `logs_query service=… host=…` — the actual lines.

**The Swarm trap.** Prod and testing both run Swarm stacks, one box each, and
their `job="docker"` streams carry **no `service` label**. A raw
`{host="prod", service="backend"}` query matches nothing and reads as "no
logs" when it means "wrong selector". Use the `logs_query` tool's `service`
argument, which rewrites it to a container name-regex with the churning task
suffix wildcarded. Prod service names are `thermograph_web`, `_worker`,
`_frontend`, `_db`, `_autoscaler`, `_lake`; testing's are the same shape under
`thermograph-testing_testing-*` (`testing-web`, `testing-worker`,
`testing-frontend`, `testing-lake`, `testing-daemon` — note the doubled
`testing`), **plus its own `thermograph-testing_db`**, which it gained at the
2026-08-17 cutover. (`dev`, still compose on vps1, used to carry a `service`
label instead — moot now that the environment is retired.)

Retention is 30 days. `thermograph-test_*` containers are rehearsal leftovers —
ignore them.

## Databases

No database is exposed over TCP. Each listens only on its private docker
network, and prod's and testing's are both Swarm **overlays their own hosts
cannot route to** — so `ssh -L` is *impossible* for either. `sql_query` execs
into the container instead, which works identically everywhere.

**Every environment now owns its database container.** That is new, and the
history is worth keeping: `beta` had none and borrowed prod's, and when the real
environment moved to its own TimescaleDB on central1 the borrowed pointer stayed
behind — so queries kept succeeding against a `thermograph_beta` database sitting
inside prod's instance that nothing had written to since the cutover. A wrong
`dbHost` does not fail; it answers.

Reads run as each environment's own read-only role (`thermograph_ro` for prod,
`thermograph_testing_ro` for testing — NOSUPERUSER, `pg_read_all_data`).
Read-only is enforced by Postgres, not by convention:

```
sql_query(env="prod", sql="create table t(i int)")
→ ERROR: permission denied for schema public
```

`write:true` escalates to the app's superuser role and takes effect immediately
with no confirmation. On prod, know what you are doing.

## The lake

`lake_query` runs DuckDB over the ERA5 Iceberg warehouse in Contabo object
storage — one shared warehouse readable from every environment, so `env` only
picks where the engine runs. Filter on partition columns (tile/year/month): the
scan is partition-pruned, and a predicate on them is the difference between a
fast query and reading the whole warehouse.

## When something breaks

1. `fleet_status` — is it up at all?
2. `logs_query host=prod contains="(?i)error|traceback"` since the incident.
3. `deployed_version env=prod` — did something ship just before it started?
4. `onedev_prs` / `onedev_ci` — what landed recently, and did CI actually pass?
5. Write down what you found with `notes_write` **before** fixing it. The
   reconstruction is worth more than the fix, and it will be gone by tomorrow.

For anything the typed tools do not cover, `run_on_host` gives a shell as the
`agent` user (passwordless sudo on the VPSes). Every call is audit-logged.

## Things that will bite you

- **Prod and testing container names change on every redeploy** (Swarm task
  suffix, on both). Never hardcode one; resolve via `docker ps --filter name=`
  at call time.
- **Contabo object storage is path-style only**, `region=default`. Virtual-host
  addressing fails outright.
- The **`backups/` prefix** in that bucket belongs to the nightly backup jobs.
  Do not write outside the lake's own prefix.
- **Terraform has no state anywhere.** It is executable documentation; a casual
  `terraform apply` would try to re-provision live prod from scratch.
- **Grafana dashboards are provisioned from repo JSON.** An edit through the UI
  or API is silently overwritten on the next provision — change them with
  `dashboard_write`, which opens a PR.
- Backups are a single copy on the same box as the database, with no offsite
  yet, and restores must handle TimescaleDB's `continuous_agg` circular-FK
  warning (`--disable-triggers`).