Aromba — When Things Break

The rule: the user’s server keeps taking care of itself no matter what happens to us, to Telegram, or to the AI provider. Chores, watchers, the Warden and the receipt never depend on anything outside the server. Only the chat and the dashboard do.

1. Design that makes this true

Local schedule. The scheduler, chore state and approvals live in SQLite on the server. No relay call is needed to run a night.
Local queue. Every outbound message (alert, receipt, reply) is written to the local queue first, then sent. If the relay is unreachable, it queues locally and flushes in order when the connection returns; messages older than 24 h collapse into one summary line so the user doesn’t get a flood — alerts fold into that line as a count with their worst value, never silently. The queue holds ten thousand messages; past that the oldest ordinary messages go first, alerts last.
Reconnect forever. The daemon reconnects to the relay with backoff up to 5 minutes, indefinitely. No human action needed.
Terminal is the root. aromba status, aromba receipt, aromba panic, aromba resume, aromba pair work with no network at all.
Warden watches the Worker; systemd watches both. Restart=always on both services; the Warden also restarts the Worker itself if it stops answering the local socket for 60 s.
Signed everything. Binary, updates, remote-freeze signal. A bad signature is treated as an attack (ladder step 3), not as a glitch.

2. Failure table

What breaks	User sees	Keeps working	Recovery	We do
aro.mba relay down	nothing at first; after 10 min offline the bot can’t deliver, so nothing arrives. Dashboard shows “relay unreachable, last sync HH:MM”.	chores, watchers, receipts (queued), terminal	daemon reconnects; queue flushes; one “I was offline 03:10–04:20, here’s what happened” message	status page updated within 10 min; second relay instance takes over (phase 3+); post-incident note in the changelog
Telegram Bot API down	no messages; relay shows the bot unreachable	everything on the server; dashboard live (phase 4)	relay retries; messages delivered in order when Telegram returns	nothing to do but wait; status page notes it
AI provider down or slow	“The model is unreachable: …. I’ll keep trying. The chores and the watchers are unaffected.” Then, after half an hour, “still unreachable”; and one line when it answers again. Said once per outage, never once per question	everything except the assistant	the daemon retries by itself: one small call every minute for 30 min, then hourly, until it answers; user can switch provider with `aromba key`	—
User’s key invalid / out of credit	“Your AI key stopped working (provider says: …). `aromba key` to replace.”	everything else	user replaces the key	Models page explains common provider errors
User’s calendar unreachable, or its address reset (`calendar` skill)	“Calendar work unreachable since 10:15; nothing is held for it until it answers.” Or: “Calendar work: its address stopped working. `aromba calendar remove work`, then `add`, on the server.”	everything; the last good copy counts for 2 hours, then nothing is held for the calendar	the Worker retries every 15 minutes; the user sets a new address	—
A reminder cannot reach the chat (relay or Telegram down)	nothing then; the next back-online line counts what expired	the calendar itself: items, holds and the demo check run on the server	a reminder and a demo check carry the moment they stop being worth sending; the queue here and the relay’s both drop them then, instead of sending them late	the relay honours that deadline in its own queue
User’s server offline for days/weeks	on return: one message “vultr-fra-01 is back after 12 days. Skipped 12 nights; running tonight. 3 alerts while away: …”	—	night chores run on the next quiet window, not on boot; probation clock pauses while offline	—
Server IP changed / rebuilt from snapshot	works: pairing is by server ID and relay token, not IP. A restore from an image with the same ID and a running original → both connect; relay flags a duplicate and asks in the chat which one to keep	—	user picks; the other gets `aromba pair`	duplicate detection in the relay
Update broke the daemon	systemd counts three failed starts of either service within ten minutes and runs the rollback unit with the previous release’s Warden binary — so a broken new one cannot block its own rollback; a single crash sets nothing off, and on a confirmed release nothing is rolled back or restarted (PLAN D102); the rollback stops both services, the previous release comes back, the Worker sends “Update 0.4.1 failed to start; rolled back to 0.4.0.” A release that ran for five minutes with the Worker talking to the Warden is confirmed and never rolled back by a later crash loop.	everything, on the old version	automatic; the user is told; auto-update pauses until the next release	staged rollout halts when rollback reports rise (with reports on, a server reports its rollback once, for the version that failed); fix, re-release
Update broke a chore	chore fails, retries once, then disables itself: “Docker cleanup failed twice after update; disabled. Undo available for last night.”	all other chores	user re-enables after the fix; `undo` restores if something was changed	chore tests in CI per OS; hotfix release
Our release key compromised	daemons refuse unsigned or wrongly signed updates; remote freeze can’t be forged	everything	we publish a key-rotation release signed with the offline backup key; daemons accept the rotation only through that path	rotate, disclose in the changelog and SECURITY.md within 24 h
Relay database lost	dashboard history gone (phase 4); pairing intact if the pairing table is restored from backup	everything on the server; chat resumes after pairing table restore	relay restores from the last backup (hourly); if pairing is lost, every user gets “Please re-pair: run `aromba pair`” once	hourly encrypted backups off-box; restore drill monthly
Warden crashes	Worker can’t execute anything privileged; assistant answers read-only; bot: “My guard process restarted; nothing privileged ran for 40 s.”	reads, chat	systemd restarts it within seconds; Worker re-handshakes	crash report if opted in
Worker crashes	a gap in the chat; Warden restarts it	scheduled chores already handed to the Warden finish; new ones wait	automatic	—
Server’s own disk full (100%)	Aromba’s emergency path: journal to a 1 MB reserved ring; `disk-95` event fires the emergency cleanup (green only); alert “Disk 100%: freed 2.1 GB from caches; needs you for the rest.”	watchers, alerts	user acts on the alert	—
Clock badly off on the server	schedules drift; certificates look expired	—	time-sync fix chore (green) runs; alert if it can’t	—
DNS broken on the server	can’t reach relay or provider	local everything	daemon tries the relay by IP fallback baked into the binary; alert via terminal only	—
Pairing link leaked before use	whoever taps first owns the chat	—	token is single-use and 15 min; `aromba pair` from the terminal invalidates it and re-pairs	—
Hub or catalog down	skills can’t be browsed or installed; installed skills unaffected	everything installed	wait	status page
We disappear (project abandoned)	nothing changes on the server; chat stops when the relay stops	chores, watchers, receipts, terminal, forever	user runs the open-source relay themselves: `aromba relay --self-hosted`, then `aromba pair` against it	the relay is open source from day one for exactly this reason

3. Degraded-mode messages

One message per transition, never repeated:

Offline → online: “Back online. Offline 03:10–04:20. Ran: Docker cleanup (36 GB). Skipped: nothing. Alerts while away: 1 (disk 91%, now 78%).”
Provider lost: “AI is unreachable; chores unaffected.” Provider back: nothing — the next answer just works.
Rollback: “Update X failed to start; rolled back to Y. Auto-update paused until the next release.”
Duplicate server: “Two servers claim to be vultr-fra-01. Which one is real?” [This one, started 10:04] [The other, started 03:12]

4. Our side — operations

Status page at status.aro.mba (static, updated by hand or by the relay’s own health check): relay, Telegram delivery, hub, dashboard.
Monitoring the relay with Aromba itself. The relay VPS runs Aromba; James’s phone is its chat. Dogfooding is the monitoring.
Backups: relay Postgres hourly, encrypted, to a different provider; monthly restore drill; the pairing table is the only thing that must never be lost.
Second relay from phase 3: active-passive, DNS failover, shared Postgres. Before that, ✅ a documented 15-minute rebuild from the repo — relay/deploy/README.md, “Rebuilding from nothing”: what you must have (the bot token, the newest dump, DNS), four commands, and what is lost (at most the hour since the last dump). The restore is a script, restore.sh, and make test-deploy runs it on every run against a real dump: a row written, dumped, deleted, put back, and the relay running again. A backup nobody restores is the one that has been broken since March.
Growth triggers for the relay box: 500 paired servers → 2 vCPU / 4 GB; 5,000 → Postgres on its own instance; relay disk at 70% → an alert through the Aromba that runs on the relay itself.
Staged rollout with automatic halt (05-security.md section 11) is the main defense against our own mistakes.
Incident notes in the changelog for anything that affected users: what, how long, what changed. Same day.
Support: GitHub issues and the Telegram group. No email support in phase 1–3.

5. What we test before each release

Kill the relay for an hour with 20 test daemons: queues flush in order, one summary each.
Kill Telegram delivery (fake API returning 5xx): no message loss.
Provider returns 500 / timeout / invalid key: correct messages, chores unaffected.
Ship a deliberately broken binary to a test server: rollback within 3 minutes, message sent.
Fill a test server’s disk to 100%: emergency path frees space and alerts.
Take a test server offline for 3 days: return message correct, night chores run on the next window.