fleettickets/docs/deployment-and-operations.md

# Deployment & Operations — fleettickets

Operational runbook for the INC ingest pipeline as deployed on **Coolify**
(host `kianiadee@twala.rahamafresh.com`, key `~/.ssh/id_ed25519`). Covers the
container, environment, schedule, auto-deploy webhook, the source-bucket cutover
procedure, and verification. Secrets are referenced by **where to retrieve them**,
never by value.

## What's deployed

| Thing | Detail |
|---|---|
| Coolify app | **`fleettickets`** — id `15`, uuid `g14mwzo73q20g70vc6fzumya`, build pack `dockerfile`, git `main` |
| Container | built from this repo's `Dockerfile` (`python:3.12-slim`, `TZ=Africa/Nairobi`); kept alive with `tail -f /dev/null` (no web server) |
| Ingest | a Coolify **Scheduled Task** `inc_tickets` running `python import_tickets.py --from-bucket --apply` |
| DB | `tickets` schema in the shared `tracksolid_db` (internal host `timescale_db:5432`) |
| Source | **`isptickets`** S3 bucket, `automations/inc/changes/<EAT-ts>.csv` CDC stream (see `../n8n-s3-ticket-exports.md` and `../README.md`) |

Resolve the live container name (Coolify appends a random suffix):
```bash
ssh -i ~/.ssh/id_ed25519 kianiadee@twala.rahamafresh.com \
  'docker ps --filter name=g14mwzo73q20g70vc6fzumya --format "{{.Names}}" | head -1'
```

## Schedule (cron)

The Scheduled Task runs **`*/20 6-20 * * *`** — every 20 min, **06:00–20:40 EAT**.
Coolify evaluates task cron in the server timezone (`server_settings.server_timezone`
= `Africa/Nairobi`), so **no UTC conversion** — write EAT directly. The `--from-bucket`
run is a cheap no-op when no new change file has arrived (watermark guard), so a dense
schedule is safe.

To change the frequency, edit the task in the Coolify UI, or in `coolify-db`:
```sql
UPDATE scheduled_tasks SET frequency = '*/20 6-20 * * *', updated_at = now()
WHERE name = 'inc_tickets';   -- id 3
```
Coolify's scheduler re-reads `scheduled_tasks` each minute, so the change is picked up
without a redeploy. Execution history: `scheduled_task_executions`.

> The repo's `Dockerfile`, `run_ingest.sh`, and `README.md` document this same cron for
> the plain-host/VM fallback (`CRON_TZ=Africa/Nairobi`).

## Environment variables

Set on the Coolify app (Environment Variables). Names only — values live in Coolify:

| Var | Purpose |
|---|---|
| `DATABASE_URL` | `tracksolid_db` (internal `timescale_db:5432`) |
| `RUSTFS_ENDPOINT` | `https://s3.rahamafresh.com` |
| `RUSTFS_ACCESS_KEY` / `RUSTFS_SECRET_KEY` | `isptickets` bucket credentials |
| `RUSTFS_REGION` | `us-east-1` |
| `TICKETS_BUCKET` | `isptickets` |
| `GEOCODER_PROVIDER` / `GEOCODER_API_KEY` | keyed geocoder (LocationIQ/OpenCage) |

**Env vars are Laravel-encrypted in `coolify-db` — never raw-`UPDATE` them.** Change them
in the Coolify UI, or via `artisan tinker` (which re-encrypts on save):
```bash
ssh -i ~/.ssh/id_ed25519 kianiadee@twala.rahamafresh.com 'docker exec -i coolify php artisan tinker' <<'PHP'
$e = \App\Models\EnvironmentVariable::where('resourceable_type','App\\Models\\Application')
       ->where('resourceable_id',15)->where('key','TICKETS_BUCKET')->first();
$e->value = 'isptickets'; $e->save(); echo $e->value.PHP_EOL;
PHP
```
An env change only takes effect after the container is **recreated** (a redeploy — see below),
since Coolify injects env at container create time.

## Deploys

### Auto-deploy (Forgejo → Coolify webhook)

A push to `main` should auto-deploy. This needs **both** the Coolify per-app Auto-Deploy
toggle (Configuration → Advanced) **and** a webhook on the Forgejo repo. The webhook was
missing originally (the toggle alone is not enough); it now exists as hook id `3` on
`kianiadee/fleettickets`:

| Field | Value |
|---|---|
| URL | `https://stage.rahamafresh.com/webhooks/source/gitea/events/manual` |
| Type / content-type | `gitea` / `json` |
| Events / branch filter | `push` / `main` |
| Secret | the app's `manual_webhook_secret_gitea` (Coolify HMAC-validates `X-Hub-Signature-256`) |

Recreate / inspect it via the Forgejo API (auth: `git credential fill`, host
`repo.rahamafresh.com`, basic auth to `/api/v1` — no `tea`/`gh` needed). Get the secret by
decrypting it in Coolify:
```bash
ssh -i ~/.ssh/id_ed25519 kianiadee@twala.rahamafresh.com \
  "docker exec -i coolify php artisan tinker --execute=\"echo \\App\\Models\\Application::find(15)->manual_webhook_secret_gitea;\""
```
```bash
# list / test the webhook (USER:PASS from git credential fill)
curl -s -u "$USER:$PASS" https://repo.rahamafresh.com/api/v1/repos/kianiadee/fleettickets/hooks
curl -s -u "$USER:$PASS" -X POST https://repo.rahamafresh.com/api/v1/repos/kianiadee/fleettickets/hooks/3/tests
```
A successful test shows a webhook hit in `docker logs coolify` (no `invalid_signature`
audit) and a new row in `application_deployment_queues`.

### Manual deploy (no push)

Trigger the same action as Coolify's Deploy button via tinker:
```bash
ssh -i ~/.ssh/id_ed25519 kianiadee@twala.rahamafresh.com 'docker exec -i coolify php artisan tinker' <<'PHP'
$app = \App\Models\Application::where('uuid','g14mwzo73q20g70vc6fzumya')->first();
$uuid = new \Visus\Cuid2\Cuid2;
echo json_encode(queue_application_deployment(
  application: $app, deployment_uuid: $uuid, force_rebuild: false, is_api: true)).PHP_EOL;
echo $uuid.PHP_EOL;
PHP
```
Watch it: `SELECT id, status, created_at FROM application_deployment_queues WHERE
application_id = '15' ORDER BY created_at DESC LIMIT 3;` (note: `application_id` is the
**numeric id stored as text**).

## Source-bucket cutover (when the provider moves buckets)

If the provider moves the INC feed to a new bucket (as happened `tickets` → `isptickets`,
2026-06-25):

1. **Inspect** the new bucket (read-only) — confirm `automations/inc/changes/` layout,
   timestamp range, schema parity. CRQ (`automations/crq/`) stays out of scope.
2. **Update env** (UI or tinker): `RUSTFS_ACCESS_KEY`, `RUSTFS_SECRET_KEY`,
   `TICKETS_BUCKET` → the new bucket (endpoint usually unchanged).
3. **Reconcile the DB** to current. The loader drains every `changes/` file newer than the
   watermark (`tickets.import_meta.metadata.source_max_key`), oldest→newest, upserting on
   `ticket_id`:
   - If the watermark **predates** the new bucket's first file, a normal
     `--from-bucket --apply` drains the whole new stream — no reseed needed.
   - Otherwise use **`--reseed`** (ignores the watermark, drains all `changes/` once):
     `python import_tickets.py --from-bucket --reseed --apply` (see README "Bucket cutover").
   The new stream's periodic full-state re-emissions make this converge even across the
   cutover gap. Idempotent upserts + never-delete make it non-destructive.
   - For a one-off, you can run it in the live container with the new creds inlined:
     `docker exec -e TICKETS_BUCKET=… -e RUSTFS_ACCESS_KEY=… -e RUSTFS_SECRET_KEY=… <container>
     sh -c "cd /app && python import_tickets.py --from-bucket --apply"`.
4. **Re-geocode** new clusters/locations: `--geocode-clusters --apply` then
   `--geocode-locations --apply` (existing gazetteer persists; only new keys are looked up).
5. **Redeploy** so the Scheduled Task's container picks up the new env (push `main` → webhook,
   or manual deploy). Old bucket is left untouched for rollback.

## Verification

```bash
DB=$(docker ps --filter name=timescale_db --format "{{.Names}}" | head -1)
docker exec -i "$DB" psql -U postgres -d tracksolid_db <<'SQL'
-- watermark + freshness
SELECT export_type, records_ingested, ingested_at, metadata->>'source_max_key'
FROM tickets.import_meta WHERE dataset='inc';
-- counts
SELECT count(*) total_inc,
       count(*) FILTER (WHERE (raw->>'is_actionable')::boolean) AS open
FROM tickets.inc;
-- map payload sanity
SELECT reporting.fn_tickets_for_map() -> 'summary' ->> 'ticket_count';
SQL
```
- New bucket `changes/` empties as files move to `automations/inc/processed/`.
- A plain `--from-bucket --apply` reports "nothing new" until the next change file lands.
- FleetOps Tickets map freshness reflects the new `ingested_at`.

## Rollback

- **Bucket:** revert the three env vars to the old bucket + creds and redeploy. The old
  bucket and its `processed/` history are untouched; upserts are idempotent and rows are
  never deleted, so re-running is safe.
- **Cron:** `UPDATE scheduled_tasks SET frequency = <old> WHERE name='inc_tickets';`