fleetanalytics_mcp/pgbouncer/README.md

# PgBouncer for `timescale_db` (connection pooling)

> **Scope note:** this is **stack-wide infrastructure**, shared by every service that
> talks to `timescale_db` — it is only *parked* in the analytics-MCP repo because that
> is where the "too many connections" investigation happened. It arguably belongs in
> the backend/ingestion repo (`tracksolid_timescale_grafana_prod`). Move it there when
> convenient.

## Why

The DB runs at `max_connections = 100`. About nine services each keep a persistent
pool open — and several connect as the **`postgres` superuser**, holding connections
**idle for hours**. When those pools fill under load simultaneously, the sum crosses
~97 and new connections fail with `FATAL: sorry, too many clients already`.

PgBouncer fixes this structurally: clients connect to PgBouncer (cheap, thousands
allowed), and it multiplexes them onto a **small, fixed set** of real backend
connections. The DB's connection count then depends on the pool size you choose, not
on how many app pools exist.

```
9 app pools  ──▶  PgBouncer :6432  ──▶  ≤25 real backends  ──▶  timescale_db :5432
 (hundreds)        (transaction mode)
```

## Files

| File | Purpose |
|---|---|
| `pgbouncer.ini` | pooling + auth config (transaction mode, `auth_query`) |
| `auth_setup.sql` | creates `pgbouncer_auth` + `pgbouncer.user_lookup()` on the DB |
| `userlist.txt.example` | how to generate the real (gitignored) `userlist.txt` |
| `docker-compose.yml` | the PgBouncer service (join the DB network) |

## Deploy (once)

```bash
# 0) on the host, generate a password for the auth role
( umask 077; openssl rand -hex 24 > ~/.pgbouncer_auth.pw )

# 1) create the auth role + lookup function (as postgres superuser)
DB=$(docker ps --filter name=timescale_db --format '{{.Names}}' | head -1)
docker exec -i "$DB" psql -U postgres -d tracksolid_db -v ON_ERROR_STOP=1 \
  -v pgb_pw="$(cat ~/.pgbouncer_auth.pw)" < pgbouncer/auth_setup.sql

# 2) build userlist.txt from the stored verifier (formats always match this way)
docker exec -i "$DB" psql -U postgres -d tracksolid_db -tAc \
  "SELECT '\"pgbouncer_auth\" \"' || passwd || '\"' \
     FROM pg_shadow WHERE usename='pgbouncer_auth'" > pgbouncer/userlist.txt

# 3) set the real DB network name in docker-compose.yml (networks.dbnet.name), then:
docker compose -f pgbouncer/docker-compose.yml up -d
```

## Cut services over (incrementally)

Repoint each app's `DATABASE_URL` host/port from `timescale_db:5432` to
`pgbouncer:6432` — **same** dbname, user, and password — and redeploy it.

**Migrate the superuser app pools first** (`webhook_receiver`, `ingest_worker`,
`dashboard_api` backend, `worker`/`cron`/`gateway`) — they are the heaviest
consumers. Do them one at a time and watch `SHOW POOLS;` (below).

## ⚠️ Transaction-pooling caveats — read before cutting over

`pool_mode = transaction` returns the backend to the pool at every COMMIT/ROLLBACK,
so **session-scoped features don't survive across transactions**:

- **Server-side prepared statements** — the app must not rely on them, or set the
  driver to not cache them (e.g. asyncpg `statement_cache_size=0`; libpq simple
  query / psycopg2 default is fine). PgBouncer ≥1.21 supports prepared statements in
  transaction mode if you set `max_prepared_statements > 0` — enable that if an app
  needs them.
- **`SET`/`RESET` that must persist between transactions**, session `LISTEN/NOTIFY`,
  advisory locks held across transactions, `WITH HOLD` cursors, session temp tables.
- **Per-connection `options` startup GUCs are ignored** (see `ignore_startup_parameters`).
  Apps that set GUCs via the `options=` DSN param must instead pin them at the **role**
  level: `ALTER ROLE <app> SET statement_timeout = '...';` etc.

### The analytics MCP specifically

The MCP sends `options=-c default_transaction_read_only=on -c statement_timeout=30000`
on its DSN and calls `set_session(readonly=True)`. Behind transaction pooling:

- The `options` GUCs are dropped — **but** `analytics_ro` already has
  `default_transaction_read_only=on` and `statement_timeout=30s` pinned at the role
  level (`scripts/analytics_ro_role.sql`), so read-only enforcement is preserved.
- `set_session(readonly=True)` issues a `SET` that can leak across pooled clients.
  Before pointing the MCP at PgBouncer, either drop that call (role default covers it)
  or run the **MCP only in `session` pooling** (add a second `[databases]` alias with
  `pool_mode=session`). Given the MCP is a *minor* consumer, the simplest path is to
  **leave the MCP connecting directly** and pool only the heavy superuser apps.

## Operating

```bash
# admin console
docker exec -it pgbouncer psql -h 127.0.0.1 -p 6432 -U pgbouncer_auth pgbouncer
#   SHOW POOLS;     -- cl_active / sv_active / waiting per pool
#   SHOW CLIENTS;   -- connected clients
#   SHOW STATS;     -- throughput

# sanity: confirm the DB now sees a small, stable backend count
docker exec -i "$DB" psql -U postgres -d tracksolid_db -c \
  "SELECT usename, count(*) FROM pg_stat_activity GROUP BY 1 ORDER BY 2 DESC;"
```

**Sizing rule:** total backends PgBouncer opens = `Σ(default_pool_size per database) +
reserve_pool_size`. Keep that **well under** `max_connections` (100), leaving headroom
for superuser/admin/background-worker connections that bypass PgBouncer. The shipped
config (20 + 5 reserve, one database) tops out at ~25 backends.