docs: comprehensive README — column reference, query runbook, DQ/SLA notes, status

Add tickets.inc column reference (typed generated columns + geom/geog), a querying
runbook (map fn, inc_open_sla, closures/day, nearest-vehicle KNN), data-quality &
SLA caveats (source sla_status only valid when closed, ~30% null created_at_service,
mttr semantics, content lag, history gap), and a status/roadmap section.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
david kiania 2026-06-16 00:10:27 +03:00
parent 8e119e2328
commit e17553ccbf

View file

@ -49,6 +49,31 @@ Source coordinates are empty in the feed, so geocoding is required:
>25 km from the cluster centroid** (wrong-city guard). Results cache in
`tickets.geo_locations`.
### Columns on `tickets.inc`
| Column | Type | Notes |
|---|---|---|
| `ticket_id` | text (PK) | e.g. `WOT0715527` |
| `raw` | jsonb | full source record — the source of truth |
| `normalized_status` · `raw_status` | text | use `normalized_status` for filtering (canonical) |
| `bucket` | text | lifecycle: `closed` / `pending` |
| `is_actionable` | boolean | the open/closed flag (open = `true`) |
| `cluster` · `region` · `location_name` | text | `region` lowercased; `cluster` feeds the gazetteer |
| `assigned_team` · `owner` | text | closure attribution dimensions |
| `sla_status` | text | source `Compliant`/`Breached` — **only meaningful once closed** |
| `mttr` | numeric | **minutes** (source is decimal hours); null until closed |
| `created_at_service` · `scheduled_at` · `closed_at` · `first_seen_at` · `last_seen_at` · `source_created_at` · `source_updated_at` | timestamptz | EAT→UTC via `tickets.eat_ts()`. **lifecycle** = `created_at_service`→`closed_at`; **export bookkeeping** = `first_seen_at`/`last_seen_at`/`source_*` |
| `latitude` · `longitude` | double precision | `COALESCE(feed, geocoded)` — populated from `geom` |
| `geom` | geometry(Point,4326) | display / the map |
| `geog` | geography(Point,4326) | **routing** — metres-accurate distance (GiST indexed) |
| `geo_source` | text | precision: `feed` / `location` / `cluster` / `none` |
| `ingested_at` | timestamptz | when we last upserted this row |
Dropped from the unpacked columns (still in `raw`): `service_type`, `is_alarm`,
`is_auto_created`, `is_auto_closed` (all single-cardinality), plus the ingest-time
drops below. **`reporting.fn_tickets_for_map`** reads from `raw` and serves the map;
**`tickets.inc_open_sla`** is the open-ticket SLA view for dashboards/dispatch.
## Setup
```bash
@ -112,3 +137,54 @@ and runs the ingest; schedule it with a crontab line
`fn_tickets_for_map`).
- The curated/geocoded coordinates are written `verified = false` — review
`tickets.geo_clusters` / `tickets.geo_locations` and flip `verified` once checked.
## Querying
```sql
-- map payload (GeoJSON + summary, incl. summary.freshness) — what dashboard_api serves
SELECT reporting.fn_tickets_for_map(); -- open-only by default
SELECT reporting.fn_tickets_for_map(p_open_only := false); -- all geocoded tickets
-- open tickets by SLA (derived) + by cluster — via the view
SELECT sla_state, count(*) FROM tickets.inc_open_sla GROUP BY 1;
SELECT cluster, count(*), round(avg(hours_open),1) AS avg_hrs
FROM tickets.inc_open_sla GROUP BY 1 ORDER BY 2 DESC;
-- closures / creations per day (EAT)
SELECT (closed_at AT TIME ZONE 'Africa/Nairobi')::date AS d, count(*)
FROM tickets.inc WHERE closed_at IS NOT NULL GROUP BY 1 ORDER BY 1 DESC;
-- nearest open tickets to a vehicle (lng, lat) — metres, index-accelerated KNN
SELECT ticket_id, cluster, hours_open,
round(ST_Distance(geog, ST_SetSRID(ST_MakePoint(:lng,:lat),4326)::geography))::int AS metres
FROM tickets.inc_open_sla
ORDER BY geog <-> ST_SetSRID(ST_MakePoint(:lng,:lat),4326)::geography
LIMIT 10;
```
## Data-quality & SLA notes
Findings to keep in mind (see the PRD for detail):
- **Source `sla_status` is only meaningful for *closed* tickets.** It reads
`Compliant` for essentially all *open* tickets, so for open work use the **derived**
state in `tickets.inc_open_sla` (`now() created_at_service` vs the contract's 48h).
- **`created_at_service` is missing on ~30% of rows** (incl. most open ones); the SLA
view falls back to `first_seen_at` and flags it via `sla_clock_source`.
- **`mttr` is not wall-clock** `closed_at created_at_service` and the source's
`Breached`/`Compliant` does **not** match a plain 48h threshold — pin the contract's
exact SLA definition before trusting cross-field SLA math.
- **Content lag:** the feed's *file* timestamps are current, but the ticket *content*
trails ~2 days (the underlying `…wm_task.xlsx` source), so creation/closure dates
run a couple of days behind wall-clock.
- **History:** `tickets.inc` is current-state (upsert). Closure/creation/MTTR
*event* series work directly; **open-backlog-over-time** needs a history capture
(append-only `closure_events` or daily snapshots) — not yet built.
## Status / roadmap
Live: INC ingestion deployed on Coolify (hourly `15 7-19 * * *` EAT), schema +
generated columns + geocoding + the `inc_open_sla` view in `tracksolid_db`.
Next (Phase 2): time-series analytics (closure rate, MTTR/SLA trends), then FleetNow
vehicle **dispatch** off `geog`, and **team closure attribution**. **CRQ** is a
separate future project that will reuse this machinery against `automations/crq/`.