chore(db): purge unused ops + dwh_gold schemas
Drop the dormant ops (workshop / tickets / dispatch / SLA / odometer) and dwh_gold (nightly ETL aggregates) schemas plus their dependents — features never implemented, no live writer or scheduled refresh. - Prod DB (already applied): DROP SCHEMA ops/dwh_gold CASCADE, plus tracksolid.dispatch_log, v_sla_inflight, v_utilisation_daily. - migrations/12_drop_ops.sql + 13_drop_dwh_gold.sql (forward, all IF EXISTS) registered in run_migrations.py for rebuild durability. - grafana: removed 8 now-broken panels (In-flight SLA, Idle Cost, Utilisation Heatmap, Row 7 Field-Service SLAs) from daily_operations; panel count 21 -> 13. - docs: scrubbed CLAUDE.md, PLATFORM_OVERVIEW.html (-19KB), DATA_FLOW.md; pre-drop seed snapshot in docs/reports/260605_ops_purge_backup.md. The separate tracksolid_dwh server (31.97.44.246:5888) is unrelated and untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
e060933c55
commit
8c5a43f3b8
8 changed files with 1152 additions and 322 deletions
27
CLAUDE.md
27
CLAUDE.md
|
|
@ -56,7 +56,7 @@ See `docs/CONNECTIONS.md` for the full shape. Summary:
|
||||||
|
|
||||||
- **SSH:** `ssh -i ~/.ssh/id_ed25519 kianiadee@stage.rahamafresh.com`
|
- **SSH:** `ssh -i ~/.ssh/id_ed25519 kianiadee@stage.rahamafresh.com`
|
||||||
- **DB name:** `tracksolid_db` · **DB user:** `postgres` (internal) · `tracksolid_owner` (app) · `grafana_ro` (read-only)
|
- **DB name:** `tracksolid_db` · **DB user:** `postgres` (internal) · `tracksolid_owner` (app) · `grafana_ro` (read-only)
|
||||||
- **DB schemas:** `tracksolid` (live, single source of truth) · `dwh_gold` (aggregates) · `ops` (workshop / tickets / odometer) · `infrastructure`. The legacy `tracksolid_2` schema no longer exists — migrations 02–06 applied 2026-04-18.
|
- **DB schemas:** `tracksolid` (live, single source of truth) · `reporting` (map-dashboard read layer) · `infrastructure`. The legacy `tracksolid_2` schema no longer exists (migrations 02–06, 2026-04-18); the `ops` and `dwh_gold` schemas were purged 2026-06-05 (migrations 12/13) as unused.
|
||||||
- **DB access:** `DATABASE_URL` points to `timescale_db:5432` (internal Docker network — not reachable locally). Use `docker exec` pattern above. See `docs/CONNECTIONS.md` for full reference.
|
- **DB access:** `DATABASE_URL` points to `timescale_db:5432` (internal Docker network — not reachable locally). Use `docker exec` pattern above. See `docs/CONNECTIONS.md` for full reference.
|
||||||
- **DWH target DB:** `tracksolid_dwh` at `31.97.44.246:5888` (separate PostGIS instance, public IP). Users: `dwh_owner` (bronze writes + `dwh_control`), `grafana_ro` (reads bronze/silver/gold/`dwh_control`). Always connect with `sslmode=require`. Fed by the n8n `dwh_extract` + `dwh_load_bronze` workflows — see `docs/DWH_PIPELINE.md`.
|
- **DWH target DB:** `tracksolid_dwh` at `31.97.44.246:5888` (separate PostGIS instance, public IP). Users: `dwh_owner` (bronze writes + `dwh_control`), `grafana_ro` (reads bronze/silver/gold/`dwh_control`). Always connect with `sslmode=require`. Fed by the n8n `dwh_extract` + `dwh_load_bronze` workflows — see `docs/DWH_PIPELINE.md`.
|
||||||
- **Container naming:** Coolify appends a random suffix. Always resolve with:
|
- **Container naming:** Coolify appends a random suffix. Always resolve with:
|
||||||
|
|
@ -93,10 +93,11 @@ dwh/ # DWH migrations for tracksolid_dwh@31.97.44.246:588
|
||||||
# 261002_bronze_constraints_audit.sql — ON CONFLICT key assertion
|
# 261002_bronze_constraints_audit.sql — ON CONFLICT key assertion
|
||||||
# 261003_dwh_roles.sql — role contract assertion
|
# 261003_dwh_roles.sql — role contract assertion
|
||||||
# 261004_dwh_observability_views.sql — freshness/failure views
|
# 261004_dwh_observability_views.sql — freshness/failure views
|
||||||
migrations/ # Numbered SQL migrations 02–10, applied in order by run_migrations.py
|
migrations/ # Numbered SQL migrations 02–13, applied in order by run_migrations.py
|
||||||
# 02 full schema · 03 webhook · 04 distance fix · 05 enhancements
|
# 02 full schema · 03 webhook · 04 distance fix · 05 enhancements
|
||||||
# 06 ops/analytics · 07 views · 08 config · 09 trips enrichment
|
# 06 ops/analytics · 07 views · 08 config · 09 trips enrichment
|
||||||
# 10_driver_clock_views.sql · 10_pgbouncer_auth.sql
|
# 10_driver_clock_views.sql · 10_pgbouncer_auth.sql · 11 reporting
|
||||||
|
# 12 drop ops schema · 13 drop dwh_gold schema (both 2026-06-05)
|
||||||
Dockerfile # Custom image for ingest/webhook containers
|
Dockerfile # Custom image for ingest/webhook containers
|
||||||
pyproject.toml # Python project + uv dependency spec
|
pyproject.toml # Python project + uv dependency spec
|
||||||
backup/ # pg_dump sidecar scripts and config
|
backup/ # pg_dump sidecar scripts and config
|
||||||
|
|
@ -126,12 +127,13 @@ tracksolid.alarms -- Alarm events (alarm_type, alarm_name, alarm_time
|
||||||
tracksolid.obd_readings -- OBD diagnostics (push only, awaiting webhook registration)
|
tracksolid.obd_readings -- OBD diagnostics (push only, awaiting webhook registration)
|
||||||
tracksolid.device_events -- Power on/off tamper events (push only)
|
tracksolid.device_events -- Power on/off tamper events (push only)
|
||||||
tracksolid.ingestion_log -- API call audit trail — 875 runs / 24h, 0 failures at last check (2026-04-19)
|
tracksolid.ingestion_log -- API call audit trail — 875 runs / 24h, 0 failures at last check (2026-04-19)
|
||||||
tracksolid.dispatch_log -- Dispatch decisions for SLA tracking (migration 06; empty until ops integration)
|
tracksolid.schema_migrations -- Applied migrations 02–13
|
||||||
tracksolid.schema_migrations -- Applied files: 02,03,04,05,06 (last 06 on 2026-04-18)
|
-- PURGED 2026-06-05 (migrations 12 + 13): the dormant `ops` schema (tickets, service_log,
|
||||||
dwh_gold.fact_daily_fleet_metrics -- Nightly ETL aggregates per vehicle per day (run refresh_daily_metrics)
|
-- odometer_readings, cost_rates, kpi_targets, vw_service_forecast), tracksolid.dispatch_log,
|
||||||
ops.service_log -- Workshop service history (migration 06)
|
-- and the `dwh_gold` schema (dim_vehicles, fact_daily_fleet_metrics, refresh_daily_metrics).
|
||||||
ops.odometer_readings -- Physical odometer captures (migration 06)
|
-- Those workshop/dispatch/SLA/utilisation features were never implemented. Do NOT reintroduce
|
||||||
ops.tickets -- Ticket skeleton for ops integration (migration 06; empty)
|
-- references to ops.* or dwh_gold.* — they no longer exist. (The separate tracksolid_dwh DB
|
||||||
|
-- at 31.97.44.246:5888 is unrelated and untouched.)
|
||||||
```
|
```
|
||||||
|
|
||||||
Full DDL: `02_tracksolid_full_schema_rev.sql` + migrations `03`–`06`.
|
Full DDL: `02_tracksolid_full_schema_rev.sql` + migrations `03`–`06`.
|
||||||
|
|
@ -146,8 +148,8 @@ tracksolid.v_currently_idle -- §2.2 idle lens
|
||||||
tracksolid.v_driver_aggregates_daily -- §3.1 + §3.2 aggression index source
|
tracksolid.v_driver_aggregates_daily -- §3.1 + §3.2 aggression index source
|
||||||
tracksolid.v_fleet_km_daily -- §7 Panel 5 distance trend
|
tracksolid.v_fleet_km_daily -- §7 Panel 5 distance trend
|
||||||
tracksolid.v_alarms_daily -- §7 Panel 7 alarm frequency
|
tracksolid.v_alarms_daily -- §7 Panel 7 alarm frequency
|
||||||
tracksolid.v_utilisation_daily -- §7 Panel 8 utilisation heatmap (gated on dwh_gold ETL)
|
-- v_utilisation_daily (dwh_gold) and v_sla_inflight (ops) were DROPPED 2026-06-05 with
|
||||||
tracksolid.v_sla_inflight -- §4.5 SLA panels (gated on ops.tickets)
|
-- their schemas (migrations 12/13); their Grafana panels were removed from the dashboard.
|
||||||
```
|
```
|
||||||
|
|
||||||
All views carry a `COMMENT ON VIEW` referencing their spec — `\d+ tracksolid.v_*` shows the provenance.
|
All views carry a `COMMENT ON VIEW` referencing their spec — `\d+ tracksolid.v_*` shows the provenance.
|
||||||
|
|
@ -222,7 +224,7 @@ dwh_control.v_watermark_lag -- Grafana: extract vs. load lag per table
|
||||||
7. **Secrets from env only.** Connection strings, API keys, and passwords live in `.env`. Reference variable names from `docs/CONNECTIONS.md`, never values.
|
7. **Secrets from env only.** Connection strings, API keys, and passwords live in `.env`. Reference variable names from `docs/CONNECTIONS.md`, never values.
|
||||||
8. **Two developers, one incoming.** Write code and docs that a second developer (mixed technical/operations background) can follow without prior context.
|
8. **Two developers, one incoming.** Write code and docs that a second developer (mixed technical/operations background) can follow without prior context.
|
||||||
9. **Forgejo API auth:** credentials stored in macOS keychain. Retrieve with `git credential fill` (host=repo.rahamafresh.com). Use basic auth against `https://repo.rahamafresh.com/api/v1` directly — no `tea` or `gh` needed.
|
9. **Forgejo API auth:** credentials stored in macOS keychain. Retrieve with `git credential fill` (host=repo.rahamafresh.com). Use basic auth against `https://repo.rahamafresh.com/api/v1` directly — no `tea` or `gh` needed.
|
||||||
10. **Single live schema.** All live data lives in `tracksolid`. Aggregates live in `dwh_gold`; workshop/ticket integrations live in `ops`. Do not reintroduce references to the retired `tracksolid_2` schema.
|
10. **Single live schema.** All live data lives in `tracksolid`; the map-dashboard read layer lives in `reporting`. Do not reintroduce references to the retired `tracksolid_2`, `ops`, or `dwh_gold` schemas (the latter two purged 2026-06-05, migrations 12/13).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -256,6 +258,5 @@ Latest full snapshot: `docs/reports/260412_baseline_report.md`
|
||||||
| MEDIUM | Investigate 44 silent devices (only 19 of 63 reporting) — SIM installed? Activated? |
|
| MEDIUM | Investigate 44 silent devices (only 19 of 63 reporting) — SIM installed? Activated? |
|
||||||
| MEDIUM | Co-develop client KPI framework (see `docs/KPI_FRAMEWORK.md`) |
|
| MEDIUM | Co-develop client KPI framework (see `docs/KPI_FRAMEWORK.md`) |
|
||||||
| LOW | Populate geofences — depot boundaries, city zones |
|
| LOW | Populate geofences — depot boundaries, city zones |
|
||||||
| LOW | Schedule nightly ETL: `SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1)` (cron or n8n) |
|
|
||||||
| HIGH | Deploy DWH bronze pipeline: apply `dwh/26100{1,2,3,4}.sql` to `tracksolid_dwh`, import + wire the two n8n workflows, verify first run via `dwh_control.v_table_freshness`. Runbook: `docs/DWH_PIPELINE.md` |
|
| HIGH | Deploy DWH bronze pipeline: apply `dwh/26100{1,2,3,4}.sql` to `tracksolid_dwh`, import + wire the two n8n workflows, verify first run via `dwh_control.v_table_freshness`. Runbook: `docs/DWH_PIPELINE.md` |
|
||||||
| MEDIUM | Rotate `dwh_owner` / `grafana_ro` passwords on `tracksolid_dwh` — plaintext in `dwh/260423_dwh_ddl_v1.sql` is a pre-existing flaw to clean up separately |
|
| MEDIUM | Rotate `dwh_owner` / `grafana_ro` passwords on `tracksolid_dwh` — plaintext in `dwh/260423_dwh_ddl_v1.sql` is a pre-existing flaw to clean up separately |
|
||||||
|
|
|
||||||
240
docs/DATA_FLOW.md
Normal file
240
docs/DATA_FLOW.md
Normal file
|
|
@ -0,0 +1,240 @@
|
||||||
|
# Data Flow — Ingestion → Aggregation → Views → Functions → Consumers
|
||||||
|
|
||||||
|
**Scope:** the *live* fleet pipeline only (`tracksolid` + `reporting`). The `ops` and
|
||||||
|
`dwh_gold` schemas were **purged on 2026-06-05** (migrations 12/13) — those workshop /
|
||||||
|
dispatch / SLA / utilisation features were never implemented. They are shown below only as
|
||||||
|
a struck-out "removed" footnote for historical context. (The *separate* `tracksolid_dwh`
|
||||||
|
server at 31.97.44.246:5888 is unrelated and was not touched.)
|
||||||
|
|
||||||
|
**Verified against prod 2026-06-05** (TimescaleDB hypertable + continuous-aggregate
|
||||||
|
catalog, `pg_depend` view graph, ingestion `INSERT` targets, `dashboard_api` queries,
|
||||||
|
Grafana panel SQL). Key facts that surprised the docs:
|
||||||
|
|
||||||
|
- Only `position_history` (and the empty/planned `heartbeats`, `fuel_readings`,
|
||||||
|
`temperature_readings`) are **hypertables**. `trips` and `alarms` are **plain tables**.
|
||||||
|
- `tracksolid.v_mileage_daily_cagg` is a **real TimescaleDB continuous aggregate**, not a
|
||||||
|
plain view — and it currently has **no downstream consumer**.
|
||||||
|
- `reporting.v_trips` is a **matview**, refreshed every 5 min by the in-process
|
||||||
|
`dashboard_api` background loop (FIX-D02), pg advisory-lock `920145`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Mermaid
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
API["Tracksolid / Jimi API<br/>poll + push webhooks · OAuth2"]
|
||||||
|
|
||||||
|
subgraph ING["Ingestion — ts_shared_rev.py: get_conn() · api_post() · clean*()"]
|
||||||
|
IM["ingest_movement_rev.py"]
|
||||||
|
IE["ingest_events_rev.py"]
|
||||||
|
WR["webhook_receiver_rev.py"]
|
||||||
|
end
|
||||||
|
API --> IM & IE & WR
|
||||||
|
|
||||||
|
subgraph L1["L1 · Base tables + hypertables — schema: tracksolid (single source of truth)"]
|
||||||
|
LP["live_positions<br/>current fix / IMEI"]
|
||||||
|
PH[("position_history<br/>HYPERTABLE · high-res GPS trail")]
|
||||||
|
TR["trips<br/>plain table — NOT a hypertable"]
|
||||||
|
DV["devices<br/>vehicle / driver registry"]
|
||||||
|
AL["alarms<br/>plain table"]
|
||||||
|
ILOG["ingestion_log · api_token_cache"]
|
||||||
|
EMPTY["heartbeats* · fuel_readings* · temperature_readings* (HYPERTABLES, empty)<br/>device_events · fault_codes · obd_readings · parking_events · lbs_readings · geofences (empty)"]
|
||||||
|
end
|
||||||
|
IM --> LP & PH & TR & DV
|
||||||
|
IM --> ILOG
|
||||||
|
IE --> AL
|
||||||
|
WR --> EMPTY
|
||||||
|
|
||||||
|
subgraph L2["L2 · Aggregation"]
|
||||||
|
CAGG[("v_mileage_daily_cagg<br/>CONTINUOUS AGGREGATE")]
|
||||||
|
VT[("reporting.v_trips<br/>MATVIEW · ix_v_trips_trip_id")]
|
||||||
|
RLOG["reporting.refresh_log"]
|
||||||
|
end
|
||||||
|
PH -->|Timescale cont-agg policy| CAGG
|
||||||
|
TR --> VT
|
||||||
|
DV --> VT
|
||||||
|
VT -->|"REFRESH CONCURRENTLY every 300s<br/>(dashboard_api loop, adv-lock 920145)"| RLOG
|
||||||
|
|
||||||
|
subgraph L3L["L3 · Reporting views — fed by v_trips matview"]
|
||||||
|
FILT["v_filter_vehicles · v_filter_drivers<br/>v_filter_cost_centres · v_filter_cities"]
|
||||||
|
SUMM["v_daily/weekly/monthly_summary<br/>v_*_cost_centre · v_trips_today"]
|
||||||
|
VLP["v_live_positions"]
|
||||||
|
end
|
||||||
|
VT --> FILT & SUMM
|
||||||
|
LP --> VLP
|
||||||
|
|
||||||
|
subgraph L3R["L3 · Grafana views — tracksolid.* (read base tables directly)"]
|
||||||
|
GV["v_fleet_today · v_fleet_status · v_active_dispatch_map<br/>v_currently_idle · v_alarms_daily · v_fleet_km_daily<br/>v_ingestion_health · v_vehicles_not_moved_today<br/>v_driver_aggregates_daily · v_fleet_trace · v_driver_clock_*"]
|
||||||
|
end
|
||||||
|
LP --> GV
|
||||||
|
TR --> GV
|
||||||
|
DV --> GV
|
||||||
|
AL --> GV
|
||||||
|
PH --> GV
|
||||||
|
|
||||||
|
subgraph L4["L4 · Functions — reporting.* (the only API entrypoints)"]
|
||||||
|
FLP["fn_live_positions(cost_centre, acc_status)"]
|
||||||
|
FVT["fn_vehicle_track(vehicle_number, hours)"]
|
||||||
|
FTM["fn_trips_for_map(veh[], driver, cc, city, start, end)"]
|
||||||
|
NP["normalize_plate(p) · helper"]
|
||||||
|
end
|
||||||
|
LP --> FLP
|
||||||
|
DV --> FLP
|
||||||
|
PH --> FVT
|
||||||
|
VT --> FTM
|
||||||
|
NP -.-> FTM
|
||||||
|
|
||||||
|
subgraph L5["L5 · Consumers"]
|
||||||
|
DAPI["dashboard_api_rev.py<br/>FastAPI :8890 · 2 workers"]
|
||||||
|
SPA["SPAs: liveposition.* · fleetintelligence.*<br/>(rustfs / S3 single-file maps)"]
|
||||||
|
GRAF["Grafana"]
|
||||||
|
end
|
||||||
|
FLP --> DAPI
|
||||||
|
FVT --> DAPI
|
||||||
|
FTM --> DAPI
|
||||||
|
FILT --> DAPI
|
||||||
|
DAPI -->|"HTTPS · fleetapi.rahamafresh.com"| SPA
|
||||||
|
GV --> GRAF
|
||||||
|
|
||||||
|
CAGG -.->|no consumer yet| NONE(["⚠ unconsumed — no panel / API reads it"])
|
||||||
|
|
||||||
|
subgraph PARK["REMOVED 2026-06-05 — purged via migrations 12 / 13 (never implemented)"]
|
||||||
|
GONE["ops.* (tickets · dispatch_log · service_log · odometer_readings · cost_rates · kpi_targets · vw_service_forecast)<br/>dwh_gold.* (dim_vehicles · fact_daily_fleet_metrics · refresh_daily_metrics)<br/>tracksolid.v_sla_inflight · tracksolid.v_utilisation_daily + their Grafana panels"]
|
||||||
|
end
|
||||||
|
|
||||||
|
classDef hyper fill:#e1f0ff,stroke:#3b82f6,color:#0b3d91;
|
||||||
|
classDef mat fill:#fff3cd,stroke:#d4a017,color:#664d03;
|
||||||
|
classDef cagg fill:#e7f7e7,stroke:#2e9e2e,color:#1e5e1e;
|
||||||
|
classDef parked fill:#f0f0f0,stroke:#999,stroke-dasharray:5 5,color:#555;
|
||||||
|
classDef warn fill:#fdecea,stroke:#d93025,color:#a52714;
|
||||||
|
class PH hyper;
|
||||||
|
class VT mat;
|
||||||
|
class CAGG cagg;
|
||||||
|
class PARK,GONE parked;
|
||||||
|
class NONE warn;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ASCII
|
||||||
|
|
||||||
|
```
|
||||||
|
╔══════════════════════════════════════════════╗
|
||||||
|
║ TRACKSOLID / JIMI API ║
|
||||||
|
║ (poll + push webhooks, OAuth2) ║
|
||||||
|
╚════════════════════╤═════════════════════════╝
|
||||||
|
│
|
||||||
|
┌──────────────────────────┬───────────┴───────────┬────────────────────────────┐
|
||||||
|
▼ ▼ ▼ ▼
|
||||||
|
┌─────────────────┐ ┌────────────────────┐ ┌──────────────────────┐
|
||||||
|
│ ingest_movement │ │ ingest_events │ │ webhook_receiver │ (all via ts_shared_rev.py:
|
||||||
|
│ _rev.py │ │ _rev.py │ │ _rev.py │ get_conn() pool, api_post(),
|
||||||
|
└────────┬────────┘ └─────────┬──────────┘ └──────────┬───────────┘ clean*() , token cache)
|
||||||
|
│ │ │
|
||||||
|
│ writes │ writes │ writes (push)
|
||||||
|
▼ ▼ ▼
|
||||||
|
╔════════════════════════════════════════════════════════════════════════════════════════════╗
|
||||||
|
║ L1 · BASE TABLES + HYPERTABLES schema: tracksolid (single source of truth) ║
|
||||||
|
║ ║
|
||||||
|
║ live_positions ── current fix / IMEI (plain) api_token_cache (auth) ║
|
||||||
|
║ position_history ◀═══ HYPERTABLE (high-res GPS trail, partitioned by gps_time) ║
|
||||||
|
║ trips ── trip summaries (plain table — NOT a hypertable) ║
|
||||||
|
║ devices ── vehicle/driver registry (plain) ║
|
||||||
|
║ alarms ── alarm events (plain table) ║
|
||||||
|
║ ingestion_log ── API audit trail (plain) ║
|
||||||
|
║ heartbeats*, fuel_readings*, temperature_readings* ◀═ HYPERTABLES (empty / planned push) ║
|
||||||
|
║ device_events, fault_codes, obd_readings, parking_events, lbs_readings, geofences (empty) ║
|
||||||
|
╚══════╤══════════════════════════╤══════════════════════════════════╤════════════════════════╝
|
||||||
|
│ │ │
|
||||||
|
│ TimescaleDB │ matview refresh │ direct reads
|
||||||
|
│ cont-agg policy │ (in-app loop) │ (Grafana SQL)
|
||||||
|
▼ ▼ ▼
|
||||||
|
╔══════════════════╗ ╔════════════════════════════════╗ ╔══════════════════════════════════╗
|
||||||
|
║ L2 · AGGREGATION ║ ║ L2 · AGGREGATION (matview) ║ ║ L3 · GRAFANA VIEWS (tracksolid.*) ║
|
||||||
|
║ ║ ║ ║ ║ read base tables directly ║
|
||||||
|
║ v_mileage_daily ║ ║ reporting.v_trips [MATVIEW] ║ ║ ║
|
||||||
|
║ _cagg ║ ║ ◀── trips + devices ║ ║ v_fleet_today v_fleet_status ║
|
||||||
|
║ CONTINUOUS AGG ║ ║ unique ix_v_trips_trip_id ║ ║ v_active_dispatch_map ║
|
||||||
|
║ ◀═ position_ ║ ║ ║ ║ v_currently_idle v_alarms_daily ║
|
||||||
|
║ history ║ ║ REFRESH … CONCURRENTLY every ║ ║ v_fleet_km_daily v_ingestion_health║
|
||||||
|
║ (auto refresh ║ ║ VTRIPS_REFRESH_INTERVAL_S=300 ║ ║ v_vehicles_not_moved_today ║
|
||||||
|
║ policy) ║ ║ by dashboard_api bg asyncio ║ ║ v_driver_aggregates_daily ║
|
||||||
|
╚════════╤═════════╝ ║ loop, pg advisory-lock 920145 ║ ║ v_fleet_trace ║
|
||||||
|
│ ║ │ ║ ║ v_driver_clock_daily/_today/_attnd ║
|
||||||
|
│ ║ └──▶ reporting.refresh_log ║ ╚══════════════╤═════════════════════╝
|
||||||
|
│ ╚═══════════════╤════════════════╝ │
|
||||||
|
│ │ │
|
||||||
|
│ ▼ │
|
||||||
|
│ ╔════════════════════════════════════════════╗ │
|
||||||
|
│ ║ L3 · REPORTING VIEWS (← v_trips matview) ║ │
|
||||||
|
│ ║ v_filter_vehicles v_filter_drivers ║ │
|
||||||
|
│ ║ v_filter_cost_centres v_filter_cities ║ │
|
||||||
|
│ ║ v_daily/weekly/monthly_summary ║ │
|
||||||
|
│ ║ v_*_cost_centre v_trips_today ║ │
|
||||||
|
│ ║ v_live_positions (view, ← live_positions) ║ │
|
||||||
|
│ ╚════════════════════════╤═══════════════════╝ │
|
||||||
|
│ │ │
|
||||||
|
│ ╔══════════════════════════════════╧═══════════════════╗ │
|
||||||
|
│ ║ L4 · FUNCTIONS (schema reporting — API entrypoints) ║ │
|
||||||
|
│ ║ ║ │
|
||||||
|
│ ║ fn_live_positions(cost_centre, acc_status) ║ │
|
||||||
|
│ ║ ◀── live_positions + devices ║ │
|
||||||
|
│ ║ fn_vehicle_track(vehicle_number, hours) ║ │
|
||||||
|
│ ║ ◀── position_history (hypertable trail) ────────╫─────┘ (also reads L1)
|
||||||
|
│ ║ fn_trips_for_map(veh[],driver,cc,city,start,end) ║
|
||||||
|
│ ║ ◀── reporting.v_trips (matview) ║
|
||||||
|
│ ║ normalize_plate(p) ── helper used by the above ║
|
||||||
|
│ ╚════════════════════════════╤══════════════════════════╝
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
(cagg currently ╔═══════════════════════════════════════════════╗
|
||||||
|
unconsumed — ║ L5 · CONSUMERS ║
|
||||||
|
no panel/API yet) ║ ║
|
||||||
|
║ dashboard_api_rev.py (FastAPI :8890, ×2 wkrs) ║
|
||||||
|
║ POST /webhook/fleet-dashboard → fn_trips_for_map + v_filter_*
|
||||||
|
║ GET /webhook/live-positions → fn_live_positions
|
||||||
|
║ GET /webhook/live-positions/track → fn_vehicle_track
|
||||||
|
║ │ ║
|
||||||
|
║ ▼ HTTPS (fleetapi.rahamafresh.com) ║
|
||||||
|
║ SPAs: liveposition.* · fleetintelligence.* ║
|
||||||
|
║ (rustfs/S3 single-file maps) ║
|
||||||
|
║ ║
|
||||||
|
║ Grafana ──SQL──▶ tracksolid.v_* (L3 right) ║
|
||||||
|
╚═══════════════════════════════════════════════╝
|
||||||
|
|
||||||
|
|
||||||
|
┌─ REMOVED 2026-06-05 · purged via migrations 12 / 13 (never implemented) ───────────────┐
|
||||||
|
│ ops.* : tickets, dispatch_log, service_log, odometer_readings, │
|
||||||
|
│ cost_rates, kpi_targets, vw_service_forecast │
|
||||||
|
│ dwh_gold.* : dim_vehicles, fact_daily_fleet_metrics, refresh_daily_metrics() │
|
||||||
|
│ tracksolid : v_sla_inflight, v_utilisation_daily (+ their Grafana panels removed) │
|
||||||
|
│ → Schemas dropped from prod. The separate tracksolid_dwh server is unrelated/untouched.│
|
||||||
|
└────────────────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How to read it
|
||||||
|
|
||||||
|
- **L1** is where ingestion lands. Only `position_history` (and three empty/planned
|
||||||
|
hypertables) are TimescaleDB hypertables; `trips`/`alarms` are ordinary tables.
|
||||||
|
- **Two aggregation mechanisms** run in parallel: the TimescaleDB **continuous aggregate**
|
||||||
|
`v_mileage_daily_cagg` (refreshed by a Timescale policy off `position_history`), and the
|
||||||
|
**`reporting.v_trips` matview** (refreshed every 5 min by the in-process `dashboard_api`
|
||||||
|
loop — the FIX-D02 self-refresher that replaced the dead n8n job).
|
||||||
|
- **L4 functions are the only thing the API calls.** SPAs never touch tables directly:
|
||||||
|
SPA → `dashboard_api` → `fn_*` → (matview / tables).
|
||||||
|
- **Grafana** takes the right-hand path, reading the `tracksolid.v_*` analytics views
|
||||||
|
straight off L1.
|
||||||
|
|
||||||
|
## Caveats / housekeeping notes
|
||||||
|
|
||||||
|
1. **`v_mileage_daily_cagg` has no consumer** — nothing in the API or Grafana reads it. It
|
||||||
|
is a live continuous aggregate maintaining itself for nobody. Candidate for removal or
|
||||||
|
wiring into a panel.
|
||||||
|
2. **`ops` and `dwh_gold` were purged** (2026-06-05, migrations 12/13) along with
|
||||||
|
`v_utilisation_daily`, `v_sla_inflight`, and their Grafana panels — the features were
|
||||||
|
never implemented.
|
||||||
|
3. See the companion housekeeping audit (2026-06-05) for the full unused-object list; the
|
||||||
|
only clean ad-hoc drop is `public.trips_viz_v1`.
|
||||||
File diff suppressed because one or more lines are too long
47
docs/reports/260605_ops_purge_backup.md
Normal file
47
docs/reports/260605_ops_purge_backup.md
Normal file
|
|
@ -0,0 +1,47 @@
|
||||||
|
# ops schema purge — pre-drop backup (2026-06-05)
|
||||||
|
|
||||||
|
Snapshot taken immediately before dropping the `ops` schema + `tracksolid.dispatch_log`
|
||||||
|
+ the dependent view `tracksolid.v_sla_inflight`. The dispatch/SLA/workshop features were
|
||||||
|
never implemented and are being purged (may take a different direction later).
|
||||||
|
|
||||||
|
Only two `ops` tables held rows, and both are **migration-08 seed data**
|
||||||
|
(`migrations/08_analytics_config.sql`), i.e. regenerable — this file is belt-and-suspenders.
|
||||||
|
Every other ops table (`tickets`, `service_log`, `odometer_readings`) and the view
|
||||||
|
`vw_service_forecast` were empty. `tracksolid.dispatch_log` was empty.
|
||||||
|
|
||||||
|
## ops.cost_rates (3 rows)
|
||||||
|
|
||||||
|
| rate_key | scope_type | scope_value | metric | amount | currency | effective_from | notes |
|
||||||
|
|---|---|---|---|---|---|---|---|
|
||||||
|
| fuel.nairobi | city | nairobi | fuel_per_litre | 195.00 | KES | 2026-04-27 | Placeholder pump price — confirm with Finance. |
|
||||||
|
| fuel.mombasa | city | mombasa | fuel_per_litre | 195.00 | KES | 2026-04-27 | Placeholder pump price — confirm with Finance. |
|
||||||
|
| fuel.kampala | city | kampala | fuel_per_litre | 5200.00 | UGX | 2026-04-27 | Placeholder pump price — confirm with Finance. |
|
||||||
|
|
||||||
|
## ops.kpi_targets (12 rows)
|
||||||
|
|
||||||
|
| target_id | kpi_key | scope_type | scope_value | target | amber | red | direction | effective_from | notes |
|
||||||
|
|---|---|---|---|---|---|---|---|---|---|
|
||||||
|
| 1 | utilisation_pct | global | | 70.00 | 60.00 | 50.00 | higher_is_better | 2026-04-27 | Fleet utilisation: drive_hours / engine_on_hours. |
|
||||||
|
| 2 | idle_pct | global | | 15.00 | 20.00 | 25.00 | lower_is_better | 2026-04-27 | Idle as % of engine-on time. |
|
||||||
|
| 3 | idle_pct | cost_centre | osp patrol | 15.00 | 20.00 | 25.00 | lower_is_better | 2026-04-27 | OSP patrol idle target — same as global until calibrated. |
|
||||||
|
| 4 | fuel_kes_per_100km | global | | 12.00 | 14.00 | 16.00 | lower_is_better | 2026-04-27 | Fuel litres per 100km equivalent — uses fuel_100km on devices. |
|
||||||
|
| 5 | mttr_hours | global | | 4.00 | 6.00 | 8.00 | lower_is_better | 2026-04-27 | Mean Time To Resolve, field-service ticket. |
|
||||||
|
| 6 | alarms_per_100km | global | | 2.00 | 3.00 | 5.00 | lower_is_better | 2026-04-27 | Safety event density. |
|
||||||
|
| 7 | utilisation_pct | global | | 70.00 | 60.00 | 50.00 | higher_is_better | 2026-05-01 | Fleet utilisation: drive_hours / engine_on_hours. |
|
||||||
|
| 8 | idle_pct | global | | 15.00 | 20.00 | 25.00 | lower_is_better | 2026-05-01 | Idle as % of engine-on time. |
|
||||||
|
| 9 | idle_pct | cost_centre | osp patrol | 15.00 | 20.00 | 25.00 | lower_is_better | 2026-05-01 | OSP patrol idle target — same as global until calibrated. |
|
||||||
|
| 10 | fuel_kes_per_100km | global | | 12.00 | 14.00 | 16.00 | lower_is_better | 2026-05-01 | Fuel litres per 100km equivalent — uses fuel_100km on devices. |
|
||||||
|
| 11 | mttr_hours | global | | 4.00 | 6.00 | 8.00 | lower_is_better | 2026-05-01 | Mean Time To Resolve, field-service ticket. |
|
||||||
|
| 12 | alarms_per_100km | global | | 2.00 | 3.00 | 5.00 | lower_is_better | 2026-05-01 | Safety event density. |
|
||||||
|
|
||||||
|
## What was dropped
|
||||||
|
|
||||||
|
```sql
|
||||||
|
DROP VIEW IF EXISTS tracksolid.v_sla_inflight; -- depended on ops.tickets + dispatch_log
|
||||||
|
DROP SCHEMA IF EXISTS ops CASCADE; -- tickets, service_log, odometer_readings,
|
||||||
|
-- cost_rates, kpi_targets, vw_service_forecast
|
||||||
|
DROP TABLE IF EXISTS tracksolid.dispatch_log; -- empty; only fed v_sla_inflight
|
||||||
|
```
|
||||||
|
|
||||||
|
**Not dropped in this step:** `dwh_gold` schema and `tracksolid.v_utilisation_daily`
|
||||||
|
(separate decision, pending).
|
||||||
File diff suppressed because it is too large
Load diff
30
migrations/12_drop_ops.sql
Normal file
30
migrations/12_drop_ops.sql
Normal file
|
|
@ -0,0 +1,30 @@
|
||||||
|
-- 12_drop_ops.sql
|
||||||
|
-- Purge the dormant `ops` schema (workshop / tickets / dispatch / SLA / odometer)
|
||||||
|
-- and the dispatch/SLA artefacts that lived in `tracksolid`. These features were
|
||||||
|
-- never implemented and may take a different direction; the schema and views were
|
||||||
|
-- empty (or seed-only) and unused by the live map/dashboard pipeline.
|
||||||
|
--
|
||||||
|
-- Created by migrations 06 (ops schema + dispatch_log), 07 (v_sla_inflight) and
|
||||||
|
-- 08 (ops.cost_rates / ops.kpi_targets seed). Per the repo rule we do NOT rewrite
|
||||||
|
-- those applied migrations — this forward migration drops the objects instead. On
|
||||||
|
-- a fresh rebuild 06/07/08 create them, then this file removes them again.
|
||||||
|
--
|
||||||
|
-- Pre-drop snapshot of the only seeded tables: docs/reports/260605_ops_purge_backup.md
|
||||||
|
-- (the seed is also reproducible from 08_analytics_config.sql).
|
||||||
|
--
|
||||||
|
-- Every statement is IF EXISTS so the file is safe to re-apply.
|
||||||
|
--
|
||||||
|
-- NOTE: dwh_gold and tracksolid.v_utilisation_daily are intentionally NOT touched
|
||||||
|
-- here — that is a separate decision.
|
||||||
|
|
||||||
|
-- View first: it reads ops.tickets + tracksolid.dispatch_log, so it must go before
|
||||||
|
-- the objects it depends on (avoids an implicit CASCADE surprise).
|
||||||
|
DROP VIEW IF EXISTS tracksolid.v_sla_inflight;
|
||||||
|
|
||||||
|
-- The whole ops schema: tickets, service_log, odometer_readings, cost_rates,
|
||||||
|
-- kpi_targets, and the view vw_service_forecast. CASCADE clears intra-schema deps.
|
||||||
|
DROP SCHEMA IF EXISTS ops CASCADE;
|
||||||
|
|
||||||
|
-- Dispatch feature table — lived in the tracksolid schema, empty, only fed
|
||||||
|
-- v_sla_inflight (now dropped).
|
||||||
|
DROP TABLE IF EXISTS tracksolid.dispatch_log;
|
||||||
23
migrations/13_drop_dwh_gold.sql
Normal file
23
migrations/13_drop_dwh_gold.sql
Normal file
|
|
@ -0,0 +1,23 @@
|
||||||
|
-- 13_drop_dwh_gold.sql
|
||||||
|
-- Purge the dormant `dwh_gold` aggregate schema and its dependent view. The nightly
|
||||||
|
-- ETL (dwh_gold.refresh_daily_metrics) was never scheduled, both fact/dim tables were
|
||||||
|
-- empty, and nothing in the live map/dashboard pipeline reads them. These analytics
|
||||||
|
-- may take a different direction later.
|
||||||
|
--
|
||||||
|
-- Created by migrations 02 (schema), 05 (dwh_gold expansion + refresh_daily_metrics)
|
||||||
|
-- and 07 (tracksolid.v_utilisation_daily). Per the repo rule we do NOT rewrite those
|
||||||
|
-- applied migrations — this forward migration drops the objects instead. On a fresh
|
||||||
|
-- rebuild 02/05/07 create them, then this file removes them again.
|
||||||
|
--
|
||||||
|
-- Both tables were empty at drop time, so there is no data backup (cf. the ops purge,
|
||||||
|
-- which had seed rows — docs/reports/260605_ops_purge_backup.md). Companion to
|
||||||
|
-- 12_drop_ops.sql; together they retire the unused ops + dwh_gold analytics layers.
|
||||||
|
--
|
||||||
|
-- Every statement is IF EXISTS so the file is safe to re-apply.
|
||||||
|
|
||||||
|
-- View first: it reads dwh_gold.dim_vehicles + dwh_gold.fact_daily_fleet_metrics.
|
||||||
|
DROP VIEW IF EXISTS tracksolid.v_utilisation_daily;
|
||||||
|
|
||||||
|
-- The whole dwh_gold schema: dim_vehicles, fact_daily_fleet_metrics, the
|
||||||
|
-- dim_vehicles sequence/indexes, and the refresh_daily_metrics() function.
|
||||||
|
DROP SCHEMA IF EXISTS dwh_gold CASCADE;
|
||||||
|
|
@ -35,6 +35,8 @@ MIGRATIONS = [
|
||||||
"09_trips_enrichment.sql", # trips.route_geom + addresses + plate + v_trips_enriched
|
"09_trips_enrichment.sql", # trips.route_geom + addresses + plate + v_trips_enriched
|
||||||
"10_pgbouncer_auth.sql", # pgbouncer role + user_lookup() for SCRAM passthrough
|
"10_pgbouncer_auth.sql", # pgbouncer role + user_lookup() for SCRAM passthrough
|
||||||
"11_reporting_schema.sql", # reporting.* map-dashboard read layer (dashboard_api)
|
"11_reporting_schema.sql", # reporting.* map-dashboard read layer (dashboard_api)
|
||||||
|
"12_drop_ops.sql", # purge dormant ops schema + dispatch_log + v_sla_inflight
|
||||||
|
"13_drop_dwh_gold.sql", # purge dormant dwh_gold schema + v_utilisation_daily
|
||||||
]
|
]
|
||||||
|
|
||||||
# ── Tables that must exist before the service is allowed to start ─────────────
|
# ── Tables that must exist before the service is allowed to start ─────────────
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue