Compare commits
No commits in common. "main" and "quality-program-2026-04-12" have entirely different histories.
main
...
quality-pr
63 changed files with 337 additions and 14654 deletions
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -13,9 +13,6 @@ __pycache__/
|
|||
|
||||
# OS
|
||||
.DS_Store
|
||||
|
||||
# OSM extracts (large, reproducible from download — see docs/OSM_POI_EXPORT.md)
|
||||
*.osm.pbf
|
||||
Thumbs.db
|
||||
|
||||
# Backups
|
||||
|
|
|
|||
76
CLAUDE.md
76
CLAUDE.md
|
|
@ -19,7 +19,7 @@ docker exec $DB psql -U postgres -d tracksolid_db -c "SELECT COUNT(*) FROM track
|
|||
|
||||
**Run a migration file:**
|
||||
```bash
|
||||
docker exec -i $DB psql -U postgres -d tracksolid_db < migrations/07_your_migration.sql
|
||||
docker exec -i $DB psql -U postgres -d tracksolid_db < 07_your_migration.sql
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -56,7 +56,7 @@ See `docs/CONNECTIONS.md` for the full shape. Summary:
|
|||
|
||||
- **SSH:** `ssh -i ~/.ssh/id_ed25519 kianiadee@stage.rahamafresh.com`
|
||||
- **DB name:** `tracksolid_db` · **DB user:** `postgres` (internal) · `tracksolid_owner` (app) · `grafana_ro` (read-only)
|
||||
- **DB schemas:** `tracksolid` (live, single source of truth) · `reporting` (map-dashboard read layer) · `infrastructure`. The legacy `tracksolid_2` schema no longer exists (migrations 02–06, 2026-04-18); the `ops` and `dwh_gold` schemas were purged 2026-06-05 (migrations 12/13) as unused.
|
||||
- **DB schemas:** `tracksolid` (live, single source of truth) · `dwh_gold` (aggregates) · `ops` (workshop / tickets / odometer) · `infrastructure`. The legacy `tracksolid_2` schema no longer exists — migrations 02–06 applied 2026-04-18.
|
||||
- **DB access:** `DATABASE_URL` points to `timescale_db:5432` (internal Docker network — not reachable locally). Use `docker exec` pattern above. See `docs/CONNECTIONS.md` for full reference.
|
||||
- **DWH target DB:** `tracksolid_dwh` at `31.97.44.246:5888` (separate PostGIS instance, public IP). Users: `dwh_owner` (bronze writes + `dwh_control`), `grafana_ro` (reads bronze/silver/gold/`dwh_control`). Always connect with `sslmode=require`. Fed by the n8n `dwh_extract` + `dwh_load_bronze` workflows — see `docs/DWH_PIPELINE.md`.
|
||||
- **Container naming:** Coolify appends a random suffix. Always resolve with:
|
||||
|
|
@ -66,25 +66,6 @@ See `docs/CONNECTIONS.md` for the full shape. Summary:
|
|||
e.g. `docker ps --filter name=timescale_db --format "{{.Names}}" | head -1`
|
||||
- **Env vars:** loaded from `.env` via `env_file` in `docker-compose.yaml`. See `docs/CONNECTIONS.md` for variable names. Never hardcode secrets.
|
||||
|
||||
### Map dashboards & read-API
|
||||
|
||||
The map UIs read the **`dashboard_api`** service (FastAPI, `dashboard_api_rev.py`) at
|
||||
`https://fleetapi.rahamafresh.com` — the stable replacement for the retired n8n webhooks. It serves
|
||||
GeoJSON from the `reporting.*` functions (`fn_live_positions`, `fn_vehicle_track`, `fn_trips_for_map`)
|
||||
+ filter options. **`dashboard_api` is a STANDALONE Traefik-labelled bridge container, NOT Coolify-managed** —
|
||||
it bind-mounts the host file `~/dashboard_api/dashboard_api_rev.py` and is (re)deployed by
|
||||
`~/deploy_dashboard_api.sh` on the host (an env/CORS change needs a *recreate*, not a restart). Three
|
||||
single-page apps consume it:
|
||||
|
||||
| Dashboard | What | Hosting |
|
||||
|---|---|---|
|
||||
| `liveposition.rahamafresh.com` | live positions only | `index.html` in rustfs bucket `liveposition` behind an nginx proxy |
|
||||
| `fleetintelligence.rahamafresh.com` | historical trips only | `index.html` in rustfs bucket `fleetintelligence` behind an nginx proxy |
|
||||
| `fleetnow.rahamafresh.com` | **merged** live + trips (current best UI) | **own repo** `repo.rahamafresh.com/kianiadee/fleetnow.git`, deployed via **Coolify (Dockerfile → nginx)** |
|
||||
|
||||
All three origins must be in the API's `DASHBOARD_CORS_ORIGINS` (see FIX-D03). **FleetNow is the
|
||||
single source of truth for the merged map and lives in its own repo — edit it there, not here.**
|
||||
|
||||
---
|
||||
|
||||
## 4. Codebase Map
|
||||
|
|
@ -102,32 +83,25 @@ docker-compose.yaml # Services: timescale_db, ingest_movement, ingest_ev
|
|||
grafana/ # Grafana provisioning (baked into image)
|
||||
n8n-workflows/ # n8n workflow exports (incl. dwh_extract, dwh_load_bronze)
|
||||
docs/ # Reference docs (connections, API, KPIs, project context)
|
||||
docs/PLATFORM_OVERVIEW.html # Current-state platform reference (architecture, deploy, read-API,
|
||||
# full DB schema, Grafana panels) — open in a browser. Post n8n→fleetapi.
|
||||
docs/DWH_PIPELINE.md # DWH pipeline operations runbook (setup, troubleshooting)
|
||||
docs/OSM_POI_EXPORT.md # Runbook: OSM .pbf → POI GeoJSON → FleetNow map layer (Shell stations)
|
||||
docs/superpowers/ # Pitch specs and implementation plans (not deployed code)
|
||||
scripts/export_osm_pois.py # OSM .pbf → GeoJSON+CSV POI exporter (amenity/brand filter); see OSM_POI_EXPORT.md
|
||||
dwh/ # DWH migrations for tracksolid_dwh@31.97.44.246:5888
|
||||
# 260423_dwh_ddl_v1.sql — bronze/silver/gold schemas + roles
|
||||
# 261001_dwh_control.sql — watermarks + run log
|
||||
# 261002_bronze_constraints_audit.sql — ON CONFLICT key assertion
|
||||
# 261003_dwh_roles.sql — role contract assertion
|
||||
# 261004_dwh_observability_views.sql — freshness/failure views
|
||||
migrations/ # Numbered SQL migrations 02–13, applied in order by run_migrations.py
|
||||
# 02 full schema · 03 webhook · 04 distance fix · 05 enhancements
|
||||
# 06 ops/analytics · 07 views · 08 config · 09 trips enrichment
|
||||
# 10_driver_clock_views.sql · 10_pgbouncer_auth.sql · 11 reporting
|
||||
# 12 drop ops schema · 13 drop dwh_gold schema (both 2026-06-05)
|
||||
02_tracksolid_full_schema_rev.sql # Full schema bootstrap
|
||||
03..06_*.sql # Incremental migrations (06 adds assigned_city, dispatch_log, ops.*)
|
||||
07_analytics_views.sql # Analytics views migration (applied 2026-04-21)
|
||||
Dockerfile # Custom image for ingest/webhook containers
|
||||
pyproject.toml # Python project + uv dependency spec
|
||||
OPERATIONS_MANUAL.md # Day-to-day ops runbook
|
||||
backup/ # pg_dump sidecar scripts and config
|
||||
data/ # Source CSVs (FS Logistics 144-device list, FSG vehicles)
|
||||
legacy/ # Superseded pre-_rev scripts + old pipeline notes (NOT deployed)
|
||||
docs/manuals/ # OPERATIONS_MANUAL, grafana + DWH manuals, docker commands, DB manual
|
||||
docs/reference/ # 01_BusinessAnalytics.md (SQL library — read before writing queries),
|
||||
# tracksolidApiDocumentation.md, 260507_pgbouncer_deployment.md
|
||||
docs/reports/ # Baseline reports, audit output, improvement reviews
|
||||
01_BusinessAnalytics.md # SQL analytics library — read before writing queries
|
||||
20260414_FS__Logistics - final_fixed.csv # 144-device driver/vehicle source data
|
||||
tracksolidApiDocumentation.md # API endpoint reference
|
||||
260412_baseline_report.md # Fleet state snapshot (Apr 2026)
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -148,13 +122,12 @@ tracksolid.alarms -- Alarm events (alarm_type, alarm_name, alarm_time
|
|||
tracksolid.obd_readings -- OBD diagnostics (push only, awaiting webhook registration)
|
||||
tracksolid.device_events -- Power on/off tamper events (push only)
|
||||
tracksolid.ingestion_log -- API call audit trail — 875 runs / 24h, 0 failures at last check (2026-04-19)
|
||||
tracksolid.schema_migrations -- Applied migrations 02–13
|
||||
-- PURGED 2026-06-05 (migrations 12 + 13): the dormant `ops` schema (tickets, service_log,
|
||||
-- odometer_readings, cost_rates, kpi_targets, vw_service_forecast), tracksolid.dispatch_log,
|
||||
-- and the `dwh_gold` schema (dim_vehicles, fact_daily_fleet_metrics, refresh_daily_metrics).
|
||||
-- Those workshop/dispatch/SLA/utilisation features were never implemented. Do NOT reintroduce
|
||||
-- references to ops.* or dwh_gold.* — they no longer exist. (The separate tracksolid_dwh DB
|
||||
-- at 31.97.44.246:5888 is unrelated and untouched.)
|
||||
tracksolid.dispatch_log -- Dispatch decisions for SLA tracking (migration 06; empty until ops integration)
|
||||
tracksolid.schema_migrations -- Applied files: 02,03,04,05,06 (last 06 on 2026-04-18)
|
||||
dwh_gold.fact_daily_fleet_metrics -- Nightly ETL aggregates per vehicle per day (run refresh_daily_metrics)
|
||||
ops.service_log -- Workshop service history (migration 06)
|
||||
ops.odometer_readings -- Physical odometer captures (migration 06)
|
||||
ops.tickets -- Ticket skeleton for ops integration (migration 06; empty)
|
||||
```
|
||||
|
||||
Full DDL: `02_tracksolid_full_schema_rev.sql` + migrations `03`–`06`.
|
||||
|
|
@ -169,8 +142,8 @@ tracksolid.v_currently_idle -- §2.2 idle lens
|
|||
tracksolid.v_driver_aggregates_daily -- §3.1 + §3.2 aggression index source
|
||||
tracksolid.v_fleet_km_daily -- §7 Panel 5 distance trend
|
||||
tracksolid.v_alarms_daily -- §7 Panel 7 alarm frequency
|
||||
-- v_utilisation_daily (dwh_gold) and v_sla_inflight (ops) were DROPPED 2026-06-05 with
|
||||
-- their schemas (migrations 12/13); their Grafana panels were removed from the dashboard.
|
||||
tracksolid.v_utilisation_daily -- §7 Panel 8 utilisation heatmap (gated on dwh_gold ETL)
|
||||
tracksolid.v_sla_inflight -- §4.5 SLA panels (gated on ops.tickets)
|
||||
```
|
||||
|
||||
All views carry a `COMMENT ON VIEW` referencing their spec — `\d+ tracksolid.v_*` shows the provenance.
|
||||
|
|
@ -198,7 +171,7 @@ dwh_control.v_watermark_lag -- Grafana: extract vs. load lag per table
|
|||
|
||||
## 6. API Critical Facts
|
||||
|
||||
**Always read `docs/reference/tracksolidApiDocumentation.md` before adding a new endpoint call.**
|
||||
**Always read `tracksolidApiDocumentation.md` before adding a new endpoint call.**
|
||||
|
||||
| Fact | Detail |
|
||||
|---|---|
|
||||
|
|
@ -229,9 +202,6 @@ dwh_control.v_watermark_lag -- Grafana: extract vs. load lag per table
|
|||
| FIX-M19 | `ts_shared_rev.py`, `ingest_movement_rev.py` | Multi-account support: fleet spans `fireside`, `Fireside@HQ`, `Fireside_MSA` (156 devices total). `sync_devices`, `poll_live_positions`, `poll_parking` iterate `TRACKSOLID_TARGETS` (comma-separated env var). New helper `get_active_imeis_by_target()` scopes parking calls to the right account |
|
||||
| FIX-E06 | `ingest_events_rev.py` | Alarm field mapping: `alertTypeId`/`alarmTypeName`/`alertTime` |
|
||||
| BUG-02 | Migration 04 | Historical `distance_m` rows ÷1,000,000 → renamed to `distance_km` |
|
||||
| FIX-D01 | `dashboard_api_rev.py` | `POST /webhook/fleet-dashboard` read body as JSON, but the SPA posts `x-www-form-urlencoded` → `request.json()` threw, filters silently dropped, map always returned the whole fleet. Now parsed by Content-Type (`parse_qs` for form, JSON still accepted). Commit `f1387d1` |
|
||||
| FIX-D02 | `dashboard_api_rev.py` | `reporting.v_trips` matview froze on 2026-06-01 when n8n (which ran the scheduled refresh) was retired → dashboard showed "no trips". Added an in-process background refresher (`REFRESH MATERIALIZED VIEW CONCURRENTLY` every `VTRIPS_REFRESH_INTERVAL_S`, default 300s; pg advisory-lock guarded for `--workers`; logs to `reporting.refresh_log` source=`dashboard_api`). Commit `30b3515` |
|
||||
| FIX-D03 | `dashboard_api_rev.py`, `~/deploy_dashboard_api.sh` (host) | Added `https://fleetnow.rahamafresh.com` to `DASHBOARD_CORS_ORIGINS` default for the merged **FleetNow** dashboard. The standalone bridge container inherits its env from `webhook_receiver`, which already carries the old two-origin value — so the deploy script's *conditional* append never fired. The script now **strips any inherited `DASHBOARD_CORS_ORIGINS` and sets all three origins unconditionally**, and **guards the `mv`** so a missing staged `dashboard_api_rev.py` doesn't abort the run under `set -e` (env changes need a container *recreate*, not a restart). Commit `d95e5c2` |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -239,14 +209,14 @@ dwh_control.v_watermark_lag -- Grafana: extract vs. load lag per table
|
|||
|
||||
1. **No prod push without explicit user confirmation.** Always state what you are about to push and wait.
|
||||
2. **Never rewrite a migration that is already applied.** Check `tracksolid.schema_migrations` first. Add a new numbered migration file for any schema change.
|
||||
3. **Read before writing.** Before suggesting any code change, read the relevant source file. Before writing a query, check `docs/reference/01_BusinessAnalytics.md` for an existing pattern.
|
||||
3. **Read before writing.** Before suggesting any code change, read the relevant source file. Before writing a query, check `01_BusinessAnalytics.md` for an existing pattern.
|
||||
4. **Reuse shared utilities.** All DB access via `get_conn()`, all API calls via `api_post()`, all cleaning via `clean()` / `clean_num()` / `clean_int()` / `clean_ts()` in `ts_shared_rev.py`. Do not reinvent these.
|
||||
5. **Resolve container names dynamically.** Never hardcode the Coolify suffix. Use `docker ps --filter name=<service>`.
|
||||
6. **SSH only when asked.** Default workflow is local code → commit → push. SSH into the instance only when explicitly asked to test or run something live.
|
||||
7. **Secrets from env only.** Connection strings, API keys, and passwords live in `.env`. Reference variable names from `docs/CONNECTIONS.md`, never values.
|
||||
8. **Two developers, one incoming.** Write code and docs that a second developer (mixed technical/operations background) can follow without prior context.
|
||||
9. **Forgejo API auth:** credentials stored in macOS keychain. Retrieve with `git credential fill` (host=repo.rahamafresh.com). Use basic auth against `https://repo.rahamafresh.com/api/v1` directly — no `tea` or `gh` needed.
|
||||
10. **Single live schema.** All live data lives in `tracksolid`; the map-dashboard read layer lives in `reporting`. Do not reintroduce references to the retired `tracksolid_2`, `ops`, or `dwh_gold` schemas (the latter two purged 2026-06-05, migrations 12/13).
|
||||
10. **Single live schema.** All live data lives in `tracksolid`. Aggregates live in `dwh_gold`; workshop/ticket integrations live in `ops`. Do not reintroduce references to the retired `tracksolid_2` schema.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -265,7 +235,7 @@ dwh_control.v_watermark_lag -- Grafana: extract vs. load lag per table
|
|||
| Cities active | Nairobi (primary), Mombasa (deploying), Kampala (4 devices in CSV) |
|
||||
| Service flags | KDK 829A GP (239,264 km), Belta KCU-647D (235,000 km) |
|
||||
|
||||
Latest full snapshot: `docs/reports/260412_baseline_report.md`
|
||||
Latest full snapshot: `260412_baseline_report.md`
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -273,7 +243,6 @@ Latest full snapshot: `docs/reports/260412_baseline_report.md`
|
|||
|
||||
| Priority | Item |
|
||||
|---|---|
|
||||
| HIGH | **Redeploy the Grafana service in Coolify** to apply `daily_operations_dashboard.json` — 5 panel areas (In-flight SLA, Idle Cost, Utilisation Heatmap, Row 7 Field-Service SLAs) that queried the now-dropped `v_sla_inflight`/`v_utilisation_daily` were removed. The DB views are already gone, so **live Grafana shows errors on those panels until the redeploy** (purge commit `8c5a43f`, 2026-06-05). |
|
||||
| HIGH | Run `import_drivers_csv.py --apply` — 144 X3/JC400P devices with names + plates waiting |
|
||||
| HIGH | Register webhooks: `/pushoil` `/pushtem` `/pushlbs` (auto-register on push now done — commit 257643c) |
|
||||
| HIGH | Investigate X3-63282 in Kampala — legitimate or unauthorised? |
|
||||
|
|
@ -281,5 +250,6 @@ Latest full snapshot: `docs/reports/260412_baseline_report.md`
|
|||
| MEDIUM | Investigate 44 silent devices (only 19 of 63 reporting) — SIM installed? Activated? |
|
||||
| MEDIUM | Co-develop client KPI framework (see `docs/KPI_FRAMEWORK.md`) |
|
||||
| LOW | Populate geofences — depot boundaries, city zones |
|
||||
| LOW | Schedule nightly ETL: `SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1)` (cron or n8n) |
|
||||
| HIGH | Deploy DWH bronze pipeline: apply `dwh/26100{1,2,3,4}.sql` to `tracksolid_dwh`, import + wire the two n8n workflows, verify first run via `dwh_control.v_table_freshness`. Runbook: `docs/DWH_PIPELINE.md` |
|
||||
| MEDIUM | Rotate `dwh_owner` / `grafana_ro` passwords on `tracksolid_dwh` — plaintext in `dwh/260423_dwh_ddl_v1.sql` is a pre-existing flaw to clean up separately |
|
||||
|
|
|
|||
|
|
@ -1,324 +0,0 @@
|
|||
"""
|
||||
dashboard_api_rev.py — Fireside Communications · Map Dashboard Read API
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
Stable replacement for the n8n webhooks that fed the Live Position and Fleet
|
||||
Trips map dashboards. n8n was acting only as a thin HTTP→SQL proxy; this
|
||||
service does the same job directly against the proven reporting.* functions,
|
||||
removing n8n's credential-management / reload / version-drift fragility from
|
||||
the live-data path.
|
||||
|
||||
It REUSES the existing stack: ts_shared_rev's psycopg2 pool and DATABASE_URL,
|
||||
the same Docker image, the same Coolify deploy. The reporting.* functions
|
||||
(already verified to return correct GeoJSON) are the single source of truth.
|
||||
|
||||
Endpoints mirror the original n8n webhook paths so the only frontend change
|
||||
is the base URL (the `N8N_BASE` constant in each dashboard SPA):
|
||||
|
||||
GET /webhook/live-positions?cost_centre=&acc_status=
|
||||
→ { summary, geojson } (reporting.fn_live_positions)
|
||||
GET /webhook/live-positions/track?vehicle_number=&hours=
|
||||
(alias: /webhook/vehicle-track)
|
||||
→ GeoJSON Feature (reporting.fn_vehicle_track)
|
||||
GET /webhook/fleet-dashboard
|
||||
→ { drivers, cost_centres, cities, vehicles } (filter options)
|
||||
POST /webhook/fleet-dashboard body: {period, vehicle_numbers, driver,
|
||||
cost_centre, assigned_city,
|
||||
start_date, end_date}
|
||||
→ trips payload (reporting.fn_trips_for_map)
|
||||
GET /health
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
from urllib.parse import parse_qs
|
||||
|
||||
import psycopg2
|
||||
import psycopg2.extras
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from ts_shared_rev import close_pool, get_conn, get_logger
|
||||
|
||||
log = get_logger("dashboard_api")
|
||||
|
||||
# Comma-separated list of allowed browser origins (the dashboard domains).
|
||||
_ALLOWED_ORIGINS = [
|
||||
o.strip()
|
||||
for o in os.getenv(
|
||||
"DASHBOARD_CORS_ORIGINS",
|
||||
"https://liveposition.rahamafresh.com,https://fleetintelligence.rahamafresh.com,https://fleetnow.rahamafresh.com",
|
||||
).split(",")
|
||||
if o.strip()
|
||||
]
|
||||
|
||||
|
||||
# ── v_trips materialized-view refresher ─────────────────────────────────────
|
||||
# The Fleet Trips dashboard reads reporting.v_trips (a materialized view). Its
|
||||
# refresh used to be a scheduled n8n workflow; when n8n was retired the matview
|
||||
# went stale (data froze). We now keep it fresh in-process: a background loop
|
||||
# refreshes it on an interval. A Postgres advisory lock makes this safe across
|
||||
# uvicorn workers (only one worker refreshes per tick); the work runs in a
|
||||
# thread so the async event loop never blocks on the ~9s REFRESH.
|
||||
_DATABASE_URL = os.environ["DATABASE_URL"]
|
||||
_REFRESH_INTERVAL_S = int(os.getenv("VTRIPS_REFRESH_INTERVAL_S", "300"))
|
||||
_REFRESH_LOCK_KEY = 920_145 # arbitrary, stable advisory-lock key for this job
|
||||
|
||||
|
||||
def _refresh_v_trips_once() -> str:
|
||||
"""Refresh reporting.v_trips. Blocking — call via asyncio.to_thread.
|
||||
|
||||
Uses a dedicated autocommit connection: REFRESH ... CONCURRENTLY cannot run
|
||||
inside a transaction block (so the pooled get_conn, which wraps a txn, won't
|
||||
do). DATABASE_URL connects as a superuser, which may REFRESH the matview
|
||||
even though reporting_refresher owns it.
|
||||
"""
|
||||
conn = psycopg2.connect(_DATABASE_URL, connect_timeout=10)
|
||||
try:
|
||||
conn.autocommit = True
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT pg_try_advisory_lock(%s)", (_REFRESH_LOCK_KEY,))
|
||||
if not cur.fetchone()[0]:
|
||||
return "skipped (another worker holds the lock)"
|
||||
try:
|
||||
t0 = time.monotonic()
|
||||
cur.execute("REFRESH MATERIALIZED VIEW CONCURRENTLY reporting.v_trips")
|
||||
dur_ms = int((time.monotonic() - t0) * 1000)
|
||||
cur.execute(
|
||||
"INSERT INTO reporting.refresh_log"
|
||||
"(refreshed_at, source, duration_ms, row_count, notes) "
|
||||
"VALUES (now(), 'dashboard_api', %s,"
|
||||
" (SELECT count(*) FROM reporting.v_trips), 'scheduled')",
|
||||
(dur_ms,),
|
||||
)
|
||||
return f"refreshed in {dur_ms}ms"
|
||||
finally:
|
||||
cur.execute("SELECT pg_advisory_unlock(%s)", (_REFRESH_LOCK_KEY,))
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
async def _refresh_loop():
|
||||
# Brief startup delay so the first refresh doesn't race container init.
|
||||
await asyncio.sleep(15)
|
||||
while True:
|
||||
try:
|
||||
result = await asyncio.to_thread(_refresh_v_trips_once)
|
||||
log.info("v_trips refresh: %s", result)
|
||||
except Exception:
|
||||
log.exception("v_trips refresh failed (will retry next interval)")
|
||||
await asyncio.sleep(_REFRESH_INTERVAL_S)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
log.info(
|
||||
"Dashboard API starting (v1.1). Origins=%s. v_trips refresh every %ss.",
|
||||
_ALLOWED_ORIGINS, _REFRESH_INTERVAL_S,
|
||||
)
|
||||
refresher = asyncio.create_task(_refresh_loop())
|
||||
yield
|
||||
refresher.cancel()
|
||||
try:
|
||||
await refresher
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
close_pool()
|
||||
|
||||
|
||||
app = FastAPI(title="Fireside Map Dashboard API", lifespan=lifespan)
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=_ALLOWED_ORIGINS,
|
||||
allow_methods=["GET", "POST", "OPTIONS"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
_EMPTY_GEOJSON = {"type": "FeatureCollection", "features": []}
|
||||
|
||||
|
||||
def _clean(v):
|
||||
"""Treat missing / blank / sentinel values as None (= SQL wildcard)."""
|
||||
if v is None:
|
||||
return None
|
||||
s = str(v).strip()
|
||||
return s if s and s.lower() not in ("null", "undefined") else None
|
||||
|
||||
|
||||
# ── Health ────────────────────────────────────────────────────────────────────
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
# ── Live positions (#004) ───────────────────────────────────────────────────
|
||||
|
||||
@app.get("/webhook/live-positions")
|
||||
def live_positions(cost_centre: str | None = None, acc_status: str | None = None):
|
||||
try:
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"SELECT reporting.fn_live_positions(%s, %s)",
|
||||
(_clean(cost_centre), _clean(acc_status)),
|
||||
)
|
||||
payload = cur.fetchone()[0] or {}
|
||||
return JSONResponse(
|
||||
{
|
||||
"summary": payload.get("summary") or {},
|
||||
"geojson": payload.get("geojson") or _EMPTY_GEOJSON,
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
log.exception("live-positions failed")
|
||||
return JSONResponse(
|
||||
{
|
||||
"error": {
|
||||
"type": "unknown",
|
||||
"message": "Live-position feed is unavailable. Try again in a few seconds.",
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# `/webhook/live-positions/track` is the path the Live Positions SPA actually
|
||||
# calls; `/webhook/vehicle-track` is kept as an alias. Both hit the same handler
|
||||
# so the only frontend change is the base URL (N8N_BASE).
|
||||
@app.get("/webhook/live-positions/track")
|
||||
@app.get("/webhook/vehicle-track")
|
||||
def vehicle_track(vehicle_number: str | None = None, hours: int = 1):
|
||||
veh = _clean(vehicle_number)
|
||||
if not veh:
|
||||
return JSONResponse({"error": "vehicle_number is required"})
|
||||
hours = max(1, min(24, hours or 1))
|
||||
try:
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"SELECT reporting.fn_vehicle_track(%s, %s::int)", (veh, hours)
|
||||
)
|
||||
feature = cur.fetchone()[0]
|
||||
return JSONResponse(
|
||||
feature
|
||||
or {"type": "Feature", "geometry": {"type": "LineString", "coordinates": []}, "properties": {}}
|
||||
)
|
||||
except Exception:
|
||||
log.exception("vehicle-track failed for %s", veh)
|
||||
return JSONResponse({"error": "vehicle-track unavailable"})
|
||||
|
||||
|
||||
# ── Fleet trips (#002) ───────────────────────────────────────────────────────
|
||||
|
||||
_FILTER_OPTIONS_SQL = """
|
||||
SELECT
|
||||
(SELECT array_agg(driver ORDER BY driver) FROM reporting.v_filter_drivers) AS drivers,
|
||||
(SELECT array_agg(cost_centre ORDER BY cost_centre) FROM reporting.v_filter_cost_centres) AS cost_centres,
|
||||
(SELECT array_agg(assigned_city ORDER BY assigned_city) FROM reporting.v_filter_cities) AS cities,
|
||||
(SELECT jsonb_agg(jsonb_build_object(
|
||||
'vehicle_number', vehicle_number, 'drivers', drivers,
|
||||
'cost_centre', cost_centre, 'assigned_city', assigned_city)
|
||||
ORDER BY vehicle_number) FROM reporting.v_filter_vehicles) AS vehicles
|
||||
"""
|
||||
|
||||
|
||||
@app.get("/webhook/fleet-dashboard")
|
||||
def fleet_filter_options():
|
||||
try:
|
||||
with get_conn() as conn:
|
||||
with conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cur:
|
||||
cur.execute(_FILTER_OPTIONS_SQL)
|
||||
row = cur.fetchone() or {}
|
||||
return JSONResponse(
|
||||
{
|
||||
"drivers": row.get("drivers") or [],
|
||||
"cost_centres": row.get("cost_centres") or [],
|
||||
"cities": row.get("cities") or [],
|
||||
"vehicles": row.get("vehicles") or [],
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
log.exception("fleet-dashboard filter options failed")
|
||||
return JSONResponse({"drivers": [], "cost_centres": [], "cities": [], "vehicles": []})
|
||||
|
||||
|
||||
def _preset_to_range(period: str | None, start_date, end_date):
|
||||
"""Mirror of the n8n preset_to_range node."""
|
||||
today = datetime.now(timezone.utc).date()
|
||||
p = (period or "").strip()
|
||||
if p == "today":
|
||||
return today, today
|
||||
if p == "30d":
|
||||
return today - timedelta(days=29), today
|
||||
if p == "custom":
|
||||
def _d(v, default):
|
||||
v = _clean(v)
|
||||
if not v:
|
||||
return default
|
||||
try:
|
||||
return date.fromisoformat(v)
|
||||
except ValueError:
|
||||
return default
|
||||
return _d(start_date, today), _d(end_date, today)
|
||||
# default + '7d'
|
||||
return today - timedelta(days=6), today
|
||||
|
||||
|
||||
@app.post("/webhook/fleet-dashboard")
|
||||
async def fleet_trips(request: Request):
|
||||
# The dashboard SPA posts application/x-www-form-urlencoded (not JSON), so
|
||||
# parse by content-type. Reading the raw body + parse_qs avoids pulling in
|
||||
# python-multipart. JSON is still accepted defensively (n8n-compat callers).
|
||||
body: dict = {}
|
||||
ctype = request.headers.get("content-type", "").lower()
|
||||
try:
|
||||
raw = await request.body()
|
||||
if "application/json" in ctype:
|
||||
parsed = json.loads(raw or b"{}")
|
||||
body = parsed if isinstance(parsed, dict) else {}
|
||||
else:
|
||||
# x-www-form-urlencoded — parse_qs yields lists; keep the last value.
|
||||
body = {k: v[-1] for k, v in parse_qs(raw.decode("utf-8", "replace")).items()}
|
||||
except Exception:
|
||||
body = {}
|
||||
if isinstance(body.get("body"), dict):
|
||||
body = body["body"]
|
||||
|
||||
start, end = _preset_to_range(
|
||||
body.get("period"), body.get("start_date"), body.get("end_date")
|
||||
)
|
||||
veh = _clean(body.get("vehicle_numbers")) # comma-separated string or None
|
||||
try:
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT reporting.fn_trips_for_map(
|
||||
CASE WHEN %(veh)s IS NULL THEN NULL
|
||||
ELSE string_to_array(%(veh)s, ',') END,
|
||||
%(driver)s, %(cc)s, %(city)s, %(start)s::date, %(end)s::date
|
||||
)
|
||||
""",
|
||||
{
|
||||
"veh": veh,
|
||||
"driver": _clean(body.get("driver")),
|
||||
"cc": _clean(body.get("cost_centre")),
|
||||
"city": _clean(body.get("assigned_city")),
|
||||
"start": start,
|
||||
"end": end,
|
||||
},
|
||||
)
|
||||
payload = cur.fetchone()[0]
|
||||
return JSONResponse(payload if payload is not None else {})
|
||||
except Exception:
|
||||
log.exception("fleet-dashboard trips failed")
|
||||
return JSONResponse(
|
||||
{"error": {"type": "unknown", "message": "Fleet feed is unavailable. Try again in a few seconds."}}
|
||||
)
|
||||
|
|
@ -59,31 +59,6 @@ services:
|
|||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
dashboard_api:
|
||||
# Stable read-API for the Live Position + Fleet Trips map dashboards.
|
||||
# Replaces the n8n webhooks (n8n was only a thin HTTP->SQL proxy).
|
||||
# Calls reporting.fn_live_positions / fn_vehicle_track / fn_trips_for_map.
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
command: sh -c "uvicorn dashboard_api_rev:app --host 0.0.0.0 --port 8890 --workers 2"
|
||||
restart: always
|
||||
depends_on:
|
||||
timescale_db:
|
||||
condition: service_healthy
|
||||
env_file: .env
|
||||
environment:
|
||||
# Browser origins allowed to call this API (the dashboard domains).
|
||||
- DASHBOARD_CORS_ORIGINS=${DASHBOARD_CORS_ORIGINS:-https://liveposition.rahamafresh.com,https://fleetintelligence.rahamafresh.com}
|
||||
# No host port binding — set a domain (e.g. fleetapi.rahamafresh.com) in the
|
||||
# Coolify UI pointing to this service on port 8890. The dashboards then point
|
||||
# their N8N_BASE at that domain; paths (/webhook/...) are unchanged.
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8890/health"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
grafana:
|
||||
build:
|
||||
context: ./grafana
|
||||
|
|
@ -106,7 +81,7 @@ services:
|
|||
|
||||
pgbouncer:
|
||||
# Connection pooler in front of timescale_db.
|
||||
# Runbook: docs/reference/260507_pgbouncer_deployment.md
|
||||
# Runbook: 260507_pgbouncer_deployment.md
|
||||
# Internal Docker network only — no host port. SCRAM passthrough via
|
||||
# auth_query against the public.user_lookup() function (migration 10).
|
||||
image: edoburu/pgbouncer
|
||||
|
|
|
|||
|
|
@ -1,240 +0,0 @@
|
|||
# Data Flow — Ingestion → Aggregation → Views → Functions → Consumers
|
||||
|
||||
**Scope:** the *live* fleet pipeline only (`tracksolid` + `reporting`). The `ops` and
|
||||
`dwh_gold` schemas were **purged on 2026-06-05** (migrations 12/13) — those workshop /
|
||||
dispatch / SLA / utilisation features were never implemented. They are shown below only as
|
||||
a struck-out "removed" footnote for historical context. (The *separate* `tracksolid_dwh`
|
||||
server at 31.97.44.246:5888 is unrelated and was not touched.)
|
||||
|
||||
**Verified against prod 2026-06-05** (TimescaleDB hypertable + continuous-aggregate
|
||||
catalog, `pg_depend` view graph, ingestion `INSERT` targets, `dashboard_api` queries,
|
||||
Grafana panel SQL). Key facts that surprised the docs:
|
||||
|
||||
- Only `position_history` (and the empty/planned `heartbeats`, `fuel_readings`,
|
||||
`temperature_readings`) are **hypertables**. `trips` and `alarms` are **plain tables**.
|
||||
- `tracksolid.v_mileage_daily_cagg` is a **real TimescaleDB continuous aggregate**, not a
|
||||
plain view — and it currently has **no downstream consumer**.
|
||||
- `reporting.v_trips` is a **matview**, refreshed every 5 min by the in-process
|
||||
`dashboard_api` background loop (FIX-D02), pg advisory-lock `920145`.
|
||||
|
||||
---
|
||||
|
||||
## Mermaid
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
API["Tracksolid / Jimi API<br/>poll + push webhooks · OAuth2"]
|
||||
|
||||
subgraph ING["Ingestion — ts_shared_rev.py: get_conn() · api_post() · clean*()"]
|
||||
IM["ingest_movement_rev.py"]
|
||||
IE["ingest_events_rev.py"]
|
||||
WR["webhook_receiver_rev.py"]
|
||||
end
|
||||
API --> IM & IE & WR
|
||||
|
||||
subgraph L1["L1 · Base tables + hypertables — schema: tracksolid (single source of truth)"]
|
||||
LP["live_positions<br/>current fix / IMEI"]
|
||||
PH[("position_history<br/>HYPERTABLE · high-res GPS trail")]
|
||||
TR["trips<br/>plain table — NOT a hypertable"]
|
||||
DV["devices<br/>vehicle / driver registry"]
|
||||
AL["alarms<br/>plain table"]
|
||||
ILOG["ingestion_log · api_token_cache"]
|
||||
EMPTY["heartbeats* · fuel_readings* · temperature_readings* (HYPERTABLES, empty)<br/>device_events · fault_codes · obd_readings · parking_events · lbs_readings · geofences (empty)"]
|
||||
end
|
||||
IM --> LP & PH & TR & DV
|
||||
IM --> ILOG
|
||||
IE --> AL
|
||||
WR --> EMPTY
|
||||
|
||||
subgraph L2["L2 · Aggregation"]
|
||||
CAGG[("v_mileage_daily_cagg<br/>CONTINUOUS AGGREGATE")]
|
||||
VT[("reporting.v_trips<br/>MATVIEW · ix_v_trips_trip_id")]
|
||||
RLOG["reporting.refresh_log"]
|
||||
end
|
||||
PH -->|Timescale cont-agg policy| CAGG
|
||||
TR --> VT
|
||||
DV --> VT
|
||||
VT -->|"REFRESH CONCURRENTLY every 300s<br/>(dashboard_api loop, adv-lock 920145)"| RLOG
|
||||
|
||||
subgraph L3L["L3 · Reporting views — fed by v_trips matview"]
|
||||
FILT["v_filter_vehicles · v_filter_drivers<br/>v_filter_cost_centres · v_filter_cities"]
|
||||
SUMM["v_daily/weekly/monthly_summary<br/>v_*_cost_centre · v_trips_today"]
|
||||
VLP["v_live_positions"]
|
||||
end
|
||||
VT --> FILT & SUMM
|
||||
LP --> VLP
|
||||
|
||||
subgraph L3R["L3 · Grafana views — tracksolid.* (read base tables directly)"]
|
||||
GV["v_fleet_today · v_fleet_status · v_active_dispatch_map<br/>v_currently_idle · v_alarms_daily · v_fleet_km_daily<br/>v_ingestion_health · v_vehicles_not_moved_today<br/>v_driver_aggregates_daily · v_fleet_trace · v_driver_clock_*"]
|
||||
end
|
||||
LP --> GV
|
||||
TR --> GV
|
||||
DV --> GV
|
||||
AL --> GV
|
||||
PH --> GV
|
||||
|
||||
subgraph L4["L4 · Functions — reporting.* (the only API entrypoints)"]
|
||||
FLP["fn_live_positions(cost_centre, acc_status)"]
|
||||
FVT["fn_vehicle_track(vehicle_number, hours)"]
|
||||
FTM["fn_trips_for_map(veh[], driver, cc, city, start, end)"]
|
||||
NP["normalize_plate(p) · helper"]
|
||||
end
|
||||
LP --> FLP
|
||||
DV --> FLP
|
||||
PH --> FVT
|
||||
VT --> FTM
|
||||
NP -.-> FTM
|
||||
|
||||
subgraph L5["L5 · Consumers"]
|
||||
DAPI["dashboard_api_rev.py<br/>FastAPI :8890 · 2 workers"]
|
||||
SPA["SPAs: liveposition.* · fleetintelligence.*<br/>(rustfs / S3 single-file maps)"]
|
||||
GRAF["Grafana"]
|
||||
end
|
||||
FLP --> DAPI
|
||||
FVT --> DAPI
|
||||
FTM --> DAPI
|
||||
FILT --> DAPI
|
||||
DAPI -->|"HTTPS · fleetapi.rahamafresh.com"| SPA
|
||||
GV --> GRAF
|
||||
|
||||
CAGG -.->|no consumer yet| NONE(["⚠ unconsumed — no panel / API reads it"])
|
||||
|
||||
subgraph PARK["REMOVED 2026-06-05 — purged via migrations 12 / 13 (never implemented)"]
|
||||
GONE["ops.* (tickets · dispatch_log · service_log · odometer_readings · cost_rates · kpi_targets · vw_service_forecast)<br/>dwh_gold.* (dim_vehicles · fact_daily_fleet_metrics · refresh_daily_metrics)<br/>tracksolid.v_sla_inflight · tracksolid.v_utilisation_daily + their Grafana panels"]
|
||||
end
|
||||
|
||||
classDef hyper fill:#e1f0ff,stroke:#3b82f6,color:#0b3d91;
|
||||
classDef mat fill:#fff3cd,stroke:#d4a017,color:#664d03;
|
||||
classDef cagg fill:#e7f7e7,stroke:#2e9e2e,color:#1e5e1e;
|
||||
classDef parked fill:#f0f0f0,stroke:#999,stroke-dasharray:5 5,color:#555;
|
||||
classDef warn fill:#fdecea,stroke:#d93025,color:#a52714;
|
||||
class PH hyper;
|
||||
class VT mat;
|
||||
class CAGG cagg;
|
||||
class PARK,GONE parked;
|
||||
class NONE warn;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ASCII
|
||||
|
||||
```
|
||||
╔══════════════════════════════════════════════╗
|
||||
║ TRACKSOLID / JIMI API ║
|
||||
║ (poll + push webhooks, OAuth2) ║
|
||||
╚════════════════════╤═════════════════════════╝
|
||||
│
|
||||
┌──────────────────────────┬───────────┴───────────┬────────────────────────────┐
|
||||
▼ ▼ ▼ ▼
|
||||
┌─────────────────┐ ┌────────────────────┐ ┌──────────────────────┐
|
||||
│ ingest_movement │ │ ingest_events │ │ webhook_receiver │ (all via ts_shared_rev.py:
|
||||
│ _rev.py │ │ _rev.py │ │ _rev.py │ get_conn() pool, api_post(),
|
||||
└────────┬────────┘ └─────────┬──────────┘ └──────────┬───────────┘ clean*() , token cache)
|
||||
│ │ │
|
||||
│ writes │ writes │ writes (push)
|
||||
▼ ▼ ▼
|
||||
╔════════════════════════════════════════════════════════════════════════════════════════════╗
|
||||
║ L1 · BASE TABLES + HYPERTABLES schema: tracksolid (single source of truth) ║
|
||||
║ ║
|
||||
║ live_positions ── current fix / IMEI (plain) api_token_cache (auth) ║
|
||||
║ position_history ◀═══ HYPERTABLE (high-res GPS trail, partitioned by gps_time) ║
|
||||
║ trips ── trip summaries (plain table — NOT a hypertable) ║
|
||||
║ devices ── vehicle/driver registry (plain) ║
|
||||
║ alarms ── alarm events (plain table) ║
|
||||
║ ingestion_log ── API audit trail (plain) ║
|
||||
║ heartbeats*, fuel_readings*, temperature_readings* ◀═ HYPERTABLES (empty / planned push) ║
|
||||
║ device_events, fault_codes, obd_readings, parking_events, lbs_readings, geofences (empty) ║
|
||||
╚══════╤══════════════════════════╤══════════════════════════════════╤════════════════════════╝
|
||||
│ │ │
|
||||
│ TimescaleDB │ matview refresh │ direct reads
|
||||
│ cont-agg policy │ (in-app loop) │ (Grafana SQL)
|
||||
▼ ▼ ▼
|
||||
╔══════════════════╗ ╔════════════════════════════════╗ ╔══════════════════════════════════╗
|
||||
║ L2 · AGGREGATION ║ ║ L2 · AGGREGATION (matview) ║ ║ L3 · GRAFANA VIEWS (tracksolid.*) ║
|
||||
║ ║ ║ ║ ║ read base tables directly ║
|
||||
║ v_mileage_daily ║ ║ reporting.v_trips [MATVIEW] ║ ║ ║
|
||||
║ _cagg ║ ║ ◀── trips + devices ║ ║ v_fleet_today v_fleet_status ║
|
||||
║ CONTINUOUS AGG ║ ║ unique ix_v_trips_trip_id ║ ║ v_active_dispatch_map ║
|
||||
║ ◀═ position_ ║ ║ ║ ║ v_currently_idle v_alarms_daily ║
|
||||
║ history ║ ║ REFRESH … CONCURRENTLY every ║ ║ v_fleet_km_daily v_ingestion_health║
|
||||
║ (auto refresh ║ ║ VTRIPS_REFRESH_INTERVAL_S=300 ║ ║ v_vehicles_not_moved_today ║
|
||||
║ policy) ║ ║ by dashboard_api bg asyncio ║ ║ v_driver_aggregates_daily ║
|
||||
╚════════╤═════════╝ ║ loop, pg advisory-lock 920145 ║ ║ v_fleet_trace ║
|
||||
│ ║ │ ║ ║ v_driver_clock_daily/_today/_attnd ║
|
||||
│ ║ └──▶ reporting.refresh_log ║ ╚══════════════╤═════════════════════╝
|
||||
│ ╚═══════════════╤════════════════╝ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ╔════════════════════════════════════════════╗ │
|
||||
│ ║ L3 · REPORTING VIEWS (← v_trips matview) ║ │
|
||||
│ ║ v_filter_vehicles v_filter_drivers ║ │
|
||||
│ ║ v_filter_cost_centres v_filter_cities ║ │
|
||||
│ ║ v_daily/weekly/monthly_summary ║ │
|
||||
│ ║ v_*_cost_centre v_trips_today ║ │
|
||||
│ ║ v_live_positions (view, ← live_positions) ║ │
|
||||
│ ╚════════════════════════╤═══════════════════╝ │
|
||||
│ │ │
|
||||
│ ╔══════════════════════════════════╧═══════════════════╗ │
|
||||
│ ║ L4 · FUNCTIONS (schema reporting — API entrypoints) ║ │
|
||||
│ ║ ║ │
|
||||
│ ║ fn_live_positions(cost_centre, acc_status) ║ │
|
||||
│ ║ ◀── live_positions + devices ║ │
|
||||
│ ║ fn_vehicle_track(vehicle_number, hours) ║ │
|
||||
│ ║ ◀── position_history (hypertable trail) ────────╫─────┘ (also reads L1)
|
||||
│ ║ fn_trips_for_map(veh[],driver,cc,city,start,end) ║
|
||||
│ ║ ◀── reporting.v_trips (matview) ║
|
||||
│ ║ normalize_plate(p) ── helper used by the above ║
|
||||
│ ╚════════════════════════════╤══════════════════════════╝
|
||||
│ │
|
||||
▼ ▼
|
||||
(cagg currently ╔═══════════════════════════════════════════════╗
|
||||
unconsumed — ║ L5 · CONSUMERS ║
|
||||
no panel/API yet) ║ ║
|
||||
║ dashboard_api_rev.py (FastAPI :8890, ×2 wkrs) ║
|
||||
║ POST /webhook/fleet-dashboard → fn_trips_for_map + v_filter_*
|
||||
║ GET /webhook/live-positions → fn_live_positions
|
||||
║ GET /webhook/live-positions/track → fn_vehicle_track
|
||||
║ │ ║
|
||||
║ ▼ HTTPS (fleetapi.rahamafresh.com) ║
|
||||
║ SPAs: liveposition.* · fleetintelligence.* ║
|
||||
║ (rustfs/S3 single-file maps) ║
|
||||
║ ║
|
||||
║ Grafana ──SQL──▶ tracksolid.v_* (L3 right) ║
|
||||
╚═══════════════════════════════════════════════╝
|
||||
|
||||
|
||||
┌─ REMOVED 2026-06-05 · purged via migrations 12 / 13 (never implemented) ───────────────┐
|
||||
│ ops.* : tickets, dispatch_log, service_log, odometer_readings, │
|
||||
│ cost_rates, kpi_targets, vw_service_forecast │
|
||||
│ dwh_gold.* : dim_vehicles, fact_daily_fleet_metrics, refresh_daily_metrics() │
|
||||
│ tracksolid : v_sla_inflight, v_utilisation_daily (+ their Grafana panels removed) │
|
||||
│ → Schemas dropped from prod. The separate tracksolid_dwh server is unrelated/untouched.│
|
||||
└────────────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How to read it
|
||||
|
||||
- **L1** is where ingestion lands. Only `position_history` (and three empty/planned
|
||||
hypertables) are TimescaleDB hypertables; `trips`/`alarms` are ordinary tables.
|
||||
- **Two aggregation mechanisms** run in parallel: the TimescaleDB **continuous aggregate**
|
||||
`v_mileage_daily_cagg` (refreshed by a Timescale policy off `position_history`), and the
|
||||
**`reporting.v_trips` matview** (refreshed every 5 min by the in-process `dashboard_api`
|
||||
loop — the FIX-D02 self-refresher that replaced the dead n8n job).
|
||||
- **L4 functions are the only thing the API calls.** SPAs never touch tables directly:
|
||||
SPA → `dashboard_api` → `fn_*` → (matview / tables).
|
||||
- **Grafana** takes the right-hand path, reading the `tracksolid.v_*` analytics views
|
||||
straight off L1.
|
||||
|
||||
## Caveats / housekeeping notes
|
||||
|
||||
1. **`v_mileage_daily_cagg` has no consumer** — nothing in the API or Grafana reads it. It
|
||||
is a live continuous aggregate maintaining itself for nobody. Candidate for removal or
|
||||
wiring into a panel.
|
||||
2. **`ops` and `dwh_gold` were purged** (2026-06-05, migrations 12/13) along with
|
||||
`v_utilisation_daily`, `v_sla_inflight`, and their Grafana panels — the features were
|
||||
never implemented.
|
||||
3. See the companion housekeeping audit (2026-06-05) for the full unused-object list; the
|
||||
only clean ad-hoc drop is `public.trips_viz_v1`.
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
# OSM POI Export → FleetNow map layer
|
||||
|
||||
How we turn an OpenStreetMap extract into a toggleable **FleetNow overlay layer**
|
||||
(e.g. the 232 Shell fuel stations). Reproducible end-to-end.
|
||||
|
||||
## 1. Get a Kenya extract
|
||||
|
||||
Download a `.osm.pbf` (Geofabrik / BBBike), e.g. `kenya-260605.osm.pbf`.
|
||||
|
||||
> **Do not commit the `.pbf`** — it's ~331 MB and is gitignored (`*.osm.pbf`).
|
||||
> It's fully reproducible from the download.
|
||||
|
||||
## 2. Export the POIs
|
||||
|
||||
No system tooling required — `pyosmium`'s prebuilt wheel is fetched by `uv`:
|
||||
|
||||
```bash
|
||||
uv run --no-project --with osmium python scripts/export_osm_pois.py \
|
||||
kenya-260605.osm.pbf --amenity fuel --brand Shell \
|
||||
--out-geojson shell_stations.geojson --out-csv shell_stations.csv
|
||||
```
|
||||
|
||||
- Gas stations are OSM **`amenity=fuel`**. Brand is the **`brand`** tag — but only
|
||||
~36% of Kenyan stations carry it (≈92% have a `name`), so when `--brand` is given
|
||||
and a feature has no `brand` tag, the script falls back to matching `name`/`operator`.
|
||||
- Omit `--brand` to export **every** station of that amenity.
|
||||
- Nodes use their own coordinate; ways/areas use their node **centroid**.
|
||||
|
||||
**Reference counts** (extract `kenya-260605`, captured 2026-06-08):
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Total fuel stations (`amenity=fuel`) | 1,794 (1,582 nodes + 212 areas) |
|
||||
| With a `brand` tag | 659 (36%) · with a `name` | 1,652 (92%) |
|
||||
| **Shell** | **232** (209 `brand=Shell` + 23 by name) |
|
||||
| Other top brands | Total 154 · Rubis 147 · TotalEnergies 50 · Kobil 10 |
|
||||
|
||||
## 3. Add it to FleetNow as a layer
|
||||
|
||||
Copy the GeoJSON into the FleetNow repo's `layers/` directory and register it —
|
||||
full details in the **fleetnow** repo `README.md` → *"Map overlay layers"*:
|
||||
|
||||
```bash
|
||||
cp shell_stations.geojson ../fleetnow/layers/
|
||||
# then add one entry to the OVERLAYS array in fleetnow/index.html, commit + push.
|
||||
```
|
||||
|
||||
FleetNow keeps its own served copy under `fleetnow/layers/`; the artifact here
|
||||
(`shell_stations.geojson` / `.csv`) is the export of record. Re-running step 2 on a
|
||||
newer extract and re-copying refreshes the layer.
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -1,74 +0,0 @@
|
|||
# FleetNow — One Fleet, One Screen
|
||||
### Bringing live tracking and trip history together into a single intelligence platform
|
||||
|
||||
*Audience: Senior Directors · Prepared 2026-06-08*
|
||||
|
||||
---
|
||||
|
||||
## The big idea
|
||||
The legacy setup answered two questions in two different places: *"Where is the fleet right now?"* on one dashboard, and *"Where has it been?"* on another. **FleetNow merges them into a single live map** — so the moment you see a vehicle, its full history is one click away. No switching tools, no reconciling two screens, no losing the thread.
|
||||
|
||||
> **"Where is it now"** and **"where has it been"** — finally answered in the same place, at the same time.
|
||||
|
||||
## Why one platform beats two
|
||||
|
||||
| | Legacy: two separate dashboards | **FleetNow: unified platform** |
|
||||
|---|---|---|
|
||||
| **Workflow** | Toggle between live + history tools, rebuild context each time | One screen, one flow — live and history side by side |
|
||||
| **Decisions** | Piece the story together across tabs | Dispatch, investigate, and review in a single view |
|
||||
| **Filters** | Set them twice, hope they match | One filter (city, cost centre, vehicle) applies everywhere |
|
||||
| **Data trust** | Two sources that can quietly disagree | One source of truth — live and historical always agree |
|
||||
| **Onboarding** | Learn two tools | Learn one — faster for new staff |
|
||||
| **Upkeep** | Two apps to maintain | One platform, lower cost and risk |
|
||||
|
||||
**The headline:** merging live + historical isn't just tidier — it's *faster decisions, fewer errors, and a single trusted picture of the fleet.*
|
||||
|
||||
## What's new and powerful
|
||||
|
||||
**🗺️ A living operations map**
|
||||
- Real-time vehicle positions refreshed continuously, with movement, idle, and offline states colour-coded at a glance.
|
||||
- Click any vehicle to drop straight into its trip history — routes, stops, distances, timing.
|
||||
|
||||
**🎨 Instant visual read of the fleet**
|
||||
- **Colour-coordinated by department** (ISP, OSP, FDS…) — spot how each team is deployed across the city in a single glance, with a one-tap colour key.
|
||||
- **Distinct icons for specialist assets** — cranes, motorbikes, and pick-ups stand out from the field-service fleet, so the vehicles that close customer tickets are never confused with heavy/infrastructure units.
|
||||
|
||||
**🎯 Built for *your* operation, not a generic tracker**
|
||||
- The map shows the **operational fleet only** — personal, management, and out-of-region vehicles are filtered out, so leaders see what actually matters to service delivery.
|
||||
- Tracker + dashcam are intelligently paired into **one vehicle**, so counts and lists reflect reality (not double-counted devices).
|
||||
|
||||
**📊 One source of truth, automatically current**
|
||||
- Vehicle types, drivers, and assignments flow straight from the source system — the platform stays accurate without manual upkeep.
|
||||
- The same trusted data powers the live map, trip history, and management analytics.
|
||||
|
||||
## What this means for the business
|
||||
- **Faster, better decisions** — dispatch and respond from a single, current view.
|
||||
- **Real accountability** — every trip tied to a vehicle, driver, and department.
|
||||
- **Clarity for leadership** — see fleet activity by team and city at a glance.
|
||||
- **Lower cost & risk** — one platform to run and train on, not several.
|
||||
- **A foundation to build on** — utilisation, SLA, and cost insights plug into the same trusted base.
|
||||
|
||||
## Where we're headed
|
||||
FleetNow is a living platform. On the near-term roadmap: always-on visibility of specialist assets, richer per-department analytics, and deeper KPI dashboards for service performance — all on the same single, trusted foundation.
|
||||
|
||||
> **From two disconnected dashboards to one intelligent platform — that's the leap.**
|
||||
|
||||
---
|
||||
|
||||
## Demo flow & talking points
|
||||
|
||||
**The one-line message:** *"We merged live tracking and trip history into one intelligent platform — one screen, one source of truth, purpose-built for our field-service fleet."*
|
||||
|
||||
**Live demo (≈3–4 minutes):**
|
||||
1. **Open the live map** — "This is the whole operational fleet, right now, across our cities."
|
||||
2. **Point out the colours** — open the **Key**: "Each department has its own colour — here's ISP, OSP, FDS at a glance."
|
||||
3. **Read the states** — "Bright ones are moving now, faded are recently stopped, grey are offline — instant situational awareness."
|
||||
4. **Spot a specialist** — "These icons are our cranes and motorbikes — clearly separate from the ticket-closing fleet."
|
||||
5. **Click a vehicle → its trips** — "And here's the power of merging: from a live pin, straight into where this vehicle has been today, same screen."
|
||||
6. **Apply a filter** — "Filter by department or city once, and it holds across both live and history."
|
||||
7. **Land the close** — "One screen. One trusted source. No more switching tools."
|
||||
|
||||
**Three points to repeat:**
|
||||
- **One platform, not two** — faster decisions, less friction.
|
||||
- **One source of truth** — live and historical can't disagree.
|
||||
- **Built for us** — segmented fleet, department colours, real accountability.
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,69 +0,0 @@
|
|||
# FleetNow — Your Team, Live and in Full
|
||||
### One screen to see your crews, dispatch in the moment, and review after
|
||||
|
||||
*Audience: Heads of Department · Prepared 2026-06-08*
|
||||
|
||||
---
|
||||
|
||||
## The big idea
|
||||
You shouldn't have to jump between a "where are they now" screen and a separate "where did they go" screen to run your day. **FleetNow puts both on one map** — see your vehicles live, then click straight into any vehicle's trips without losing your place. Act in the moment; review when you need to.
|
||||
|
||||
> See your team **right now**, and everywhere they've **been today** — without ever leaving the map.
|
||||
|
||||
## Why one screen beats two (for your day)
|
||||
|
||||
| | Legacy: two dashboards | **FleetNow: one screen** |
|
||||
|---|---|---|
|
||||
| **Running your shift** | Flip between live + history tools | Everything in one place — dispatch and check history together |
|
||||
| **Finding your team** | Hunt through a full fleet list | Your department has its **own colour** — spot your crews instantly |
|
||||
| **Settling a question** | Cross-check two tools (and a spreadsheet) | Click the vehicle → see its trips on the spot |
|
||||
| **Trusting the numbers** | Two sources that can disagree | One source — live and history always match |
|
||||
|
||||
## What's new — and how it helps your team
|
||||
|
||||
**🎨 Your department, in its own colour**
|
||||
Every team has a dedicated colour on the map (ISP, OSP, FDS…). At a glance you can see *your* crews, where they are, and how they're spread across the city — with a one-tap colour key so nobody has to memorise anything.
|
||||
|
||||
**🟢 Read the situation instantly**
|
||||
Colour intensity tells the story: **bright = moving now**, softer = stopped recently, grey = offline 24h+. You can see who's active, who's idle, and who's gone quiet — without clicking a thing.
|
||||
|
||||
**🎯 Know your ticket-closers from your heavy units**
|
||||
Cranes, motorbikes, and pick-ups now carry their **own icons**, clearly set apart from the field-service vehicles that close customer tickets — so the fleet that drives your SLAs is never confused with specialist assets.
|
||||
|
||||
**👤 Every trip tied to a driver**
|
||||
Live position *and* trip history both link to the vehicle, plate, and driver — so performance conversations, job verification, and dispute resolution are backed by data, not memory.
|
||||
|
||||
**🧹 A clean, honest view**
|
||||
Personal, management, and out-of-region vehicles are filtered out, and each tracker+dashcam pair shows as **one vehicle** — so your map reflects the real operational fleet, nothing more.
|
||||
|
||||
## What this means for your department
|
||||
- **Dispatch faster** — find the nearest available crew in seconds.
|
||||
- **Chase the gaps** — spot idle or silent vehicles before they cost you a missed SLA.
|
||||
- **Back your team with facts** — utilisation, routes, and timing on demand.
|
||||
- **Less admin** — one tool, no reconciling screens or spreadsheets.
|
||||
- **Make your numbers look real** — accurate counts, clear accountability.
|
||||
|
||||
## Where it's headed
|
||||
More to come on the same trusted foundation: always-on visibility for specialist assets, per-department performance dashboards, and SLA/utilisation insights — built around how your teams actually work.
|
||||
|
||||
> **From two disconnected dashboards to one screen that runs your day.**
|
||||
|
||||
---
|
||||
|
||||
## Demo flow & talking points
|
||||
|
||||
**The one-line message:** *"We merged live tracking and trip history into one intelligent platform — one screen, one source of truth, purpose-built for our field-service fleet."*
|
||||
|
||||
**Live demo (≈3–4 minutes):**
|
||||
1. **Open the live map** — "This is the whole operational fleet, right now, across our cities."
|
||||
2. **Point out the colours** — open the **Key**: "Each department has its own colour — here's ISP, OSP, FDS at a glance."
|
||||
3. **Read the states** — "Bright ones are moving now, faded are recently stopped, grey are offline — instant situational awareness."
|
||||
4. **Spot a specialist** — "These icons are our cranes and motorbikes — clearly separate from the ticket-closing fleet."
|
||||
5. **Click a vehicle → its trips** — "And here's the power of merging: from a live pin, straight into where this vehicle has been today, same screen."
|
||||
6. **Apply a filter** — "Filter by department or city once, and it holds across both live and history."
|
||||
7. **Land the close** — "One screen. One trusted source. No more switching tools."
|
||||
|
||||
**Three points to repeat:**
|
||||
- **One platform, not two** — faster decisions, less friction.
|
||||
- **One source of truth** — live and historical can't disagree.
|
||||
- **Built for us** — segmented fleet, department colours, real accountability.
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,374 +0,0 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Tracksolid Stack — Engineering Review & Improvement Plan (2026-06-01)</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #0f1115;
|
||||
--panel: #171a21;
|
||||
--panel-2: #1d212b;
|
||||
--ink: #e6e9ef;
|
||||
--ink-dim: #aab2c0;
|
||||
--line: #2a2f3a;
|
||||
--accent: #5b9dff;
|
||||
--hi: #ff5d5d;
|
||||
--med: #ffb454;
|
||||
--lo: #5fd0a0;
|
||||
--good: #5fd0a0;
|
||||
--mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
||||
--sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html { scroll-behavior: smooth; }
|
||||
body {
|
||||
margin: 0; background: var(--bg); color: var(--ink);
|
||||
font-family: var(--sans); line-height: 1.6; font-size: 16px;
|
||||
}
|
||||
.wrap { max-width: 980px; margin: 0 auto; padding: 48px 28px 96px; }
|
||||
header.doc {
|
||||
border-bottom: 1px solid var(--line); padding-bottom: 28px; margin-bottom: 36px;
|
||||
}
|
||||
.kicker { color: var(--accent); font-family: var(--mono); font-size: 13px; letter-spacing: .12em; text-transform: uppercase; }
|
||||
h1 { font-size: 30px; line-height: 1.25; margin: 10px 0 14px; }
|
||||
.meta { color: var(--ink-dim); font-size: 14px; }
|
||||
.meta b { color: var(--ink); font-weight: 600; }
|
||||
h2 { font-size: 22px; margin: 44px 0 8px; padding-top: 10px; border-top: 1px solid var(--line); }
|
||||
h2 .num { color: var(--accent); font-family: var(--mono); margin-right: 10px; }
|
||||
h3 { font-size: 17px; margin: 26px 0 6px; }
|
||||
p { margin: 10px 0; }
|
||||
a { color: var(--accent); }
|
||||
code { font-family: var(--mono); font-size: .87em; background: var(--panel-2); padding: 1px 6px; border-radius: 4px; color: #d7e3ff; }
|
||||
pre { background: #0b0d12; border: 1px solid var(--line); border-radius: 8px; padding: 14px 16px; overflow-x: auto; font-family: var(--mono); font-size: 13px; color: #cdd6e4; }
|
||||
ul, ol { margin: 10px 0 10px 4px; padding-left: 22px; }
|
||||
li { margin: 5px 0; }
|
||||
.lead { color: var(--ink-dim); font-size: 16px; }
|
||||
|
||||
.callout { border-left: 3px solid var(--accent); background: var(--panel); border-radius: 0 8px 8px 0; padding: 14px 18px; margin: 18px 0; }
|
||||
.callout.warn { border-left-color: var(--hi); }
|
||||
.callout.warn b { color: var(--hi); }
|
||||
|
||||
.finding { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; padding: 4px 22px 18px; margin: 22px 0; }
|
||||
.badge { display: inline-block; font-family: var(--mono); font-size: 11px; font-weight: 700; letter-spacing: .06em; padding: 3px 9px; border-radius: 999px; text-transform: uppercase; vertical-align: middle; margin-left: 8px; }
|
||||
.b-hi { background: rgba(255,93,93,.15); color: var(--hi); border: 1px solid rgba(255,93,93,.35); }
|
||||
.b-med { background: rgba(255,180,84,.13); color: var(--med); border: 1px solid rgba(255,180,84,.32); }
|
||||
.b-lo { background: rgba(95,208,160,.12); color: var(--lo); border: 1px solid rgba(95,208,160,.3); }
|
||||
.b-sec { background: rgba(91,157,255,.13); color: var(--accent); border: 1px solid rgba(91,157,255,.32); }
|
||||
|
||||
.ref { font-family: var(--mono); font-size: 12.5px; color: var(--ink-dim); }
|
||||
|
||||
table { width: 100%; border-collapse: collapse; margin: 18px 0; font-size: 14.5px; }
|
||||
th, td { text-align: left; padding: 10px 12px; border-bottom: 1px solid var(--line); vertical-align: top; }
|
||||
th { color: var(--ink-dim); font-weight: 600; font-size: 12.5px; text-transform: uppercase; letter-spacing: .05em; }
|
||||
td.up-h { color: var(--hi); font-weight: 600; }
|
||||
td.up-m { color: var(--med); font-weight: 600; }
|
||||
td.up-l { color: var(--lo); font-weight: 600; }
|
||||
|
||||
.pill { font-family: var(--mono); font-size: 11px; padding: 2px 7px; border-radius: 4px; background: var(--panel-2); color: var(--ink-dim); }
|
||||
|
||||
.good-box { border: 1px solid rgba(95,208,160,.3); background: rgba(95,208,160,.05); border-radius: 10px; padding: 6px 22px 16px; margin: 22px 0; }
|
||||
.good-box h2 { border-top: none; color: var(--good); }
|
||||
|
||||
footer { margin-top: 60px; padding-top: 22px; border-top: 1px solid var(--line); color: var(--ink-dim); font-size: 13px; }
|
||||
.toc { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; padding: 16px 22px; margin: 8px 0 0; }
|
||||
.toc ol { margin: 6px 0; }
|
||||
.toc a { text-decoration: none; }
|
||||
.toc a:hover { text-decoration: underline; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
|
||||
<header class="doc">
|
||||
<div class="kicker">Engineering Review · Fireside Communications · Tracksolid Fleet Stack</div>
|
||||
<h1>Database & Microservice Assessment — Opportunities & Refactoring</h1>
|
||||
<p class="meta">
|
||||
<b>Date:</b> 2026-06-01 ·
|
||||
<b>Reviewer:</b> Claude (Opus 4.8) ·
|
||||
<b>Scope:</b> TimescaleDB/PostGIS schema + migrations, and the three ingestion microservices
|
||||
(<code>ingest_movement_rev.py</code>, <code>ingest_events_rev.py</code>, <code>webhook_receiver_rev.py</code> + shared <code>ts_shared_rev.py</code>)
|
||||
</p>
|
||||
<p class="meta">Findings are ordered by <b>greatest performance upside first</b>, as requested.</p>
|
||||
</header>
|
||||
|
||||
<div class="callout warn">
|
||||
<p><b>Access caveat — read this first.</b> The remote instance was <b>unreachable from the review environment</b>:
|
||||
every probed port (22, 5433, 5432, 443) timed out, so the IP is not whitelisted (or the host was down).
|
||||
I could <b>not</b> run <code>EXPLAIN</code>, read live row/chunk counts, confirm which indexes actually exist,
|
||||
or inspect the running images. Everything below is a <b>static review</b> of the source and migration files.
|
||||
Items needing live confirmation are tagged <span class="pill">verify live</span>.</p>
|
||||
<p style="margin-bottom:0"><b>Immediate security note:</b> <code>.env</code> is <b>committed to git</b> (it is listed in
|
||||
<code>.gitignore</code>, but was tracked before that rule existed, so the rule is a no-op). The live Tracksolid app
|
||||
secret, the Postgres superuser password, and the Grafana admin password are all in the repo history on Forgejo.
|
||||
Treat all three as compromised and rotate them.</p>
|
||||
</div>
|
||||
|
||||
<div class="toc">
|
||||
<strong>Findings</strong>
|
||||
<ol>
|
||||
<li><a href="#f1">Single-threaded scheduler holds a DB transaction open across throttled geocoding</a> <span class="badge b-hi">High</span></li>
|
||||
<li><a href="#f2">dwh_gold daily-metrics ETL is non-functional</a> <span class="badge b-hi">High</span></li>
|
||||
<li><a href="#f3">v_driver_aggregates_daily will not scale; safeguard not applied</a> <span class="badge b-hi">High</span></li>
|
||||
<li><a href="#f4">pgbouncer deployed but bypassed by the application</a> <span class="badge b-med">Medium</span></li>
|
||||
<li><a href="#f5">Migrations race across three containers with no lock</a> <span class="badge b-med">Medium</span></li>
|
||||
<li><a href="#f6">Orphaned migration: 10_driver_clock_views.sql never applied</a> <span class="badge b-med">Medium</span></li>
|
||||
<li><a href="#f7">Security gaps (webhook auth, committed secrets)</a> <span class="badge b-sec">Security</span></li>
|
||||
<li><a href="#f8">Smaller DB-design notes</a> <span class="badge b-lo">Low</span></li>
|
||||
<li><a href="#good">What's genuinely good</a></li>
|
||||
<li><a href="#plan">Suggested order of attack</a></li>
|
||||
</ol>
|
||||
</div>
|
||||
|
||||
<!-- ====================== FINDING 1 ====================== -->
|
||||
<h2 id="f1"><span class="num">1</span>Single-threaded scheduler holds a DB transaction open across throttled geocoding<span class="badge b-hi">Highest upside</span></h2>
|
||||
<div class="finding">
|
||||
<p><code>ingest_movement_rev.py</code> runs every job on one <code>schedule</code> thread
|
||||
(<span class="ref">main(), lines 674–695</span>). Within that, <code>poll_trips()</code> opens a transaction
|
||||
(<span class="ref">with get_conn(), line 343</span>) and then, <b>inside that open transaction</b>, calls
|
||||
<code>reverse_geocode()</code> twice per trip (<span class="ref">lines 392–393</span>).
|
||||
<code>reverse_geocode</code> enforces a global <b>1 request/second</b> Nominatim throttle
|
||||
(<span class="ref">ts_shared_rev.py:463, _geocode_throttle</span>).</p>
|
||||
|
||||
<h3>Consequences</h3>
|
||||
<ul>
|
||||
<li>A batch of N new trips can hold a single pooled connection open for <b>N×~2 seconds</b> of network I/O — a
|
||||
long-running transaction that pins a snapshot (bad for autovacuum's cleanup horizon) and ties up a connection.</li>
|
||||
<li>Because the scheduler is one thread, while <code>poll_trips</code> is geocoding, the <b>60-second live-position
|
||||
sweep cannot run</b>. The "live" freshness SLA silently degrades to minutes whenever trips/parking work through a
|
||||
backlog. <code>poll_track_list</code> (30 min) and <code>poll_stale_locations</code> (10 min) share the same
|
||||
thread and also block each other.</li>
|
||||
<li>Every 15 min, <code>poll_trips</code> re-runs the 8-subquery enrichment block (<code>_ENRICH_QUERY</code>,
|
||||
<span class="ref">lines 295–321</span>) for the <b>entire last hour</b> of trips, even though the
|
||||
<code>ON CONFLICT</code> mostly <code>COALESCE</code>s the result away.</li>
|
||||
</ul>
|
||||
|
||||
<h3>Recommendation</h3>
|
||||
<ul>
|
||||
<li>Move geocoding <b>out of the DB transaction</b>: collect trip rows, commit, then geocode + <code>UPDATE</code>
|
||||
in a second pass (or delegate geocoding to a queue / n8n).</li>
|
||||
<li>Gate enrichment on <code>WHERE start_address IS NULL</code> so already-enriched trips don't re-pay the cost.</li>
|
||||
<li>Run the 60s live sweep on its own thread/process so slow reporting jobs cannot starve it.
|
||||
<code>schedule</code> + <code>time.sleep(1)</code> on one thread is the wrong concurrency model when one job is
|
||||
latency-critical and others do long network I/O.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<!-- ====================== FINDING 2 ====================== -->
|
||||
<h2 id="f2"><span class="num">2</span>The <code>dwh_gold</code> daily-metrics ETL is non-functional<span class="badge b-hi">High</span></h2>
|
||||
<div class="finding">
|
||||
<p><code>dwh_gold.refresh_daily_metrics()</code> (<span class="ref">migration 05, lines 212–264</span>) selects
|
||||
<code>t.imei AS vehicle_key</code> and inserts into <code>fact_daily_fleet_metrics.vehicle_key</code>, which is
|
||||
<code>INTEGER REFERENCES dwh_gold.dim_vehicles(vehicle_key)</code> (<span class="ref">schema lines 237–243</span>).
|
||||
But <code>imei</code> is a 12–15-digit <b>TEXT</b> string:</p>
|
||||
<ul>
|
||||
<li>15-digit IMEIs overflow <code>int4</code> → <em>"integer out of range"</em>.</li>
|
||||
<li>Shorter ones violate the FK because <b>nothing ever populates <code>dim_vehicles</code></b> — no code path
|
||||
inserts into it.</li>
|
||||
</ul>
|
||||
<p>So the function cannot succeed as written, and <code>v_utilisation_daily</code> (which joins
|
||||
<code>fact → dim_vehicles → devices</code>, <span class="ref">migration 07, lines 268–286</span>) will always be
|
||||
empty. CLAUDE.md lists "schedule the nightly ETL" as a LOW open item — but scheduling it today would error on every
|
||||
run.</p>
|
||||
<p style="margin-bottom:0"><b>Recommendation:</b> redesign the gold layer around <code>imei</code> (drop the surrogate
|
||||
key, or populate <code>dim_vehicles</code> from <code>devices</code> first and look up the key), and fix the column
|
||||
type. This is a real bug hiding behind "not scheduled yet."</p>
|
||||
</div>
|
||||
|
||||
<!-- ====================== FINDING 3 ====================== -->
|
||||
<h2 id="f3"><span class="num">3</span><code>v_driver_aggregates_daily</code> will not scale, and the safeguard wasn't applied<span class="badge b-hi">High</span></h2>
|
||||
<div class="finding">
|
||||
<p>Migration 07 (<span class="ref">lines 159–223</span>) builds this view with two 31-day scans of
|
||||
<code>position_history</code> plus a <code>LAG()</code> window over <code>source='track_list'</code> rows. There is
|
||||
<b>no index on <code>position_history.source</code></b>, and the only index on the hypertable is the
|
||||
<code>(imei, gps_time)</code> primary key.</p>
|
||||
<p>The view's own header comment says <em>"convert to a continuous aggregate once the hypertable exceeds ~100k rows."</em>
|
||||
At 156 devices writing a row/minute from the poll sweep plus track_list waypoints, you cross 100k in <b>days</b>, not
|
||||
months. <span class="pill">verify live</span> current row + chunk count.</p>
|
||||
<p style="margin-bottom:0"><b>Recommendation:</b> build the speeding/harsh aggregates as a TimescaleDB continuous
|
||||
aggregate (the pattern already exists in <code>v_mileage_daily_cagg</code>), or at minimum add a partial index
|
||||
supporting the <code>source='track_list'</code> + time filter. As-is, the daily driver dashboard does a growing full
|
||||
hypertable scan on every load.</p>
|
||||
</div>
|
||||
|
||||
<!-- ====================== FINDING 4 ====================== -->
|
||||
<h2 id="f4"><span class="num">4</span>pgbouncer is deployed but the application bypasses it entirely<span class="badge b-med">Medium</span></h2>
|
||||
<div class="finding">
|
||||
<p><code>docker-compose.yaml</code> adds a pgbouncer sidecar (<span class="ref">lines 82–116</span>) "to cap
|
||||
tracksolid_db connections," but <code>.env</code> sets
|
||||
<code>DATABASE_URL=...@timescale_db:5432/...</code> — the Python pools connect <b>straight to Postgres</b>, not to
|
||||
pgbouncer's 6432.</p>
|
||||
<p>So the connection cap does nothing for the three services. The real ceiling today is the sum of per-process pools:</p>
|
||||
<pre>webhook : uvicorn --workers 2 → 2 procs × ThreadedConnectionPool(max=12) = 24
|
||||
ingest_movement = 12
|
||||
ingest_events = 12
|
||||
total ≈ 48 direct conns</pre>
|
||||
<p>At 80–156 devices this is not a live performance problem — it is wasted/contradictory infrastructure and an
|
||||
intent-vs-reality gap. You also maintain a SCRAM-passthrough <code>user_lookup()</code> SECURITY DEFINER function
|
||||
(<span class="ref">migration 10</span>) with no consumer.</p>
|
||||
<p style="margin-bottom:0"><b>Recommendation:</b> either point <code>DATABASE_URL</code> at <code>pgbouncer:6432</code>
|
||||
(transaction-pool mode disallows session features, but the code uses none beyond <code>client_encoding</code>), or
|
||||
remove the sidecar.</p>
|
||||
</div>
|
||||
|
||||
<!-- ====================== FINDING 5 ====================== -->
|
||||
<h2 id="f5"><span class="num">5</span>Migrations race across three containers with no lock<span class="badge b-med">Medium · reliability</span></h2>
|
||||
<div class="finding">
|
||||
<p>All three services run <code>python run_migrations.py</code> on startup (<span class="ref">compose lines 26, 37,
|
||||
48</span>) and start in parallel once the DB is healthy. <code>run_migrations.py</code> does check-then-act
|
||||
(<code>already_applied()</code> → <code>run_file()</code>, <span class="ref">lines 231–242</span>) with <b>no
|
||||
advisory lock</b>. On a fresh database, three containers can pass <code>already_applied()==False</code>
|
||||
simultaneously and run the same file.</p>
|
||||
<ul>
|
||||
<li>Migration 02's <code>CREATE TRIGGER</code> loop (<span class="ref">lines 255–267</span>) has no
|
||||
<code>IF NOT EXISTS</code> — concurrent runs throw, and <code>run_file()</code> treats any <code>ERROR:</code> as
|
||||
fatal → <code>sys.exit(1)</code> → a service refuses to start.</li>
|
||||
<li><code>run_file()</code> greps stderr for <code>ERROR:</code> without <code>-v ON_ERROR_STOP=1</code>, and files
|
||||
02/03 have no <code>BEGIN/COMMIT</code>, so a mid-file failure can leave partial schema that later gets mis-seeded
|
||||
as "applied."</li>
|
||||
</ul>
|
||||
<p style="margin-bottom:0"><b>Recommendation:</b> wrap the run in <code>pg_advisory_lock(<const>)</code> /
|
||||
unlock, and run psql with <code>ON_ERROR_STOP=1</code>. Low effort, removes a class of cold-start flakiness.</p>
|
||||
</div>
|
||||
|
||||
<!-- ====================== FINDING 6 ====================== -->
|
||||
<h2 id="f6"><span class="num">6</span>Orphaned migration: <code>10_driver_clock_views.sql</code> is never applied<span class="badge b-med">Medium</span></h2>
|
||||
<div class="finding">
|
||||
<p>The runner's <code>MIGRATIONS</code> list (<span class="ref">run_migrations.py:27–37</span>) includes
|
||||
<code>10_pgbouncer_auth.sql</code> but <b>not</b> <code>10_driver_clock_views.sql</code>. Two files share the
|
||||
<code>10_</code> prefix and the list is hand-maintained, so <code>v_driver_clock_daily/_today</code> (which the n8n
|
||||
tardiness workflow depends on, per the file header) exist only if someone applied them by hand — they are not
|
||||
reproducible from a clean deploy.</p>
|
||||
<p style="margin-bottom:0"><b>Recommendation:</b> rename to <code>11_</code> and add to the list. Better: switch the
|
||||
runner from a hardcoded list to globbing <code>NN_*.sql</code> sorted, so this cannot recur.</p>
|
||||
</div>
|
||||
|
||||
<!-- ====================== FINDING 7 ====================== -->
|
||||
<h2 id="f7"><span class="num">7</span>Security gaps worth fixing now<span class="badge b-sec">Security</span></h2>
|
||||
<div class="finding">
|
||||
<ul>
|
||||
<li><b>Webhook auth is effectively off.</b> <code>_validate_token</code>
|
||||
(<span class="ref">webhook_receiver_rev.py:84–87</span>) skips validation entirely when
|
||||
<code>JIMI_WEBHOOK_TOKEN</code> is empty, and it is <b>not set in <code>.env</code></b>. The push endpoints are
|
||||
exposed via Traefik, so anyone who learns the URL can inject arbitrary telemetry/alarms (each <code>/pushgps</code>
|
||||
accepts up to 5000 rows, no rate limit). Set the token and make an unset token <b>fail closed</b> in production.</li>
|
||||
<li><b>Committed secrets</b> (see top banner). Rotate the Tracksolid app secret, Postgres password, and Grafana admin
|
||||
password; <code>git rm --cached .env</code> and scrub history.</li>
|
||||
<li><code>dwh/260423_dwh_ddl_v1.sql</code> plaintext passwords are an existing known item in CLAUDE.md — same class of
|
||||
problem.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<!-- ====================== FINDING 8 ====================== -->
|
||||
<h2 id="f8"><span class="num">8</span>Smaller DB-design notes<span class="badge b-lo">Low — queue these</span></h2>
|
||||
<div class="finding">
|
||||
<ul>
|
||||
<li><b><code>v_mileage_daily_cagg</code> is built on a column that's mostly NULL.</b> It computes
|
||||
<code>MAX(current_mileage) - MIN(current_mileage)</code> (<span class="ref">schema lines 293–301</span>), but
|
||||
<code>current_mileage</code> is only populated by the poll sweep — <code>track_list</code> and <code>/pushgps</code>
|
||||
inserts leave it NULL, and odometer resets/device swaps produce negative or huge deltas. The aggregate's
|
||||
<code>dist_km</code> is unreliable. Prefer deriving daily distance from <code>trips.distance_km</code>.</li>
|
||||
<li><b><code>ingestion_log</code> has no retention and no index.</b> <code>v_ingestion_health</code> does
|
||||
<code>DISTINCT ON (endpoint) … ORDER BY endpoint, run_at DESC</code> over the whole table, which grows ~875
|
||||
rows/day forever. Add <code>(endpoint, run_at DESC)</code> plus a retention/partition policy.</li>
|
||||
<li><b>Alarm dedup is leaky on the poll path.</b> <code>alarms_dedup UNIQUE (imei, alarm_type, alarm_time)</code>
|
||||
(<span class="ref">schema line 199</span>) — the poll path inserts <code>alertTypeId</code> as
|
||||
<code>alarm_type</code> with no NOT-NULL guard, and <code>NULL</code> defeats the unique constraint
|
||||
(<code>NULL ≠ NULL</code>), so a null-type alarm can duplicate. The webhook path guards this; the poll path
|
||||
doesn't.</li>
|
||||
<li><b><code>live_positions</code>/staleness queries are seq scans</b> (no index on <code>gps_time</code>) — totally
|
||||
fine at ~156 rows today; just don't carry that pattern into anything that scans <code>position_history</code>.</li>
|
||||
<li><b>Dead/ambiguous code in <code>_parse_request</code></b> (<span class="ref">webhook lines 90–143</span>): the
|
||||
JSON-array branch <code>_parse_data_list</code> is never reached (it always falls through to
|
||||
<code>request.form()</code>); harmless but misleading given the docstring claims it handles both.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<!-- ====================== GOOD ====================== -->
|
||||
<div class="good-box">
|
||||
<h2 id="good" style="border-top:none;"><span class="num" style="color:var(--good)">✓</span>What's genuinely good</h2>
|
||||
<p>So this is balanced — the bones are solid:</p>
|
||||
<ul>
|
||||
<li>Per-row <code>SAVEPOINT</code> isolation so one bad item can't abort a batch.</li>
|
||||
<li>Time-guarded upserts via the shared <code>upsert_live_position</code> helper.</li>
|
||||
<li>Batched <code>execute_values</code> on the high-volume push / track-list paths.</li>
|
||||
<li>Hypertables with compression + retention policies.</li>
|
||||
<li>Parameterized SQL throughout — no injection surface.</li>
|
||||
<li>Clean signal handling and pool teardown.</li>
|
||||
<li>Idempotent migrations with a tracking table and <code>COMMENT ON VIEW</code> provenance.</li>
|
||||
<li><code>sync_devices</code> N+1 already parallelized with a bounded thread pool.</li>
|
||||
</ul>
|
||||
<p style="margin-bottom:0">The issues above are mostly about <b>coupling</b>, <b>one broken ETL</b>, and
|
||||
<b>scale-ahead-of-indexing</b> — not a bad foundation.</p>
|
||||
</div>
|
||||
|
||||
<!-- ====================== PLAN ====================== -->
|
||||
<h2 id="plan"><span class="num">»</span>Suggested order of attack (effort vs. upside)</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th style="width:38px">#</th><th>Action</th><th style="width:130px">Upside</th><th style="width:80px">Effort</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>1</td>
|
||||
<td>Pull geocoding out of the trips transaction + gate on <code>start_address IS NULL</code>; isolate the 60s sweep on its own thread</td>
|
||||
<td class="up-h">High — restores live freshness, frees connections</td>
|
||||
<td>M</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>2</td>
|
||||
<td>Fix or redesign <code>refresh_daily_metrics</code> / <code>dim_vehicles</code> (imei vs int key)</td>
|
||||
<td class="up-h">High — unblocks all utilisation reporting</td>
|
||||
<td>M</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>3</td>
|
||||
<td>Convert <code>v_driver_aggregates_daily</code> to a continuous aggregate (or add <code>source</code>+time index)</td>
|
||||
<td class="up-h">High and growing</td>
|
||||
<td>M</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>4</td>
|
||||
<td>Set <code>JIMI_WEBHOOK_TOKEN</code>; rotate + untrack <code>.env</code></td>
|
||||
<td class="up-h">High (security)</td>
|
||||
<td>S</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>5</td>
|
||||
<td>Advisory-lock the migration runner + <code>ON_ERROR_STOP=1</code>; add <code>10_driver_clock_views</code> / switch to glob</td>
|
||||
<td class="up-m">Medium (reliability)</td>
|
||||
<td>S</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>6</td>
|
||||
<td>Decide pgbouncer in-or-out; point <code>DATABASE_URL</code> accordingly</td>
|
||||
<td class="up-m">Medium (clarity)</td>
|
||||
<td>S</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>7</td>
|
||||
<td><code>ingestion_log</code> index + retention; fix poll-path alarm null dedup; fix cagg distance source</td>
|
||||
<td class="up-l">Low–medium</td>
|
||||
<td>S</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<div class="callout">
|
||||
<p style="margin:0"><b>Next step for live confirmation:</b> if I can get onto the box (whitelist the review IP for
|
||||
5433, or an SSH tunnel), I'll confirm the <span class="pill">verify live</span> items — actual
|
||||
<code>position_history</code> row/chunk counts, which indexes really exist, whether <code>refresh_daily_metrics</code>
|
||||
has ever succeeded, and <code>EXPLAIN ANALYZE</code> on the heavier views — and tighten the priority order with real
|
||||
numbers.</p>
|
||||
</div>
|
||||
|
||||
<footer>
|
||||
Generated 2026-06-01 by Claude (Opus 4.8) for Fireside Communications · Tracksolid Fleet Intelligence.
|
||||
Static review only — no live database access was available at review time. File references use
|
||||
<span class="ref">file:line</span> against the repository state on branch
|
||||
<code>quality-program-2026-04-12</code>.
|
||||
</footer>
|
||||
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
# ops schema purge — pre-drop backup (2026-06-05)
|
||||
|
||||
Snapshot taken immediately before dropping the `ops` schema + `tracksolid.dispatch_log`
|
||||
+ the dependent view `tracksolid.v_sla_inflight`. The dispatch/SLA/workshop features were
|
||||
never implemented and are being purged (may take a different direction later).
|
||||
|
||||
Only two `ops` tables held rows, and both are **migration-08 seed data**
|
||||
(`migrations/08_analytics_config.sql`), i.e. regenerable — this file is belt-and-suspenders.
|
||||
Every other ops table (`tickets`, `service_log`, `odometer_readings`) and the view
|
||||
`vw_service_forecast` were empty. `tracksolid.dispatch_log` was empty.
|
||||
|
||||
## ops.cost_rates (3 rows)
|
||||
|
||||
| rate_key | scope_type | scope_value | metric | amount | currency | effective_from | notes |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| fuel.nairobi | city | nairobi | fuel_per_litre | 195.00 | KES | 2026-04-27 | Placeholder pump price — confirm with Finance. |
|
||||
| fuel.mombasa | city | mombasa | fuel_per_litre | 195.00 | KES | 2026-04-27 | Placeholder pump price — confirm with Finance. |
|
||||
| fuel.kampala | city | kampala | fuel_per_litre | 5200.00 | UGX | 2026-04-27 | Placeholder pump price — confirm with Finance. |
|
||||
|
||||
## ops.kpi_targets (12 rows)
|
||||
|
||||
| target_id | kpi_key | scope_type | scope_value | target | amber | red | direction | effective_from | notes |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| 1 | utilisation_pct | global | | 70.00 | 60.00 | 50.00 | higher_is_better | 2026-04-27 | Fleet utilisation: drive_hours / engine_on_hours. |
|
||||
| 2 | idle_pct | global | | 15.00 | 20.00 | 25.00 | lower_is_better | 2026-04-27 | Idle as % of engine-on time. |
|
||||
| 3 | idle_pct | cost_centre | osp patrol | 15.00 | 20.00 | 25.00 | lower_is_better | 2026-04-27 | OSP patrol idle target — same as global until calibrated. |
|
||||
| 4 | fuel_kes_per_100km | global | | 12.00 | 14.00 | 16.00 | lower_is_better | 2026-04-27 | Fuel litres per 100km equivalent — uses fuel_100km on devices. |
|
||||
| 5 | mttr_hours | global | | 4.00 | 6.00 | 8.00 | lower_is_better | 2026-04-27 | Mean Time To Resolve, field-service ticket. |
|
||||
| 6 | alarms_per_100km | global | | 2.00 | 3.00 | 5.00 | lower_is_better | 2026-04-27 | Safety event density. |
|
||||
| 7 | utilisation_pct | global | | 70.00 | 60.00 | 50.00 | higher_is_better | 2026-05-01 | Fleet utilisation: drive_hours / engine_on_hours. |
|
||||
| 8 | idle_pct | global | | 15.00 | 20.00 | 25.00 | lower_is_better | 2026-05-01 | Idle as % of engine-on time. |
|
||||
| 9 | idle_pct | cost_centre | osp patrol | 15.00 | 20.00 | 25.00 | lower_is_better | 2026-05-01 | OSP patrol idle target — same as global until calibrated. |
|
||||
| 10 | fuel_kes_per_100km | global | | 12.00 | 14.00 | 16.00 | lower_is_better | 2026-05-01 | Fuel litres per 100km equivalent — uses fuel_100km on devices. |
|
||||
| 11 | mttr_hours | global | | 4.00 | 6.00 | 8.00 | lower_is_better | 2026-05-01 | Mean Time To Resolve, field-service ticket. |
|
||||
| 12 | alarms_per_100km | global | | 2.00 | 3.00 | 5.00 | lower_is_better | 2026-05-01 | Safety event density. |
|
||||
|
||||
## What was dropped
|
||||
|
||||
```sql
|
||||
DROP VIEW IF EXISTS tracksolid.v_sla_inflight; -- depended on ops.tickets + dispatch_log
|
||||
DROP SCHEMA IF EXISTS ops CASCADE; -- tickets, service_log, odometer_readings,
|
||||
-- cost_rates, kpi_targets, vw_service_forecast
|
||||
DROP TABLE IF EXISTS tracksolid.dispatch_log; -- empty; only fed v_sla_inflight
|
||||
```
|
||||
|
||||
**Not dropped in this step:** `dwh_gold` schema and `tracksolid.v_utilisation_daily`
|
||||
(separate decision, pending).
|
||||
|
|
@ -1,75 +0,0 @@
|
|||
# Fleet Registry — Outstanding Data-Quality Issues
|
||||
|
||||
**Date:** 2026-06-08 · **Source:** `tracksolid.devices` (verified live) · **Scope:** 181 devices → 80 vehicles
|
||||
|
||||
**Urgency:** 🔴 **RED** = fix now (blocks ops / safety / visibility) · 🟠 **AMBER** = fix soon (accountability, classification, data trust) · 🟡 **YELLOW** = cleanup.
|
||||
|
||||
> ## ⭐ Biggest single fix: run the driver/plate CSV import
|
||||
> Most items below are **missing source fields**. The prepared **`import_drivers_csv.py --apply`** has **not been run** (`vehicle_category` still 0/181, device count still 181). Running it — or entering the fields at source in the Tracksolid Pro portal — clears the bulk of this list at once.
|
||||
|
||||
---
|
||||
|
||||
## 🔴 RED — fix now
|
||||
|
||||
**R1. No driver phone — 175 / 181 (97%).** Dispatch/escalation can't contact a driver; only 6 are reachable. → Capture mobiles. **Owner:** Operations + Engineering (import).
|
||||
|
||||
**R2. 6 vehicles have NO GPS tracker** (dashcam only → invisible on the live map, no trips/mileage): `KCN 496A · KCQ 215F · KCU 237Z · KDM 306S · KDN 759G · KCZ 199P`. → Field-check; install/repair or decommission. **Owner:** Field ops.
|
||||
|
||||
**R3. 37 fully-unidentified devices being tracked** (no plate, model, or driver; "unknown" status). → Match serial/IMEI to a vehicle or decommission. **Owner:** Field ops + Engineering.
|
||||
|
||||
---
|
||||
|
||||
## 🟠 AMBER — fix soon
|
||||
|
||||
**A1. No driver name — 41 / 181 (23%).** Trips/speeding/idle can't be attributed → no accountability. → Assign drivers at source. **Owner:** Operations. *(CSV covers most.)*
|
||||
|
||||
**A2. 16 vehicles have a tracker but NO dashcam** — no incident/safety video (all 8 motorbikes, the Uganda units, several Proboxes). → Decide which need a camera; schedule installs. **Owner:** Field ops.
|
||||
|
||||
**A3. No vehicle model — 40 / 181 (22%).** Forces guesswork on the field-service vs specialist split. → Set `vehicle_models` at source. **Owner:** Operations. *(CSV covers most.)*
|
||||
|
||||
**A4. Duplicate plates corrupting the count:**
|
||||
- `KDS 453Y` entered twice — `KDS 453Y` (tracker) + `KDS 453 Y` (camera, stray space).
|
||||
- `KCC 199P` vs `KCZ 199P` — both pick-ups, both driver *Mbuvi Kioko*, one tracker + one camera → almost certainly **one vehicle under two plates**.
|
||||
→ Correct the plate in the portal, then re-sync. **Owner:** Operations.
|
||||
|
||||
**A5. Two plates disagree with themselves (model conflict):**
|
||||
|
||||
| Plate | Tracker says | Camera says | Driver |
|
||||
|---|---|---|---|
|
||||
| KCY 080X | Pick-Up | Probox | Lawrence Kijogi |
|
||||
| KCZ 223P | Pick-Up | Probox | Felix Muema |
|
||||
|
||||
→ Confirm the real type; set both devices to match. **Owner:** Operations.
|
||||
|
||||
**A6. `assigned_city` unreliable → regional reporting suspect.** 4 vehicles show two cities (e.g. KDC 490Q: Mombasa/Nairobi; KCY 838X: Mombasa/Voi) — it's inherited from the account, not the vehicle. → Set per vehicle. **Owner:** Operations + Engineering.
|
||||
|
||||
---
|
||||
|
||||
## 🟡 YELLOW — cleanup
|
||||
|
||||
**Y1. Placeholder driver names** — `Garage` (×4), `UG` (×2), `Management_Mazda` (×2), `Parked` (×1). → Replace with real names or a clear "unassigned" convention. **Owner:** Operations.
|
||||
|
||||
**Y2. Missing SIM (26) and cost-centre (31).** → Backfill from records/CSV. **Owner:** Operations + Engineering.
|
||||
|
||||
**Y3. `vehicle_category` empty (0 / 181).** Low urgency — the map/KPIs derive the segment from `vehicle_models` automatically. → Optional; CSV fills it. **Owner:** Engineering.
|
||||
|
||||
---
|
||||
|
||||
## Action plan
|
||||
|
||||
| # | Tier | Issue | Action | Owner | Cleared by CSV import? |
|
||||
|---|---|---|---|---|---|
|
||||
| R1 | 🔴 | No driver phone (97%) | Capture mobiles | Ops + Eng | ✅ mostly |
|
||||
| R2 | 🔴 | 6 vehicles, no GPS tracker | Field-check / install | Field ops | — |
|
||||
| R3 | 🔴 | 37 unidentified devices | Identify or decommission | Field ops + Eng | ✅ partly |
|
||||
| A1 | 🟠 | No driver name (23%) | Assign drivers | Ops | ✅ mostly |
|
||||
| A2 | 🟠 | 16 vehicles, no dashcam | Schedule installs | Field ops | — |
|
||||
| A3 | 🟠 | No vehicle model (22%) | Set model at source | Ops | ✅ mostly |
|
||||
| A4 | 🟠 | Duplicate plates (KDS 453Y; KCC/KCZ 199P) | Fix plate, re-sync | Ops | — |
|
||||
| A5 | 🟠 | Model conflicts (KCY 080X, KCZ 223P) | Confirm type | Ops | partial |
|
||||
| A6 | 🟠 | Unreliable assigned_city | Set per vehicle | Ops + Eng | ✅ yes |
|
||||
| Y1 | 🟡 | Placeholder driver names | Replace | Ops | partial |
|
||||
| Y2 | 🟡 | Missing SIM (26) / cost-centre (31) | Backfill | Ops + Eng | ✅ mostly |
|
||||
| Y3 | 🟡 | vehicle_category empty | Optional populate | Eng | ✅ yes |
|
||||
|
||||
**Suggested order:** (1) run/validate the **CSV import** — clears R1, R3(part), A1, A3, A6, Y2, Y3; (2) field-ops sweep for **R2 + A2** (tracker/camera hardware); (3) Operations fixes **A4/A5 plates + Y1** in the portal; (4) re-run the scan to confirm.
|
||||
File diff suppressed because it is too large
Load diff
0
documents.txt
Normal file
0
documents.txt
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -49,7 +49,7 @@ from ts_shared_rev import clean, clean_num, clean_ts, get_conn, get_logger
|
|||
|
||||
log = get_logger("csv_import")
|
||||
|
||||
DEFAULT_CSV_PATH = Path(__file__).parent / "data" / "20260427_FSG_Vehicles_mitieng.csv"
|
||||
DEFAULT_CSV_PATH = Path(__file__).parent / "20260427_FSG_Vehicles_mitieng.csv"
|
||||
|
||||
# Columns fetched from DB for diff comparison.
|
||||
DB_COLS = [
|
||||
|
|
|
|||
|
|
@ -1,554 +0,0 @@
|
|||
-- 11_reporting_schema.sql
|
||||
-- Map-dashboard read layer consumed by dashboard_api_rev.py:
|
||||
-- reporting.fn_live_positions / fn_vehicle_track / fn_trips_for_map
|
||||
-- + the v_trips materialized view, filter/summary views, and refresh_log.
|
||||
-- Captured from prod 2026-06-05 to close the reproducibility gap (these objects
|
||||
-- lived only on the live DB, in no migration). Every object uses IF NOT EXISTS /
|
||||
-- CREATE OR REPLACE so the file is safe to re-apply.
|
||||
--
|
||||
-- NOTE: the v_trips materialized view is created WITH DATA (populated once). On
|
||||
-- prod it is owned by role `reporting_refresher` and kept current by an external
|
||||
-- REFRESH job; that role + refresh schedule are infrastructure, NOT created here.
|
||||
-- On a fresh rebuild v_trips is populated at creation but will go stale until a
|
||||
-- refresh job is wired (see docs). The dashboard map functions still work, just
|
||||
-- against the snapshot until then.
|
||||
|
||||
CREATE SCHEMA IF NOT EXISTS reporting;
|
||||
|
||||
-- Bodies reference base tables unqualified (devices, trips, …) + PostGIS;
|
||||
-- resolve via search_path so this applies cleanly on a fresh DB.
|
||||
SET search_path = tracksolid, reporting, public;
|
||||
|
||||
-- ── helper ───────────────────────────────────────────────────────────────────
|
||||
CREATE OR REPLACE FUNCTION reporting.normalize_plate(p text)
|
||||
RETURNS text
|
||||
LANGUAGE sql
|
||||
IMMUTABLE PARALLEL SAFE
|
||||
AS $function$
|
||||
SELECT regexp_replace(
|
||||
regexp_replace(trim(p), '\s+', ' ', 'g'),
|
||||
'(\d) ([A-Z])$', '\1\2'
|
||||
)
|
||||
$function$;
|
||||
|
||||
-- ── refresh audit table ──────────────────────────────────────────────────────
|
||||
CREATE TABLE IF NOT EXISTS reporting.refresh_log (
|
||||
refreshed_at timestamptz DEFAULT now() NOT NULL,
|
||||
source text DEFAULT 'n8n'::text NOT NULL,
|
||||
duration_ms integer,
|
||||
row_count integer,
|
||||
notes text
|
||||
);
|
||||
|
||||
-- ── v_trips materialized view (+ indexes) ────────────────────────────────────
|
||||
CREATE MATERIALIZED VIEW IF NOT EXISTS reporting.v_trips AS
|
||||
WITH device_trip_counts AS (
|
||||
SELECT trips.imei,
|
||||
count(*) AS trip_count
|
||||
FROM trips
|
||||
GROUP BY trips.imei
|
||||
), primary_device AS (
|
||||
SELECT DISTINCT ON ((reporting.normalize_plate(d_1.vehicle_number))) reporting.normalize_plate(d_1.vehicle_number) AS vehicle_number_norm,
|
||||
d_1.imei AS primary_imei
|
||||
FROM devices d_1
|
||||
LEFT JOIN device_trip_counts c USING (imei)
|
||||
WHERE d_1.vehicle_number IS NOT NULL AND d_1.enabled_flag = 1
|
||||
ORDER BY (reporting.normalize_plate(d_1.vehicle_number)), (COALESCE(c.trip_count, 0::bigint)) DESC, d_1.activation_time, d_1.imei
|
||||
)
|
||||
SELECT t.id AS trip_id,
|
||||
t.imei,
|
||||
d.device_name,
|
||||
reporting.normalize_plate(d.vehicle_number) AS vehicle_number,
|
||||
d.vehicle_models,
|
||||
d.vehicle_category,
|
||||
d.cost_centre,
|
||||
d.assigned_city,
|
||||
d.driver_name AS assigned_driver,
|
||||
(t.start_time AT TIME ZONE 'Africa/Nairobi'::text) AS start_time,
|
||||
(t.end_time AT TIME ZONE 'Africa/Nairobi'::text) AS end_time,
|
||||
(t.start_time AT TIME ZONE 'Africa/Nairobi'::text)::date AS trip_date,
|
||||
EXTRACT(hour FROM (t.start_time AT TIME ZONE 'Africa/Nairobi'::text))::integer AS start_hour,
|
||||
EXTRACT(dow FROM (t.start_time AT TIME ZONE 'Africa/Nairobi'::text))::integer AS start_dow,
|
||||
row_number() OVER (PARTITION BY t.imei, ((t.start_time AT TIME ZONE 'Africa/Nairobi'::text)::date) ORDER BY t.start_time) AS daily_seq,
|
||||
t.distance_km,
|
||||
t.avg_speed_kmh,
|
||||
t.max_speed_kmh,
|
||||
t.idle_time_s,
|
||||
t.driving_time_s,
|
||||
t.fuel_consumed_l,
|
||||
t.waypoints_count,
|
||||
t.start_address,
|
||||
t.end_address,
|
||||
t.start_geom,
|
||||
t.end_geom,
|
||||
t.route_geom,
|
||||
st_asgeojson(st_simplifypreservetopology(t.route_geom, 0.00005::double precision))::json AS route_geojson,
|
||||
st_numpoints(t.route_geom) >= 2 AND st_length(t.route_geom::geography) > 50::double precision AS is_meaningful_route,
|
||||
(t.updated_at AT TIME ZONE 'Africa/Nairobi'::text) AS updated_at
|
||||
FROM trips t
|
||||
LEFT JOIN devices d USING (imei)
|
||||
LEFT JOIN primary_device pd ON pd.vehicle_number_norm = reporting.normalize_plate(d.vehicle_number)
|
||||
WHERE d.vehicle_number IS NULL OR pd.primary_imei IS NULL OR t.imei = pd.primary_imei
|
||||
WITH DATA;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_v_trips_city_trip_date ON reporting.v_trips USING btree (assigned_city, trip_date);
|
||||
CREATE INDEX IF NOT EXISTS ix_v_trips_cost_centre_trip_date ON reporting.v_trips USING btree (cost_centre, trip_date);
|
||||
CREATE INDEX IF NOT EXISTS ix_v_trips_driver_trip_date ON reporting.v_trips USING btree (assigned_driver, trip_date);
|
||||
CREATE INDEX IF NOT EXISTS ix_v_trips_imei_trip_date ON reporting.v_trips USING btree (imei, trip_date);
|
||||
CREATE INDEX IF NOT EXISTS ix_v_trips_meaningful_date ON reporting.v_trips USING btree (trip_date) WHERE is_meaningful_route;
|
||||
CREATE INDEX IF NOT EXISTS ix_v_trips_trip_date ON reporting.v_trips USING btree (trip_date);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ix_v_trips_trip_id ON reporting.v_trips USING btree (trip_id);
|
||||
|
||||
-- ── views (dependency-ordered) ───────────────────────────────────────────────
|
||||
CREATE OR REPLACE VIEW reporting.v_live_positions AS
|
||||
WITH primary_device AS (
|
||||
SELECT DISTINCT ON ((reporting.normalize_plate(d_1.vehicle_number))) reporting.normalize_plate(d_1.vehicle_number) AS vehicle_number,
|
||||
d_1.imei AS primary_imei
|
||||
FROM devices d_1
|
||||
LEFT JOIN live_positions lp_1 ON lp_1.imei = d_1.imei
|
||||
WHERE d_1.vehicle_number IS NOT NULL AND d_1.enabled_flag = 1
|
||||
ORDER BY (reporting.normalize_plate(d_1.vehicle_number)), (
|
||||
CASE
|
||||
WHEN (d_1.mc_type = ANY (ARRAY['GT06E'::text, 'X3'::text, 'AT4'::text])) AND lp_1.gps_time >= (now() - '24:00:00'::interval) THEN 0
|
||||
ELSE 1
|
||||
END), lp_1.gps_time DESC NULLS LAST, (
|
||||
CASE d_1.mc_type
|
||||
WHEN 'GT06E'::text THEN 1
|
||||
WHEN 'X3'::text THEN 2
|
||||
WHEN 'AT4'::text THEN 3
|
||||
WHEN 'JC400P'::text THEN 4
|
||||
ELSE 5
|
||||
END), d_1.activation_time, d_1.imei
|
||||
)
|
||||
SELECT lp.imei,
|
||||
pd.vehicle_number,
|
||||
d.driver_name AS assigned_driver,
|
||||
d.cost_centre,
|
||||
d.assigned_city,
|
||||
d.vehicle_category,
|
||||
d.vehicle_models,
|
||||
d.mc_type,
|
||||
CASE d.mc_type
|
||||
WHEN 'GT06E'::text THEN 'tracker'::text
|
||||
WHEN 'X3'::text THEN 'tracker'::text
|
||||
WHEN 'AT4'::text THEN 'tracker'::text
|
||||
WHEN 'JC400P'::text THEN 'camera'::text
|
||||
ELSE 'other'::text
|
||||
END AS device_kind,
|
||||
lp.lat,
|
||||
lp.lng,
|
||||
lp.speed,
|
||||
lp.direction,
|
||||
lp.acc_status,
|
||||
lp.device_status,
|
||||
lp.gps_signal,
|
||||
lp.gps_num,
|
||||
lp.current_mileage,
|
||||
lp.loc_desc,
|
||||
lp.gps_time,
|
||||
lp.updated_at,
|
||||
(lp.gps_time AT TIME ZONE 'Africa/Nairobi'::text) AS gps_time_eat,
|
||||
(lp.updated_at AT TIME ZONE 'Africa/Nairobi'::text) AS updated_at_eat,
|
||||
round(EXTRACT(epoch FROM now() - lp.gps_time) / 3600::numeric, 2) AS source_age_hours
|
||||
FROM live_positions lp
|
||||
JOIN primary_device pd ON pd.primary_imei = lp.imei
|
||||
JOIN devices d ON d.imei = lp.imei;
|
||||
|
||||
CREATE OR REPLACE VIEW reporting.v_trips_today AS
|
||||
SELECT trip_id,
|
||||
imei,
|
||||
device_name,
|
||||
vehicle_number,
|
||||
vehicle_models,
|
||||
vehicle_category,
|
||||
cost_centre,
|
||||
assigned_city,
|
||||
assigned_driver,
|
||||
start_time,
|
||||
end_time,
|
||||
trip_date,
|
||||
start_hour,
|
||||
start_dow,
|
||||
daily_seq,
|
||||
distance_km,
|
||||
avg_speed_kmh,
|
||||
max_speed_kmh,
|
||||
idle_time_s,
|
||||
driving_time_s,
|
||||
fuel_consumed_l,
|
||||
waypoints_count,
|
||||
start_address,
|
||||
end_address,
|
||||
start_geom,
|
||||
end_geom,
|
||||
route_geom,
|
||||
route_geojson,
|
||||
is_meaningful_route,
|
||||
updated_at
|
||||
FROM reporting.v_trips
|
||||
WHERE trip_date = (now() AT TIME ZONE 'Africa/Nairobi'::text)::date;
|
||||
|
||||
CREATE OR REPLACE VIEW reporting.v_filter_drivers AS
|
||||
SELECT DISTINCT assigned_driver AS driver
|
||||
FROM reporting.v_trips
|
||||
WHERE assigned_driver IS NOT NULL
|
||||
ORDER BY assigned_driver;
|
||||
|
||||
CREATE OR REPLACE VIEW reporting.v_filter_cost_centres AS
|
||||
SELECT DISTINCT cost_centre
|
||||
FROM reporting.v_trips
|
||||
WHERE cost_centre IS NOT NULL
|
||||
ORDER BY cost_centre;
|
||||
|
||||
CREATE OR REPLACE VIEW reporting.v_filter_vehicles AS
|
||||
SELECT vehicle_number,
|
||||
string_agg(DISTINCT assigned_driver, ', '::text ORDER BY assigned_driver) AS drivers,
|
||||
(array_agg(cost_centre ORDER BY start_time DESC NULLS LAST))[1] AS cost_centre,
|
||||
(array_agg(assigned_city ORDER BY start_time DESC NULLS LAST))[1] AS assigned_city
|
||||
FROM reporting.v_trips
|
||||
WHERE vehicle_number IS NOT NULL
|
||||
GROUP BY vehicle_number
|
||||
ORDER BY vehicle_number;
|
||||
|
||||
CREATE OR REPLACE VIEW reporting.v_filter_cities AS
|
||||
SELECT DISTINCT assigned_city
|
||||
FROM reporting.v_trips
|
||||
WHERE assigned_city IS NOT NULL
|
||||
ORDER BY assigned_city;
|
||||
|
||||
CREATE OR REPLACE VIEW reporting.v_daily_summary AS
|
||||
SELECT trip_date,
|
||||
cost_centre,
|
||||
assigned_city,
|
||||
vehicle_number,
|
||||
assigned_driver,
|
||||
count(*) AS trip_count,
|
||||
sum(distance_km) AS total_km,
|
||||
sum(driving_time_s)::numeric / 3600.0 AS driving_hours,
|
||||
sum(idle_time_s)::numeric / 3600.0 AS idle_hours,
|
||||
round(100.0 * sum(idle_time_s)::numeric / NULLIF(sum(idle_time_s + driving_time_s), 0)::numeric, 1) AS idle_pct,
|
||||
min(start_time) AS first_trip_start,
|
||||
max(end_time) AS last_trip_end,
|
||||
avg(avg_speed_kmh) AS avg_speed_kmh,
|
||||
max(max_speed_kmh) AS max_speed_kmh,
|
||||
st_asgeojson(st_simplifypreservetopology(st_collect(route_geom), 0.00005::double precision))::json AS day_routes_geojson
|
||||
FROM reporting.v_trips
|
||||
WHERE is_meaningful_route
|
||||
GROUP BY trip_date, cost_centre, assigned_city, vehicle_number, assigned_driver;
|
||||
|
||||
CREATE OR REPLACE VIEW reporting.v_weekly_summary AS
|
||||
SELECT date_trunc('week'::text, trip_date::timestamp with time zone)::date AS week_start,
|
||||
cost_centre,
|
||||
assigned_city,
|
||||
vehicle_number,
|
||||
assigned_driver,
|
||||
count(*) AS trip_count,
|
||||
count(DISTINCT trip_date) AS active_days,
|
||||
sum(distance_km) AS total_km,
|
||||
sum(driving_time_s)::numeric / 3600.0 AS driving_hours,
|
||||
sum(idle_time_s)::numeric / 3600.0 AS idle_hours,
|
||||
avg(distance_km) AS avg_trip_km
|
||||
FROM reporting.v_trips
|
||||
WHERE is_meaningful_route
|
||||
GROUP BY (date_trunc('week'::text, trip_date::timestamp with time zone)::date), cost_centre, assigned_city, vehicle_number, assigned_driver;
|
||||
|
||||
CREATE OR REPLACE VIEW reporting.v_monthly_summary AS
|
||||
SELECT date_trunc('month'::text, trip_date::timestamp with time zone)::date AS month_start,
|
||||
to_char(trip_date::timestamp with time zone, 'YYYY-MM'::text) AS month_label,
|
||||
cost_centre,
|
||||
assigned_city,
|
||||
vehicle_category,
|
||||
vehicle_number,
|
||||
assigned_driver,
|
||||
count(*) AS trip_count,
|
||||
count(DISTINCT trip_date) AS active_days,
|
||||
sum(distance_km) AS total_km,
|
||||
sum(driving_time_s)::numeric / 3600.0 AS driving_hours,
|
||||
sum(idle_time_s)::numeric / 3600.0 AS idle_hours,
|
||||
round(100.0 * sum(idle_time_s)::numeric / NULLIF(sum(idle_time_s + driving_time_s), 0)::numeric, 1) AS idle_pct,
|
||||
round(sum(distance_km) / NULLIF(count(DISTINCT trip_date), 0)::numeric, 1) AS km_per_active_day,
|
||||
round(sum(distance_km) / NULLIF(count(*), 0)::numeric, 1) AS km_per_trip,
|
||||
avg(avg_speed_kmh) AS avg_speed_kmh,
|
||||
max(max_speed_kmh) AS peak_speed_kmh
|
||||
FROM reporting.v_trips
|
||||
WHERE is_meaningful_route
|
||||
GROUP BY (date_trunc('month'::text, trip_date::timestamp with time zone)::date), (to_char(trip_date::timestamp with time zone, 'YYYY-MM'::text)), cost_centre, assigned_city, vehicle_category, vehicle_number, assigned_driver;
|
||||
|
||||
CREATE OR REPLACE VIEW reporting.v_daily_cost_centre AS
|
||||
SELECT trip_date,
|
||||
cost_centre,
|
||||
count(DISTINCT imei) AS active_vehicles,
|
||||
count(DISTINCT assigned_driver) AS active_drivers,
|
||||
count(*) AS trip_count,
|
||||
sum(distance_km) AS total_km,
|
||||
sum(driving_time_s)::numeric / 3600.0 AS driving_hours,
|
||||
sum(idle_time_s)::numeric / 3600.0 AS idle_hours,
|
||||
round(100.0 * sum(idle_time_s)::numeric / NULLIF(sum(idle_time_s + driving_time_s), 0)::numeric, 1) AS idle_pct,
|
||||
round(sum(distance_km) / NULLIF(count(DISTINCT imei), 0)::numeric, 1) AS km_per_vehicle
|
||||
FROM reporting.v_trips
|
||||
WHERE is_meaningful_route AND cost_centre IS NOT NULL
|
||||
GROUP BY trip_date, cost_centre;
|
||||
|
||||
CREATE OR REPLACE VIEW reporting.v_weekly_cost_centre AS
|
||||
SELECT date_trunc('week'::text, trip_date::timestamp with time zone)::date AS week_start,
|
||||
cost_centre,
|
||||
count(DISTINCT imei) AS active_vehicles,
|
||||
count(DISTINCT assigned_driver) AS active_drivers,
|
||||
count(DISTINCT trip_date) AS active_days,
|
||||
count(*) AS trip_count,
|
||||
sum(distance_km) AS total_km,
|
||||
sum(driving_time_s)::numeric / 3600.0 AS driving_hours,
|
||||
sum(idle_time_s)::numeric / 3600.0 AS idle_hours,
|
||||
round(100.0 * sum(idle_time_s)::numeric / NULLIF(sum(idle_time_s + driving_time_s), 0)::numeric, 1) AS idle_pct,
|
||||
round(sum(distance_km) / NULLIF(count(DISTINCT imei), 0)::numeric, 1) AS km_per_vehicle
|
||||
FROM reporting.v_trips
|
||||
WHERE is_meaningful_route AND cost_centre IS NOT NULL
|
||||
GROUP BY (date_trunc('week'::text, trip_date::timestamp with time zone)::date), cost_centre;
|
||||
|
||||
CREATE OR REPLACE VIEW reporting.v_monthly_cost_centre AS
|
||||
SELECT date_trunc('month'::text, trip_date::timestamp with time zone)::date AS month_start,
|
||||
to_char(trip_date::timestamp with time zone, 'YYYY-MM'::text) AS month_label,
|
||||
cost_centre,
|
||||
count(DISTINCT imei) AS active_vehicles,
|
||||
count(DISTINCT assigned_driver) AS active_drivers,
|
||||
count(DISTINCT trip_date) AS active_days,
|
||||
count(*) AS trip_count,
|
||||
sum(distance_km) AS total_km,
|
||||
sum(driving_time_s)::numeric / 3600.0 AS driving_hours,
|
||||
sum(idle_time_s)::numeric / 3600.0 AS idle_hours,
|
||||
round(100.0 * sum(idle_time_s)::numeric / NULLIF(sum(idle_time_s + driving_time_s), 0)::numeric, 1) AS idle_pct,
|
||||
round(sum(distance_km) / NULLIF(count(DISTINCT imei), 0)::numeric, 1) AS km_per_vehicle
|
||||
FROM reporting.v_trips
|
||||
WHERE is_meaningful_route AND cost_centre IS NOT NULL
|
||||
GROUP BY (date_trunc('month'::text, trip_date::timestamp with time zone)::date), (to_char(trip_date::timestamp with time zone, 'YYYY-MM'::text)), cost_centre;
|
||||
|
||||
-- ── refresh_log index ────────────────────────────────────────────────────────
|
||||
CREATE INDEX IF NOT EXISTS ix_refresh_log_refreshed_at ON reporting.refresh_log USING btree (refreshed_at DESC);
|
||||
|
||||
-- ── map functions ────────────────────────────────────────────────────────────
|
||||
CREATE OR REPLACE FUNCTION reporting.fn_live_positions(p_cost_centre text DEFAULT NULL::text, p_acc_status text DEFAULT NULL::text)
|
||||
RETURNS jsonb
|
||||
LANGUAGE plpgsql
|
||||
STABLE
|
||||
AS $function$
|
||||
DECLARE
|
||||
v_result jsonb;
|
||||
BEGIN
|
||||
p_cost_centre := NULLIF(p_cost_centre, '');
|
||||
p_acc_status := NULLIF(p_acc_status, '');
|
||||
WITH filtered AS (
|
||||
SELECT * FROM reporting.v_live_positions
|
||||
WHERE (p_cost_centre IS NULL OR cost_centre = p_cost_centre)
|
||||
AND (p_acc_status IS NULL OR acc_status = p_acc_status)
|
||||
)
|
||||
SELECT jsonb_build_object(
|
||||
'summary', jsonb_build_object(
|
||||
'vehicle_count', COUNT(*),
|
||||
-- "moving" and "parked" both restrict to devices that have reported
|
||||
-- within the OFFLINE_THRESHOLD (24 h) so they represent the live
|
||||
-- fleet, not equipment-failure stragglers. "offline" is its own
|
||||
-- counter for the > 24 h tail.
|
||||
'moving', COUNT(*) FILTER (WHERE acc_status = '1'
|
||||
AND source_age_hours < 24),
|
||||
'parked', COUNT(*) FILTER (WHERE acc_status = '0'
|
||||
AND source_age_hours < 24),
|
||||
'offline', COUNT(*) FILTER (WHERE source_age_hours >= 24),
|
||||
'median_speed_moving', percentile_cont(0.5) WITHIN GROUP (ORDER BY speed)
|
||||
FILTER (WHERE acc_status = '1'
|
||||
AND source_age_hours < 24
|
||||
AND speed > 0),
|
||||
'last_batch_at', to_char(MAX(updated_at) AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS'),
|
||||
'oldest_fix_at', to_char(MIN(gps_time) AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS'),
|
||||
'newest_fix_at', to_char(MAX(gps_time) AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS'),
|
||||
'last_batch_utc', MAX(updated_at),
|
||||
'newest_fix_utc', MAX(gps_time)
|
||||
),
|
||||
'geojson', jsonb_build_object(
|
||||
'type', 'FeatureCollection',
|
||||
'features', COALESCE(jsonb_agg(
|
||||
jsonb_build_object(
|
||||
'type', 'Feature',
|
||||
'properties', jsonb_build_object(
|
||||
'imei', imei,
|
||||
'vehicle_number', vehicle_number,
|
||||
'driver', assigned_driver,
|
||||
'cost_centre', cost_centre,
|
||||
'assigned_city', assigned_city,
|
||||
'vehicle_category', vehicle_category,
|
||||
'mc_type', mc_type,
|
||||
'device_kind', device_kind,
|
||||
'source_age_hours', source_age_hours,
|
||||
'speed', speed,
|
||||
'direction', direction,
|
||||
'acc_status', acc_status,
|
||||
'device_status', device_status,
|
||||
'gps_signal', gps_signal,
|
||||
'gps_num', gps_num,
|
||||
'current_mileage', current_mileage,
|
||||
'loc_desc', loc_desc,
|
||||
'gps_time', to_char(gps_time AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS'),
|
||||
'updated_at', to_char(updated_at AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS'),
|
||||
'gps_time_utc', gps_time,
|
||||
'updated_at_utc', updated_at
|
||||
),
|
||||
'geometry', jsonb_build_object(
|
||||
'type', 'Point',
|
||||
'coordinates', jsonb_build_array(lng, lat)
|
||||
)
|
||||
)
|
||||
), '[]'::jsonb)
|
||||
)
|
||||
) INTO v_result FROM filtered;
|
||||
|
||||
RETURN v_result;
|
||||
END $function$;
|
||||
|
||||
CREATE OR REPLACE FUNCTION reporting.fn_vehicle_track(p_vehicle_number text, p_hours integer DEFAULT 1)
|
||||
RETURNS jsonb
|
||||
LANGUAGE sql
|
||||
STABLE
|
||||
AS $function$
|
||||
-- IMEI lookup reuses the already-deduped reporting.v_live_positions instead
|
||||
-- of re-running the primary_device CTE against tracksolid.trips. That keeps
|
||||
-- reporting_reader off tracksolid.trips entirely.
|
||||
WITH pts AS (
|
||||
SELECT ph.gps_time, ph.lat, ph.lng, ph.speed, ph.direction
|
||||
FROM tracksolid.position_history ph
|
||||
JOIN reporting.v_live_positions lv ON lv.imei = ph.imei
|
||||
WHERE lv.vehicle_number = reporting.normalize_plate(p_vehicle_number)
|
||||
AND ph.gps_time >= NOW() - make_interval(hours => GREATEST(p_hours, 1))
|
||||
ORDER BY ph.gps_time
|
||||
)
|
||||
SELECT jsonb_build_object(
|
||||
'type', 'Feature',
|
||||
'properties', jsonb_build_object(
|
||||
'vehicle_number', reporting.normalize_plate(p_vehicle_number),
|
||||
'hours', p_hours,
|
||||
'points', (SELECT COUNT(*) FROM pts),
|
||||
'first_fix', (SELECT to_char(MIN(gps_time) AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS') FROM pts),
|
||||
'last_fix', (SELECT to_char(MAX(gps_time) AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS') FROM pts)
|
||||
),
|
||||
'geometry', jsonb_build_object(
|
||||
'type', 'LineString',
|
||||
'coordinates', COALESCE(
|
||||
(SELECT jsonb_agg(jsonb_build_array(lng, lat) ORDER BY gps_time) FROM pts),
|
||||
'[]'::jsonb)
|
||||
)
|
||||
);
|
||||
$function$;
|
||||
|
||||
CREATE OR REPLACE FUNCTION reporting.fn_trips_for_map(p_vehicle_numbers text[] DEFAULT NULL::text[], p_driver text DEFAULT NULL::text, p_cost_centre text DEFAULT NULL::text, p_assigned_city text DEFAULT NULL::text, p_start_date date DEFAULT NULL::date, p_end_date date DEFAULT NULL::date)
|
||||
RETURNS jsonb
|
||||
LANGUAGE plpgsql
|
||||
STABLE
|
||||
AS $function$
|
||||
DECLARE
|
||||
v_start date := COALESCE(p_start_date, (NOW() AT TIME ZONE 'Africa/Nairobi')::date);
|
||||
v_end date := COALESCE(p_end_date, (NOW() AT TIME ZONE 'Africa/Nairobi')::date);
|
||||
v_days int := v_end - v_start + 1;
|
||||
v_result jsonb;
|
||||
BEGIN
|
||||
p_driver := NULLIF(p_driver, '');
|
||||
p_cost_centre := NULLIF(p_cost_centre, '');
|
||||
p_assigned_city := NULLIF(p_assigned_city, '');
|
||||
-- 31-day guardrail: tripped only when NO filter is set AND range > 31 days.
|
||||
-- Vehicle list (non-empty), driver, cost-centre, OR city each waives it.
|
||||
IF (p_vehicle_numbers IS NULL OR cardinality(p_vehicle_numbers) = 0)
|
||||
AND p_driver IS NULL
|
||||
AND p_cost_centre IS NULL
|
||||
AND p_assigned_city IS NULL
|
||||
AND v_days > 31 THEN
|
||||
RAISE EXCEPTION
|
||||
'Range too wide for trip-grain map (% days). Pick a vehicle, driver, cost centre, or city — or narrow the period to 31 days or fewer.',
|
||||
v_days
|
||||
USING ERRCODE = 'check_violation';
|
||||
END IF;
|
||||
|
||||
WITH filtered AS (
|
||||
SELECT *
|
||||
FROM reporting.v_trips
|
||||
WHERE is_meaningful_route
|
||||
AND (p_vehicle_numbers IS NULL
|
||||
OR cardinality(p_vehicle_numbers) = 0
|
||||
OR vehicle_number = ANY(p_vehicle_numbers))
|
||||
AND (p_driver IS NULL OR assigned_driver = p_driver)
|
||||
AND (p_cost_centre IS NULL OR cost_centre = p_cost_centre)
|
||||
AND (p_assigned_city IS NULL OR assigned_city = p_assigned_city)
|
||||
AND (p_start_date IS NULL OR trip_date >= p_start_date)
|
||||
AND (p_end_date IS NULL OR trip_date <= p_end_date)
|
||||
)
|
||||
SELECT jsonb_build_object(
|
||||
'summary', jsonb_build_object(
|
||||
'trip_count', COUNT(*),
|
||||
'total_km', ROUND(COALESCE(SUM(distance_km), 0)::numeric, 1),
|
||||
'driving_hours', ROUND((COALESCE(SUM(driving_time_s), 0) / 3600.0)::numeric, 1),
|
||||
'idle_hours', ROUND((COALESCE(SUM(idle_time_s), 0) / 3600.0)::numeric, 1),
|
||||
'unique_vehicles', COUNT(DISTINCT vehicle_number),
|
||||
'unique_drivers', COUNT(DISTINCT assigned_driver),
|
||||
'date_min', MIN(trip_date),
|
||||
'date_max', MAX(trip_date),
|
||||
-- First trip's start (chronologically first) + reverse-geocoded location
|
||||
'first_trip_start_time',
|
||||
(array_agg(to_char(start_time, 'YYYY-MM-DD HH24:MI:SS') ORDER BY start_time))[1],
|
||||
'first_trip_start_address',
|
||||
(array_agg(start_address ORDER BY start_time))[1],
|
||||
'first_trip_vehicle',
|
||||
(array_agg(vehicle_number ORDER BY start_time))[1],
|
||||
-- Last trip's end (chronologically latest) + reverse-geocoded location
|
||||
'last_trip_end_time',
|
||||
(array_agg(to_char(end_time, 'YYYY-MM-DD HH24:MI:SS') ORDER BY end_time DESC NULLS LAST))[1],
|
||||
'last_trip_end_address',
|
||||
(array_agg(end_address ORDER BY end_time DESC NULLS LAST))[1],
|
||||
'last_trip_vehicle',
|
||||
(array_agg(vehicle_number ORDER BY end_time DESC NULLS LAST))[1]
|
||||
),
|
||||
'geojson', jsonb_build_object(
|
||||
'type', 'FeatureCollection',
|
||||
'features', COALESCE(jsonb_agg(
|
||||
jsonb_build_object(
|
||||
'type', 'Feature',
|
||||
'properties', jsonb_build_object(
|
||||
'trip_id', trip_id,
|
||||
'vehicle_number', vehicle_number,
|
||||
'driver', assigned_driver,
|
||||
'cost_centre', cost_centre,
|
||||
'assigned_city', assigned_city,
|
||||
'trip_date', trip_date,
|
||||
'daily_seq', daily_seq,
|
||||
'start_time', to_char(start_time, 'YYYY-MM-DD HH24:MI:SS'),
|
||||
'end_time', to_char(end_time, 'YYYY-MM-DD HH24:MI:SS'),
|
||||
'distance_km', ROUND(COALESCE(distance_km, 0)::numeric, 2),
|
||||
'duration_min', ROUND((COALESCE(driving_time_s, 0) / 60.0)::numeric, 0)
|
||||
),
|
||||
'geometry', route_geojson
|
||||
)
|
||||
ORDER BY vehicle_number, trip_date, daily_seq
|
||||
), '[]'::jsonb)
|
||||
)
|
||||
)
|
||||
INTO v_result
|
||||
FROM filtered;
|
||||
|
||||
RETURN v_result;
|
||||
END
|
||||
$function$;
|
||||
|
||||
-- ── grants (guarded: roles may not exist on a fresh DB) ───────────────────────
|
||||
DO $grants$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname='grafana_ro') THEN
|
||||
GRANT USAGE ON SCHEMA reporting TO grafana_ro;
|
||||
GRANT SELECT ON ALL TABLES IN SCHEMA reporting TO grafana_ro;
|
||||
GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA reporting TO grafana_ro;
|
||||
END IF;
|
||||
IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname='tracksolid_owner') THEN
|
||||
GRANT USAGE ON SCHEMA reporting TO tracksolid_owner;
|
||||
END IF;
|
||||
END $grants$;
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
-- 12_drop_ops.sql
|
||||
-- Purge the dormant `ops` schema (workshop / tickets / dispatch / SLA / odometer)
|
||||
-- and the dispatch/SLA artefacts that lived in `tracksolid`. These features were
|
||||
-- never implemented and may take a different direction; the schema and views were
|
||||
-- empty (or seed-only) and unused by the live map/dashboard pipeline.
|
||||
--
|
||||
-- Created by migrations 06 (ops schema + dispatch_log), 07 (v_sla_inflight) and
|
||||
-- 08 (ops.cost_rates / ops.kpi_targets seed). Per the repo rule we do NOT rewrite
|
||||
-- those applied migrations — this forward migration drops the objects instead. On
|
||||
-- a fresh rebuild 06/07/08 create them, then this file removes them again.
|
||||
--
|
||||
-- Pre-drop snapshot of the only seeded tables: docs/reports/260605_ops_purge_backup.md
|
||||
-- (the seed is also reproducible from 08_analytics_config.sql).
|
||||
--
|
||||
-- Every statement is IF EXISTS so the file is safe to re-apply.
|
||||
--
|
||||
-- NOTE: dwh_gold and tracksolid.v_utilisation_daily are intentionally NOT touched
|
||||
-- here — that is a separate decision.
|
||||
|
||||
-- View first: it reads ops.tickets + tracksolid.dispatch_log, so it must go before
|
||||
-- the objects it depends on (avoids an implicit CASCADE surprise).
|
||||
DROP VIEW IF EXISTS tracksolid.v_sla_inflight;
|
||||
|
||||
-- The whole ops schema: tickets, service_log, odometer_readings, cost_rates,
|
||||
-- kpi_targets, and the view vw_service_forecast. CASCADE clears intra-schema deps.
|
||||
DROP SCHEMA IF EXISTS ops CASCADE;
|
||||
|
||||
-- Dispatch feature table — lived in the tracksolid schema, empty, only fed
|
||||
-- v_sla_inflight (now dropped).
|
||||
DROP TABLE IF EXISTS tracksolid.dispatch_log;
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
-- 13_drop_dwh_gold.sql
|
||||
-- Purge the dormant `dwh_gold` aggregate schema and its dependent view. The nightly
|
||||
-- ETL (dwh_gold.refresh_daily_metrics) was never scheduled, both fact/dim tables were
|
||||
-- empty, and nothing in the live map/dashboard pipeline reads them. These analytics
|
||||
-- may take a different direction later.
|
||||
--
|
||||
-- Created by migrations 02 (schema), 05 (dwh_gold expansion + refresh_daily_metrics)
|
||||
-- and 07 (tracksolid.v_utilisation_daily). Per the repo rule we do NOT rewrite those
|
||||
-- applied migrations — this forward migration drops the objects instead. On a fresh
|
||||
-- rebuild 02/05/07 create them, then this file removes them again.
|
||||
--
|
||||
-- Both tables were empty at drop time, so there is no data backup (cf. the ops purge,
|
||||
-- which had seed rows — docs/reports/260605_ops_purge_backup.md). Companion to
|
||||
-- 12_drop_ops.sql; together they retire the unused ops + dwh_gold analytics layers.
|
||||
--
|
||||
-- Every statement is IF EXISTS so the file is safe to re-apply.
|
||||
|
||||
-- View first: it reads dwh_gold.dim_vehicles + dwh_gold.fact_daily_fleet_metrics.
|
||||
DROP VIEW IF EXISTS tracksolid.v_utilisation_daily;
|
||||
|
||||
-- The whole dwh_gold schema: dim_vehicles, fact_daily_fleet_metrics, the
|
||||
-- dim_vehicles sequence/indexes, and the refresh_daily_metrics() function.
|
||||
DROP SCHEMA IF EXISTS dwh_gold CASCADE;
|
||||
|
|
@ -1,104 +0,0 @@
|
|||
-- 14_fleet_segment_and_vehicles_view.sql
|
||||
-- Fleet segmentation + de-duplicated vehicle roster.
|
||||
--
|
||||
-- Splits the fleet into ticket-closing FIELD SERVICE vehicles vs SPECIALIST plant
|
||||
-- (cranes / pick-ups / motorbikes) that do NOT close immediate customer tickets.
|
||||
--
|
||||
-- The segment is DERIVED, not stored: it is computed from tracksolid.devices.vehicle_models,
|
||||
-- which is itself an authoritative Tracksolid API field (sync_devices() maps
|
||||
-- jimi.user.device.list -> vehicleModels, refreshed daily). Keeping it derived means it
|
||||
-- always tracks the API and needs no re-seeding. The manual tracksolid.vehicle_category
|
||||
-- column is intentionally NOT used here.
|
||||
--
|
||||
-- reporting.v_vehicles collapses the GPS-tracker + dashcam device pairs into one row per
|
||||
-- vehicle, reusing reporting.normalize_plate() and the same "primary device per normalized
|
||||
-- plate" precedence as reporting.v_trips / reporting.v_live_positions (migration 11). This
|
||||
-- auto-merges plate-spacing duplicates (e.g. 'KDS 453Y' vs 'KDS 453 Y') and resolves any
|
||||
-- within-plate model disagreement by letting the primary tracker's value win.
|
||||
--
|
||||
-- Every object uses CREATE OR REPLACE / guarded grants so the file is safe to re-apply.
|
||||
-- Provenance: docs/reports/260608_fleet_registry_data_quality.md + ~/.claude plan binary-singing-wave.
|
||||
|
||||
SET search_path = tracksolid, reporting, public;
|
||||
|
||||
-- ── classification rule (single source of truth) ─────────────────────────────
|
||||
CREATE OR REPLACE FUNCTION reporting.fn_fleet_segment(model text)
|
||||
RETURNS text
|
||||
LANGUAGE sql
|
||||
IMMUTABLE PARALLEL SAFE
|
||||
AS $function$
|
||||
SELECT CASE lower(coalesce(trim(model), ''))
|
||||
WHEN '' THEN 'unassigned' -- no model on record -> triage
|
||||
WHEN 'crane' THEN 'specialist'
|
||||
WHEN 'pick-up' THEN 'specialist'
|
||||
WHEN 'pickup' THEN 'specialist'
|
||||
WHEN 'truck' THEN 'specialist'
|
||||
WHEN 'motorbike' THEN 'specialist'
|
||||
ELSE 'field_service' -- Probox, Mazda, Van, Station Wagon, Vezel + any other named model
|
||||
END
|
||||
$function$;
|
||||
|
||||
COMMENT ON FUNCTION reporting.fn_fleet_segment(text) IS
|
||||
'Maps tracksolid.devices.vehicle_models -> field_service | specialist | unassigned. '
|
||||
'Specialist = crane/pick-up/motorbike/truck (do not close immediate customer tickets).';
|
||||
|
||||
-- ── de-duplicated vehicle roster (one row per physical vehicle) ───────────────
|
||||
CREATE OR REPLACE VIEW reporting.v_vehicles AS
|
||||
WITH device_trip_counts AS (
|
||||
SELECT trips.imei, count(*) AS trip_count
|
||||
FROM trips
|
||||
GROUP BY trips.imei
|
||||
), primary_device AS (
|
||||
SELECT DISTINCT ON ((reporting.normalize_plate(d.vehicle_number)))
|
||||
reporting.normalize_plate(d.vehicle_number) AS plate,
|
||||
d.imei AS primary_imei,
|
||||
d.vehicle_models,
|
||||
d.driver_name,
|
||||
d.driver_phone,
|
||||
d.account,
|
||||
d.assigned_city
|
||||
FROM devices d
|
||||
LEFT JOIN device_trip_counts c USING (imei)
|
||||
WHERE d.vehicle_number IS NOT NULL AND d.enabled_flag = 1
|
||||
ORDER BY (reporting.normalize_plate(d.vehicle_number)),
|
||||
(CASE WHEN d.mc_type = ANY (ARRAY['GT06E','X3','AT4']) THEN 0 ELSE 1 END),
|
||||
(COALESCE(c.trip_count, 0::bigint)) DESC,
|
||||
d.activation_time,
|
||||
d.imei
|
||||
), plate_agg AS (
|
||||
SELECT reporting.normalize_plate(d.vehicle_number) AS plate,
|
||||
bool_or(d.mc_type = ANY (ARRAY['GT06E','X3','AT4'])) AS has_tracker,
|
||||
bool_or(d.mc_type = 'JC400P') AS has_camera,
|
||||
count(*) AS device_count
|
||||
FROM devices d
|
||||
WHERE d.vehicle_number IS NOT NULL AND d.enabled_flag = 1
|
||||
GROUP BY reporting.normalize_plate(d.vehicle_number)
|
||||
)
|
||||
SELECT pd.plate,
|
||||
pd.vehicle_models AS vehicle_type,
|
||||
reporting.fn_fleet_segment(pd.vehicle_models) AS fleet_segment,
|
||||
pd.driver_name AS driver,
|
||||
pd.driver_phone,
|
||||
pd.account,
|
||||
pd.assigned_city,
|
||||
pa.has_tracker,
|
||||
pa.has_camera,
|
||||
pa.device_count,
|
||||
pd.primary_imei
|
||||
FROM primary_device pd
|
||||
JOIN plate_agg pa USING (plate);
|
||||
|
||||
COMMENT ON VIEW reporting.v_vehicles IS
|
||||
'One row per physical vehicle (tracker+dashcam pairs collapsed by normalize_plate, primary '
|
||||
'device = tracker-first then trip-count). fleet_segment derived from API-authoritative '
|
||||
'vehicle_models. Source: docs/reports/260608_fleet_registry_data_quality.md.';
|
||||
|
||||
-- ── grants (guarded: roles may not exist on a fresh DB) ───────────────────────
|
||||
DO $grants$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'grafana_ro') THEN
|
||||
GRANT USAGE ON SCHEMA reporting TO grafana_ro;
|
||||
GRANT EXECUTE ON FUNCTION reporting.fn_fleet_segment(text) TO grafana_ro;
|
||||
GRANT SELECT ON reporting.v_vehicles TO grafana_ro;
|
||||
END IF;
|
||||
END $grants$;
|
||||
|
|
@ -1,105 +0,0 @@
|
|||
-- 15_map_exclude_cost_centres.sql
|
||||
-- Hide non-operational vehicles from the LIVE tracking map (FleetNow + liveposition SPA).
|
||||
--
|
||||
-- A small, ops-editable config table lists the cost centres to exclude. reporting.v_live_positions
|
||||
-- (the base view behind reporting.fn_live_positions, which dashboard_api serves) filters out any
|
||||
-- plate whose device(s) carry an excluded cost centre. Editing the table changes the map on the
|
||||
-- next query — no code change, no redeploy.
|
||||
--
|
||||
-- Scope: LIVE map only. Trip history (reporting.v_trips materialised view) is deliberately NOT
|
||||
-- touched. Initial exclusions: personal + management (staff/personal cars) and mtn (the MTN
|
||||
-- contract / Uganda-Kampala fleet, outside Kenyan ops).
|
||||
--
|
||||
-- The v_live_positions body below is reproduced verbatim from the live prod definition
|
||||
-- (== migrations/11_reporting_schema.sql) with a single added filter in the primary_device CTE.
|
||||
-- Safe to re-apply (CREATE TABLE IF NOT EXISTS / INSERT ON CONFLICT / CREATE OR REPLACE VIEW).
|
||||
|
||||
SET search_path = tracksolid, reporting, public;
|
||||
|
||||
-- ── exclusion config (data-driven, editable without a migration) ──────────────
|
||||
CREATE TABLE IF NOT EXISTS reporting.map_excluded_cost_centres (
|
||||
cost_centre text PRIMARY KEY, -- compared case-insensitively (store lowercase)
|
||||
note text,
|
||||
added_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
INSERT INTO reporting.map_excluded_cost_centres (cost_centre, note) VALUES
|
||||
('personal', 'staff/personal vehicles — not operational fleet'),
|
||||
('management', 'management vehicles — not operational fleet'),
|
||||
('mtn', 'MTN contract / Uganda (Kampala) — outside Kenyan ops')
|
||||
ON CONFLICT (cost_centre) DO NOTHING;
|
||||
|
||||
-- ── v_live_positions: same definition + exclusion filter ──────────────────────
|
||||
CREATE OR REPLACE VIEW reporting.v_live_positions AS
|
||||
WITH primary_device AS (
|
||||
SELECT DISTINCT ON ((reporting.normalize_plate(d_1.vehicle_number))) reporting.normalize_plate(d_1.vehicle_number) AS vehicle_number,
|
||||
d_1.imei AS primary_imei
|
||||
FROM devices d_1
|
||||
LEFT JOIN live_positions lp_1 ON lp_1.imei = d_1.imei
|
||||
WHERE d_1.vehicle_number IS NOT NULL AND d_1.enabled_flag = 1
|
||||
-- exclude plates whose device(s) carry a non-operational cost centre
|
||||
AND reporting.normalize_plate(d_1.vehicle_number) NOT IN (
|
||||
SELECT reporting.normalize_plate(x.vehicle_number)
|
||||
FROM devices x
|
||||
WHERE x.vehicle_number IS NOT NULL
|
||||
AND lower(trim(x.cost_centre)) IN (
|
||||
SELECT cost_centre FROM reporting.map_excluded_cost_centres)
|
||||
)
|
||||
ORDER BY (reporting.normalize_plate(d_1.vehicle_number)), (
|
||||
CASE
|
||||
WHEN (d_1.mc_type = ANY (ARRAY['GT06E'::text, 'X3'::text, 'AT4'::text])) AND lp_1.gps_time >= (now() - '24:00:00'::interval) THEN 0
|
||||
ELSE 1
|
||||
END), lp_1.gps_time DESC NULLS LAST, (
|
||||
CASE d_1.mc_type
|
||||
WHEN 'GT06E'::text THEN 1
|
||||
WHEN 'X3'::text THEN 2
|
||||
WHEN 'AT4'::text THEN 3
|
||||
WHEN 'JC400P'::text THEN 4
|
||||
ELSE 5
|
||||
END), d_1.activation_time, d_1.imei
|
||||
)
|
||||
SELECT lp.imei,
|
||||
pd.vehicle_number,
|
||||
d.driver_name AS assigned_driver,
|
||||
d.cost_centre,
|
||||
d.assigned_city,
|
||||
d.vehicle_category,
|
||||
d.vehicle_models,
|
||||
d.mc_type,
|
||||
CASE d.mc_type
|
||||
WHEN 'GT06E'::text THEN 'tracker'::text
|
||||
WHEN 'X3'::text THEN 'tracker'::text
|
||||
WHEN 'AT4'::text THEN 'tracker'::text
|
||||
WHEN 'JC400P'::text THEN 'camera'::text
|
||||
ELSE 'other'::text
|
||||
END AS device_kind,
|
||||
lp.lat,
|
||||
lp.lng,
|
||||
lp.speed,
|
||||
lp.direction,
|
||||
lp.acc_status,
|
||||
lp.device_status,
|
||||
lp.gps_signal,
|
||||
lp.gps_num,
|
||||
lp.current_mileage,
|
||||
lp.loc_desc,
|
||||
lp.gps_time,
|
||||
lp.updated_at,
|
||||
(lp.gps_time AT TIME ZONE 'Africa/Nairobi'::text) AS gps_time_eat,
|
||||
(lp.updated_at AT TIME ZONE 'Africa/Nairobi'::text) AS updated_at_eat,
|
||||
round(EXTRACT(epoch FROM now() - lp.gps_time) / 3600::numeric, 2) AS source_age_hours
|
||||
FROM live_positions lp
|
||||
JOIN primary_device pd ON pd.primary_imei = lp.imei
|
||||
JOIN devices d ON d.imei = lp.imei;
|
||||
|
||||
COMMENT ON TABLE reporting.map_excluded_cost_centres IS
|
||||
'Cost centres hidden from the live map (reporting.v_live_positions). Edit to hide/restore; '
|
||||
'effective on next query. Seeded: personal, management, mtn. See migration 15.';
|
||||
|
||||
-- ── grants (guarded: roles may not exist on a fresh DB) ───────────────────────
|
||||
DO $grants$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'grafana_ro') THEN
|
||||
GRANT SELECT ON reporting.map_excluded_cost_centres TO grafana_ro;
|
||||
END IF;
|
||||
END $grants$;
|
||||
|
|
@ -1,99 +0,0 @@
|
|||
-- 16_live_feed_vehicle_type.sql
|
||||
-- Expose vehicle_type + fleet_segment on the live-map GeoJSON feed so FleetNow can give the
|
||||
-- specialist vehicles (Crane / Motorbike / Pick-Up) their own marker icons. All other vehicles
|
||||
-- (field-service + unassigned) keep their current marker — FleetNow only overrides icons when
|
||||
-- vehicle_type is one of the specialist types.
|
||||
--
|
||||
-- reporting.fn_live_positions is reproduced verbatim from the live prod definition
|
||||
-- (== migrations/11_reporting_schema.sql, captured via pg_get_functiondef) with TWO added
|
||||
-- feature properties:
|
||||
-- 'vehicle_type' = devices.vehicle_models (authoritative API type, surfaced by v_live_positions)
|
||||
-- 'fleet_segment' = reporting.fn_fleet_segment(vehicle_models) (field_service|specialist|unassigned)
|
||||
-- No signature change, so dependents are unaffected; STABLE function, read immediately by
|
||||
-- dashboard_api (no redeploy/restart). Safe to re-apply (CREATE OR REPLACE).
|
||||
|
||||
SET search_path = tracksolid, reporting, public;
|
||||
|
||||
CREATE OR REPLACE FUNCTION reporting.fn_live_positions(p_cost_centre text DEFAULT NULL::text, p_acc_status text DEFAULT NULL::text)
|
||||
RETURNS jsonb
|
||||
LANGUAGE plpgsql
|
||||
STABLE
|
||||
AS $function$
|
||||
DECLARE
|
||||
v_result jsonb;
|
||||
BEGIN
|
||||
p_cost_centre := NULLIF(p_cost_centre, '');
|
||||
p_acc_status := NULLIF(p_acc_status, '');
|
||||
WITH filtered AS (
|
||||
SELECT * FROM reporting.v_live_positions
|
||||
WHERE (p_cost_centre IS NULL OR cost_centre = p_cost_centre)
|
||||
AND (p_acc_status IS NULL OR acc_status = p_acc_status)
|
||||
)
|
||||
SELECT jsonb_build_object(
|
||||
'summary', jsonb_build_object(
|
||||
'vehicle_count', COUNT(*),
|
||||
-- "moving" and "parked" both restrict to devices that have reported
|
||||
-- within the OFFLINE_THRESHOLD (24 h) so they represent the live
|
||||
-- fleet, not equipment-failure stragglers. "offline" is its own
|
||||
-- counter for the > 24 h tail.
|
||||
'moving', COUNT(*) FILTER (WHERE acc_status = '1'
|
||||
AND source_age_hours < 24),
|
||||
'parked', COUNT(*) FILTER (WHERE acc_status = '0'
|
||||
AND source_age_hours < 24),
|
||||
'offline', COUNT(*) FILTER (WHERE source_age_hours >= 24),
|
||||
'median_speed_moving', percentile_cont(0.5) WITHIN GROUP (ORDER BY speed)
|
||||
FILTER (WHERE acc_status = '1'
|
||||
AND source_age_hours < 24
|
||||
AND speed > 0),
|
||||
'last_batch_at', to_char(MAX(updated_at) AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS'),
|
||||
'oldest_fix_at', to_char(MIN(gps_time) AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS'),
|
||||
'newest_fix_at', to_char(MAX(gps_time) AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS'),
|
||||
'last_batch_utc', MAX(updated_at),
|
||||
'newest_fix_utc', MAX(gps_time)
|
||||
),
|
||||
'geojson', jsonb_build_object(
|
||||
'type', 'FeatureCollection',
|
||||
'features', COALESCE(jsonb_agg(
|
||||
jsonb_build_object(
|
||||
'type', 'Feature',
|
||||
'properties', jsonb_build_object(
|
||||
'imei', imei,
|
||||
'vehicle_number', vehicle_number,
|
||||
'driver', assigned_driver,
|
||||
'cost_centre', cost_centre,
|
||||
'assigned_city', assigned_city,
|
||||
'vehicle_category', vehicle_category,
|
||||
'vehicle_type', vehicle_models,
|
||||
'fleet_segment', reporting.fn_fleet_segment(vehicle_models),
|
||||
'mc_type', mc_type,
|
||||
'device_kind', device_kind,
|
||||
'source_age_hours', source_age_hours,
|
||||
'speed', speed,
|
||||
'direction', direction,
|
||||
'acc_status', acc_status,
|
||||
'device_status', device_status,
|
||||
'gps_signal', gps_signal,
|
||||
'gps_num', gps_num,
|
||||
'current_mileage', current_mileage,
|
||||
'loc_desc', loc_desc,
|
||||
'gps_time', to_char(gps_time AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS'),
|
||||
'updated_at', to_char(updated_at AT TIME ZONE 'Africa/Nairobi',
|
||||
'YYYY-MM-DD HH24:MI:SS'),
|
||||
'gps_time_utc', gps_time,
|
||||
'updated_at_utc', updated_at
|
||||
),
|
||||
'geometry', jsonb_build_object(
|
||||
'type', 'Point',
|
||||
'coordinates', jsonb_build_array(lng, lat)
|
||||
)
|
||||
)
|
||||
), '[]'::jsonb)
|
||||
)
|
||||
) INTO v_result FROM filtered;
|
||||
|
||||
RETURN v_result;
|
||||
END $function$;
|
||||
0
push_webhook.md
Normal file
0
push_webhook.md
Normal file
|
|
@ -34,12 +34,6 @@ MIGRATIONS = [
|
|||
"08_analytics_config.sql", # ops.cost_rates, ops.kpi_targets + seed data
|
||||
"09_trips_enrichment.sql", # trips.route_geom + addresses + plate + v_trips_enriched
|
||||
"10_pgbouncer_auth.sql", # pgbouncer role + user_lookup() for SCRAM passthrough
|
||||
"11_reporting_schema.sql", # reporting.* map-dashboard read layer (dashboard_api)
|
||||
"12_drop_ops.sql", # purge dormant ops schema + dispatch_log + v_sla_inflight
|
||||
"13_drop_dwh_gold.sql", # purge dormant dwh_gold schema + v_utilisation_daily
|
||||
"14_fleet_segment_and_vehicles_view.sql", # reporting.fn_fleet_segment + reporting.v_vehicles roster
|
||||
"15_map_exclude_cost_centres.sql", # hide personal/management/mtn vehicles from the live map
|
||||
"16_live_feed_vehicle_type.sql", # add vehicle_type + fleet_segment to fn_live_positions feed
|
||||
]
|
||||
|
||||
# ── Tables that must exist before the service is allowed to start ─────────────
|
||||
|
|
@ -227,7 +221,7 @@ def main():
|
|||
|
||||
applied = skipped = 0
|
||||
for sql_file in MIGRATIONS:
|
||||
path = os.path.join("/app", "migrations", sql_file)
|
||||
path = os.path.join("/app", sql_file)
|
||||
|
||||
if not os.path.exists(path):
|
||||
print(f" SKIP {sql_file} (file not found in /app)")
|
||||
|
|
|
|||
|
|
@ -55,10 +55,10 @@ run_sql -c "
|
|||
" > /dev/null
|
||||
|
||||
# ── Find and apply pending migrations ────────────────────────────────────────
|
||||
MIGRATION_FILES=$(find "$SCRIPT_DIR/migrations" -maxdepth 1 -name '[0-9][0-9]_*.sql' | sort)
|
||||
MIGRATION_FILES=$(find "$SCRIPT_DIR" -maxdepth 1 -name '[0-9][0-9]_*.sql' | sort)
|
||||
|
||||
if [[ -z "$MIGRATION_FILES" ]]; then
|
||||
echo "No migration files found in $SCRIPT_DIR/migrations"
|
||||
echo "No migration files found in $SCRIPT_DIR"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
|
|
|
|||
|
|
@ -1,88 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Export OSM POIs (e.g. fuel stations) from a .osm.pbf to GeoJSON + CSV.
|
||||
|
||||
These exports feed FleetNow's toggleable map-overlay layers (see
|
||||
docs/OSM_POI_EXPORT.md and the fleetnow repo's README "Map overlay layers").
|
||||
|
||||
No system tooling needed — run via uv so pyosmium's prebuilt wheel is fetched:
|
||||
|
||||
uv run --no-project --with osmium python scripts/export_osm_pois.py \
|
||||
kenya-260605.osm.pbf --amenity fuel --brand Shell \
|
||||
--out-geojson shell_stations.geojson --out-csv shell_stations.csv
|
||||
|
||||
Notes:
|
||||
- Gas stations are OSM ``amenity=fuel``. Brand lives in the ``brand`` tag, but
|
||||
only ~36% of Kenyan stations carry it, so when ``--brand`` is given and a
|
||||
feature has no ``brand`` tag we fall back to matching ``name``/``operator``.
|
||||
- Omit ``--brand`` to export every feature of that amenity.
|
||||
- Nodes use their own coordinate; ways/areas use the centroid of their nodes
|
||||
(so ``locations=True`` is required on apply_file).
|
||||
"""
|
||||
import argparse, json, csv
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("pbf", help="input .osm.pbf")
|
||||
ap.add_argument("--amenity", default="fuel", help="OSM amenity value (default: fuel)")
|
||||
ap.add_argument("--brand", default=None, help="case-insensitive brand/name match; omit for all")
|
||||
ap.add_argument("--out-geojson", default="pois.geojson")
|
||||
ap.add_argument("--out-csv", default="pois.csv")
|
||||
args = ap.parse_args()
|
||||
|
||||
import osmium
|
||||
|
||||
brand_lc = args.brand.lower() if args.brand else None
|
||||
|
||||
def match(t):
|
||||
if t.get("amenity") != args.amenity:
|
||||
return False
|
||||
if brand_lc is None:
|
||||
return True
|
||||
b = (t.get("brand") or "").lower()
|
||||
if b:
|
||||
return brand_lc in b
|
||||
return brand_lc in (t.get("name") or "").lower() or brand_lc in (t.get("operator") or "").lower()
|
||||
|
||||
feats = []
|
||||
|
||||
def add(lon, lat, t, kind, oid):
|
||||
feats.append({
|
||||
"type": "Feature",
|
||||
"properties": {
|
||||
"name": t.get("name"), "brand": t.get("brand"),
|
||||
"operator": t.get("operator"), "osm_type": kind, "osm_id": oid,
|
||||
},
|
||||
"geometry": {"type": "Point", "coordinates": [round(lon, 6), round(lat, 6)]},
|
||||
})
|
||||
|
||||
class H(osmium.SimpleHandler):
|
||||
def node(self, o):
|
||||
if match(o.tags):
|
||||
add(o.location.lon, o.location.lat, o.tags, "node", o.id)
|
||||
def way(self, o):
|
||||
if match(o.tags):
|
||||
xs = []; ys = []
|
||||
for n in o.nodes:
|
||||
try:
|
||||
if n.location.valid():
|
||||
xs.append(n.location.lon); ys.append(n.location.lat)
|
||||
except Exception:
|
||||
pass
|
||||
if xs:
|
||||
add(sum(xs) / len(xs), sum(ys) / len(ys), o.tags, "way", o.id)
|
||||
|
||||
H().apply_file(args.pbf, locations=True)
|
||||
|
||||
json.dump({"type": "FeatureCollection", "features": feats}, open(args.out_geojson, "w"))
|
||||
with open(args.out_csv, "w", newline="") as f:
|
||||
w = csv.writer(f); w.writerow(["name", "lat", "lon", "brand", "operator", "osm_type", "osm_id"])
|
||||
for ft in feats:
|
||||
p = ft["properties"]; lon, lat = ft["geometry"]["coordinates"]
|
||||
w.writerow([p["name"], lat, lon, p["brand"], p["operator"], p["osm_type"], p["osm_id"]])
|
||||
|
||||
print(f"exported {len(feats)} features -> {args.out_geojson}, {args.out_csv}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -1,233 +0,0 @@
|
|||
name,lat,lon,brand,operator,brand_source,osm_type,osm_id
|
||||
Shell Kabuagi,-1.316594,36.718419,Shell,,brand_tag,node,30088444
|
||||
Shell,-1.316337,36.814074,Shell,,brand_tag,node,30092268
|
||||
Shell,-1.292004,36.784895,Shell,,brand_tag,node,30210342
|
||||
Shell,-1.292514,36.821731,Shell,,brand_tag,node,30215041
|
||||
Shell,-1.264944,36.811142,Shell,,brand_tag,node,30402252
|
||||
Shell Bellevue,-1.321222,36.841704,Shell,,brand_tag,node,30695844
|
||||
Shell,-1.259212,36.818551,,,name,node,295829498
|
||||
Shell,-3.400363,38.549144,Shell,,brand_tag,node,344711386
|
||||
Shell,-4.064245,39.671241,Shell,,brand_tag,node,344969183
|
||||
Shell,-4.04045,39.678266,Shell,,brand_tag,node,344970554
|
||||
Shell,-4.035198,39.684952,Shell,,brand_tag,node,344971084
|
||||
Shell Links Road Filling Station,-4.051857,39.688222,Shell,,brand_tag,node,346562706
|
||||
Shell,-0.71861,36.432908,Shell,,brand_tag,node,415582755
|
||||
Shell,-1.307743,36.803669,Shell,,brand_tag,node,490741522
|
||||
Shell Petrol Station,-1.288024,36.741743,Shell,,brand_tag,node,558923029
|
||||
Shell,-1.310587,36.817074,Shell,,brand_tag,node,582803761
|
||||
Shell,-1.305716,36.827261,Shell,,brand_tag,node,582803763
|
||||
Shell,-1.520877,37.267142,Shell,,brand_tag,node,582803840
|
||||
Shell Makupa,-4.041392,39.656827,Shell,Shell,brand_tag,node,612947282
|
||||
Shell Changamwe,-4.027279,39.62732,Shell,Shell,brand_tag,node,612947302
|
||||
Shell,-1.249864,36.861223,Shell,,brand_tag,node,703562830
|
||||
Shell,-1.246981,36.871498,Shell,,brand_tag,node,703562919
|
||||
Shell - Ruiru,-1.149099,36.958483,Shell,,brand_tag,node,711491743
|
||||
Shell,-1.293773,36.886453,Shell,Shell,brand_tag,node,712868923
|
||||
Shell,-1.256982,36.922912,Shell,,brand_tag,node,734051667
|
||||
Shell,-1.230839,36.80343,Shell,Shell,brand_tag,node,812840504
|
||||
Shell,-1.232602,36.875076,Shell,,brand_tag,node,900302335
|
||||
Shell Avenue Service Station,-1.285422,36.818863,Shell,,brand_tag,node,911026324
|
||||
Shell Adams Arcade,-1.300601,36.779709,Shell,,brand_tag,node,1045731889
|
||||
Shell,-1.267364,36.81745,Shell,,brand_tag,node,1147234418
|
||||
Shell,-4.085069,39.654867,Shell,Shell,brand_tag,node,1320763086
|
||||
Shell,0.181801,34.295817,Shell,Indivdual,brand_tag,node,1419926800
|
||||
Shell,-1.265026,36.740926,Shell,,brand_tag,node,1506606959
|
||||
Shell,-0.284215,36.068872,Shell,,brand_tag,node,1636429762
|
||||
Shell,-1.31465,36.859873,Shell,Shell/BP,brand_tag,node,1678061764
|
||||
Shell,-1.090556,35.871981,Shell,,brand_tag,node,1690703696
|
||||
Shell,0.03559,37.655216,Shell,,brand_tag,node,1814694405
|
||||
Shell,0.05734,37.642474,Shell,,brand_tag,node,1814694430
|
||||
Shell,0.016696,37.077472,Shell,,brand_tag,node,1830086395
|
||||
Shell,-0.363566,35.293414,Shell,,brand_tag,node,1838955746
|
||||
Shell,0.634907,34.280633,Shell,,brand_tag,node,1856237414
|
||||
Shell Bidii,-1.279267,36.848606,Shell,,brand_tag,node,1997615249
|
||||
Shell,-2.693933,38.166679,Shell,,brand_tag,node,2137912798
|
||||
Shell Baraka,-1.259949,36.785711,Shell,,brand_tag,node,2218541216
|
||||
Shell,-4.047937,39.662737,Shell,,brand_tag,node,2377336397
|
||||
Shell,2.32971,37.988219,Shell,,brand_tag,node,2462468244
|
||||
Shell,-1.281187,36.816595,Shell,,brand_tag,node,2533881587
|
||||
Shell,-1.446789,36.968782,Shell,Shell,brand_tag,node,2653484553
|
||||
Shell Roysambu,-1.217787,36.890669,Shell,,brand_tag,node,2953872784
|
||||
Shell petrol station,-0.424642,36.952432,,2NK,name,node,3078763165
|
||||
Shell petrol station,-0.421785,36.952518,,,name,node,3078776472
|
||||
Shell,-0.334861,37.646147,Shell,,brand_tag,node,3243527416
|
||||
Isiolo Service Station,0.3519,37.582829,Shell,,brand_tag,node,3247766463
|
||||
Shell,-0.500952,36.314893,Shell,,brand_tag,node,3421681816
|
||||
Bajoo Shell Services,1.749142,40.057779,Shell,,brand_tag,node,3683457938
|
||||
Buna Filling Station,2.786999,39.509127,Shell,,brand_tag,node,3695322660
|
||||
Shell,-1.395443,36.939718,Shell,,brand_tag,node,3729203207
|
||||
Shell,-1.10888,36.641976,Shell,,brand_tag,node,3808532767
|
||||
Shell,-1.299065,36.763624,Shell,,brand_tag,node,4210622090
|
||||
Shell,-1.245784,36.662797,Shell,,brand_tag,node,4324873067
|
||||
Shell,0.337182,37.578977,Shell,,brand_tag,node,4461211795
|
||||
Shell,-3.405215,38.362976,Shell,,brand_tag,node,4475364889
|
||||
Shell,-0.454161,39.645824,Shell,,brand_tag,node,4685351383
|
||||
Shell,-0.625556,34.75581,Shell,,brand_tag,node,4720565411
|
||||
Shell,-0.697715,36.427394,Shell,,brand_tag,node,4914118412
|
||||
"Shell Petrol Station, Muguga",-1.061481,37.157952,,,name,node,4946660224
|
||||
"Shell New Thika Rd, Ruiru",-1.160387,36.958055,Shell,,brand_tag,node,4947103423
|
||||
Shell,-1.226429,36.663491,Shell,,brand_tag,node,4972148621
|
||||
Shell,-1.229222,36.840393,Shell,,brand_tag,node,5119552023
|
||||
Shell,-1.264064,36.838234,Shell,,brand_tag,node,5119573528
|
||||
Shell,-0.977475,37.095977,Shell,,brand_tag,node,5126219526
|
||||
Shell,-0.929697,37.160111,Shell,,brand_tag,node,5126247329
|
||||
Shell,-1.011913,36.903025,Shell,,brand_tag,node,5162725223
|
||||
Shell Petrol Station Kajiado,-1.835716,36.799667,Shell,Shell,brand_tag,node,5179571022
|
||||
Shell,-1.180887,37.440173,Shell,,brand_tag,node,5181243643
|
||||
Shell,-1.280736,36.827944,Shell,,brand_tag,node,5217418921
|
||||
Shell,-1.171399,36.828308,Shell,,brand_tag,node,5327749961
|
||||
Shell,-1.262778,36.801048,Shell,,brand_tag,node,5340449107
|
||||
Shell,0.540157,35.296916,Shell,,brand_tag,node,5392739324
|
||||
Shell,0.448948,35.967056,,,name,node,5403105422
|
||||
Shell,-1.292007,36.84313,Shell,,brand_tag,node,5496066065
|
||||
Shell,-1.295157,36.854441,Shell,,brand_tag,node,5496079228
|
||||
Shell Kasarani Petrol Station,-1.218327,36.895867,,,name,node,5526332569
|
||||
Shell,-1.30385,36.856284,Shell,,brand_tag,node,5539579719
|
||||
Shell,-1.195935,36.751887,Shell,,brand_tag,node,5540791550
|
||||
Shell,-1.3074,36.842468,Shell,,brand_tag,node,5569591300
|
||||
Shell Banana,-1.175701,36.759722,Shell,,brand_tag,node,5581701614
|
||||
Shell Kiserian,-1.432004,36.686175,Shell,Vivo,brand_tag,node,5588931919
|
||||
Mwingi Shell Petrol Station,-0.937425,38.044514,Shell,Shell,brand_tag,node,5768445294
|
||||
Mart Petrol Station,0.494496,35.746217,Shell,,brand_tag,node,5781632986
|
||||
Shell,-1.263474,36.981924,Shell,,brand_tag,node,5887135685
|
||||
Shell,-4.060892,39.662713,Shell,,brand_tag,node,6054282205
|
||||
Shell,-4.043209,39.663984,Shell,,brand_tag,node,6054858991
|
||||
Shell,-0.984791,36.584131,Shell,,brand_tag,node,6054861952
|
||||
Shell,-1.089833,35.882185,Shell,,brand_tag,node,6054877097
|
||||
Shell,-0.769528,36.501316,Shell,,brand_tag,node,6054980226
|
||||
Shell,-1.057588,36.776239,,,name,node,6063644317
|
||||
Shell Filling station,-4.084926,39.654747,Shell,,brand_tag,node,6072960948
|
||||
Shell,-1.456045,37.004306,Shell,,brand_tag,node,6086991186
|
||||
Shell,-0.780234,34.948265,Shell,,brand_tag,node,6105930796
|
||||
Shell,1.01741,35.002629,Shell,,brand_tag,node,6106202680
|
||||
Shell,-0.929519,37.159777,Shell,,brand_tag,node,6140765579
|
||||
Shell,-0.074245,37.670579,Shell,,brand_tag,node,6144610331
|
||||
Shell,0.524964,35.251383,Shell,,brand_tag,node,6147580921
|
||||
Shell,0.489345,35.269825,Shell,,brand_tag,node,6149443773
|
||||
Shell Petrol Station,0.465501,35.297389,,,name,node,6149634448
|
||||
Shell,-1.248207,36.876,Shell,,brand_tag,node,6198796724
|
||||
Shell,-1.043083,37.066139,Shell,,brand_tag,node,6207691351
|
||||
Shell,-1.042001,37.071543,Shell,,brand_tag,node,6207707073
|
||||
Shell,-1.037044,37.071925,Shell,,brand_tag,node,6209964953
|
||||
Shell,-0.841126,37.137857,,,name,node,6212995383
|
||||
Shell,-0.485965,37.138349,Shell,,brand_tag,node,6213405879
|
||||
Shell,-1.347761,36.662815,Shell,,brand_tag,node,6216321668
|
||||
Shell Petrol Station,-1.416823,36.686915,,,name,node,6218487553
|
||||
Shell,-1.396852,36.755199,Shell,,brand_tag,node,6218698870
|
||||
PETRO Shelly,-4.085731,39.667207,,,name,node,6222293185
|
||||
Shell Maanzoni,-1.516527,37.107005,Shell,,brand_tag,node,6226894926
|
||||
Shell,-1.532784,37.131763,Shell,,brand_tag,node,6229374240
|
||||
Shell Kitengela,-1.50442,36.954102,,,name,node,6229505975
|
||||
shell,-1.735489,37.198837,Shell,,brand_tag,node,6229624406
|
||||
Shell,-2.081484,37.477328,Shell,,brand_tag,node,6232739137
|
||||
Shell,-3.973832,39.547043,Shell,,brand_tag,node,6235829860
|
||||
Shell,-4.007792,39.60037,Shell,,brand_tag,node,6235830590
|
||||
Shell,0.580828,34.557734,Shell,,brand_tag,node,6236062090
|
||||
Shell,-1.365272,38.012171,Shell,,brand_tag,node,6236068237
|
||||
Shell Filling Station,0.035,36.364412,,,name,node,6241664256
|
||||
Shell Chaka,-0.361651,36.999992,Shell,,brand_tag,node,6244921195
|
||||
Shell,0.005805,37.072986,Shell,,brand_tag,node,6246803075
|
||||
Shell,-1.260591,36.710164,Shell,,brand_tag,node,6251561800
|
||||
Shell service station Fedha,-1.316073,36.895595,Shell,,brand_tag,node,6259798932
|
||||
Shell Service Station - Kayole,-1.2829,36.90304,Shell,,brand_tag,node,6259810690
|
||||
Shell Mumias Road,-1.286405,36.88066,Shell,,brand_tag,node,6259860371
|
||||
Shell,3.094471,35.614012,Shell,,brand_tag,node,6263511612
|
||||
Shell,0.509984,35.293509,Shell,Shell Petrol Station,brand_tag,node,6327543347
|
||||
Shell,-0.144338,34.8022,Shell,,brand_tag,node,6633204913
|
||||
Shell,-0.779415,36.426665,Shell,,brand_tag,node,6644488886
|
||||
Shell,-3.578128,39.87102,Shell,,brand_tag,node,6716512885
|
||||
Shell,-3.214362,40.117773,Shell,,brand_tag,node,6793044096
|
||||
Shell,-1.273195,36.911201,Shell,,brand_tag,node,6819692388
|
||||
Shell Petrol Station,-1.055007,37.111221,Shell,,brand_tag,node,6851086245
|
||||
Shell,-0.542158,37.454875,Shell,,brand_tag,node,6860253187
|
||||
Shell,0.06024,37.636552,Shell,,brand_tag,node,6882011852
|
||||
Shell,-0.71496,37.261577,Shell,,brand_tag,node,6895782975
|
||||
shell- Bonje,-3.991998,39.562412,Shell,,brand_tag,node,6938700260
|
||||
Shell,0.453207,34.130811,Shell,,brand_tag,node,6945529552
|
||||
Shell,-0.130428,34.794978,Shell,,brand_tag,node,6969176798
|
||||
Shell,-3.211576,40.121388,Shell,,brand_tag,node,7166621203
|
||||
Shell,-3.39087,38.581827,Shell,,brand_tag,node,7166621204
|
||||
shell makupa service station,-4.040825,39.656857,Shell,,brand_tag,node,7172031316
|
||||
Shell,-0.322371,36.15036,Shell,,brand_tag,node,7187879248
|
||||
shell,-0.074322,34.682137,Shell,,brand_tag,node,7227297576
|
||||
Shell,0.28749,34.756277,Shell,,brand_tag,node,7239126633
|
||||
Shell,-1.323004,36.706266,Shell,,brand_tag,node,7305927852
|
||||
Shell- Syokimau,-1.357374,36.907769,Shell,,brand_tag,node,7507287888
|
||||
Shell,-1.210901,36.875751,Shell,Shell,brand_tag,node,7643261185
|
||||
Shell,-0.973363,37.095865,Shell,,brand_tag,node,7872591802
|
||||
Shell,-0.366585,35.284074,Shell,,brand_tag,node,7893551877
|
||||
Shell,0.87809,35.120238,Shell,,brand_tag,node,7934573084
|
||||
Shell,-1.089807,35.882577,Shell,,brand_tag,node,7988261701
|
||||
Shell,-0.541871,37.455485,Shell,,brand_tag,node,8039934769
|
||||
Shell,-1.358313,38.007156,Shell,,brand_tag,node,8047307634
|
||||
Shell,-3.92977,39.53407,Shell,,brand_tag,node,8050570392
|
||||
Shell,-0.264808,36.377656,,,name,node,8182514303
|
||||
Shell Ruai,-1.276187,37.016097,Shell,,brand_tag,node,8202067662
|
||||
Shell,-1.170384,36.91699,Shell,,brand_tag,node,8231893217
|
||||
Shell - Eastern By-pass - Ruiru,-1.166976,36.964879,Shell,,brand_tag,node,8247882582
|
||||
Shell,-1.281858,36.962424,Shell,,brand_tag,node,8293614674
|
||||
Shell,-1.280476,36.690323,Shell,,brand_tag,node,8318718279
|
||||
Shell,-1.281238,36.634916,Shell,,brand_tag,node,8321307638
|
||||
Shell,-1.267623,36.609868,Shell,,brand_tag,node,8321307639
|
||||
Shell,-1.234142,36.989696,Shell,,brand_tag,node,8338714141
|
||||
Shell,-1.267493,37.315826,Shell,,brand_tag,node,8338714151
|
||||
Shell,-1.256284,36.879297,Shell,,brand_tag,node,8345317201
|
||||
Shell,-1.425451,36.958715,Shell,,brand_tag,node,8371880141
|
||||
Shell,-0.78364,36.871819,,,name,node,8483709617
|
||||
Shell,-1.28234,37.104729,Shell,,brand_tag,node,8862700293
|
||||
Shell,-1.231358,36.924224,Shell,,brand_tag,node,9004506167
|
||||
Shell Mirema,-1.213808,36.892432,Shell,,brand_tag,node,9014259343
|
||||
Shell,-1.459477,37.25047,Shell,,brand_tag,node,9028867156
|
||||
Shell,0.027305,36.365863,Shell,,brand_tag,node,9048281880
|
||||
Shell,-0.20372,35.843587,Shell,,brand_tag,node,9053460332
|
||||
Shell,-1.229246,36.84025,Shell,,brand_tag,node,10080456017
|
||||
Shell,-3.936283,39.744896,,,name,node,10255650224
|
||||
Shell Petrol Station,-1.286734,36.740804,,,name,node,10971302670
|
||||
Shell -Bombolulu,-4.025633,39.697377,Shell,,brand_tag,node,11500364389
|
||||
Shell,-1.313698,36.720914,Shell,,brand_tag,node,11711795482
|
||||
Shell,-1.274671,36.799912,Shell,,brand_tag,node,12163148139
|
||||
Shell,-1.307959,36.781587,Shell,,brand_tag,node,12165032865
|
||||
Shell,0.28095,34.744859,Shell,,brand_tag,node,12489868997
|
||||
Shell Syokimau,-1.378091,36.927818,,,name,node,12600151901
|
||||
Shell,-1.262862,36.90678,,,name,node,12886546509
|
||||
Shell Gas Station,-0.779732,36.427412,,,name,node,13088820396
|
||||
Shell,-1.230649,34.482215,Shell,,brand_tag,node,13446964574
|
||||
Shell,-0.900865,34.53639,Shell,,brand_tag,node,13446964578
|
||||
Shell,0.078482,34.720706,Shell,,brand_tag,node,13447042942
|
||||
Shell,0.600804,35.163178,Shell,,brand_tag,node,13528448115
|
||||
shell,-0.143442,34.801363,,,name,node,13714210044
|
||||
shell,-0.077947,34.727069,Shell,,brand_tag,node,13720608494
|
||||
Shell,-1.282701,36.824838,Shell,,brand_tag,way,123365084
|
||||
Shell,-1.2799,36.82273,Shell,,brand_tag,way,125107079
|
||||
Shell,-1.289727,36.811096,Shell,,brand_tag,way,125336232
|
||||
Shell,-1.287955,36.838738,Shell,,brand_tag,way,126936993
|
||||
Shell,-1.272506,36.83183,Shell,,brand_tag,way,126995763
|
||||
Shell Service Station - Jogoo Road,-1.295715,36.860244,Shell,,brand_tag,way,129323643
|
||||
Shell,-1.268949,36.817225,Shell,,brand_tag,way,133548150
|
||||
Shell,-1.310807,36.817766,Shell,,brand_tag,way,144188726
|
||||
Shell- Lavington,-1.280811,36.769286,Shell,Shell,brand_tag,way,226322138
|
||||
Shell,-1.322412,36.708022,Shell,,brand_tag,way,226714111
|
||||
Shell,-1.263021,36.802726,Shell,,brand_tag,way,238857325
|
||||
Shell,-1.30349,36.828502,Shell,,brand_tag,way,366568545
|
||||
Shell,-0.09443,34.76444,Shell,,brand_tag,way,390104534
|
||||
Shell,-0.282248,36.09429,Shell,,brand_tag,way,483211211
|
||||
Shell,-0.285481,36.075122,Shell,,brand_tag,way,488459901
|
||||
Shell Kitengela ex-Engen,-1.491085,36.954337,Shell,,brand_tag,way,536796727
|
||||
Shell Voi,-3.400372,38.549126,Shell,Shell,brand_tag,way,593734920
|
||||
Shell,-0.865417,36.566545,,,name,way,642076596
|
||||
Shell,-1.275817,36.834388,Shell,,brand_tag,way,686179946
|
||||
Shell,-0.536827,37.452256,Shell,,brand_tag,way,732602668
|
||||
Shell Petrol Station - Greenspan,-1.288893,36.902407,Shell,,brand_tag,way,977833647
|
||||
Shell petrol station (feroze),-1.26303,36.906097,Shell,,brand_tag,way,1055608664
|
||||
Shell,-0.268103,34.971231,Shell,,brand_tag,way,1069555494
|
||||
Shell Hurlingham,-1.295405,36.799606,Shell,,brand_tag,way,1088482284
|
||||
Shell Service Station - Embakassi,-1.317926,36.917619,Shell,,brand_tag,way,1107378445
|
||||
Shell,-1.328958,36.683097,Shell,,brand_tag,way,1107729725
|
||||
Shell,-2.831336,37.524927,Shell,,brand_tag,way,1110216588
|
||||
Shell Kahawa Sukari,-1.188788,36.931934,Shell,,brand_tag,way,1135425767
|
||||
Shell,-0.074421,34.691528,Shell,,brand_tag,way,1222533725
|
||||
Shell Service Station - Likoni Road,-1.303854,36.856227,Shell,,brand_tag,way,1273152921
|
||||
Shell,-1.835509,36.799548,Shell,,brand_tag,way,1290701573
|
||||
Shell,-2.538055,36.800975,Shell,,brand_tag,way,1413265354
|
||||
Shell Fuel Station - Tala,-1.26749,37.315706,,,name,way,1434009884
|
||||
Shell,-1.266132,36.98704,Shell,Shell,brand_tag,way,1497383075
|
||||
|
File diff suppressed because one or more lines are too long
Loading…
Reference in a new issue