diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..20ebd66 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,161 @@ +# CLAUDE.md — Fireside Communications · Tracksolid Fleet Intelligence + +## 1. What This Project Is + +Fleet telematics ingestion and analytics stack for a **telco first-line support client** operating in Nairobi, Mombasa, and Kampala. The client dispatches field technicians to install, repair, and maintain home and business broadband, handle LOS signal faults, service migrations, and maintain outside plant infrastructure. The fleet is ~80 vehicles across three cities, all tracked via Tracksolid Pro (Jimi IoT API). + +This repository ingests the Tracksolid Pro API into a TimescaleDB/PostGIS database and visualises fleet and operational KPIs in Grafana. The pipeline is deployed on Coolify at `stage.rahamafresh.com`. + +**Repository:** `https://repo.rahamafresh.com/kianiadee/tracksolid_timescale_grafana_prod.git` + +--- + +## 2. Tech Stack + +| Layer | Technology | +|---|---| +| Ingestion | Python 3.12 — `ingest_movement_rev.py`, `ingest_events_rev.py`, `webhook_receiver_rev.py` | +| Shared utils | `ts_shared_rev.py` — token cache, DB pool, API signing, clean helpers | +| Database | PostgreSQL 16 + TimescaleDB 2.15 + PostGIS 3 (`tracksolid_db`) | +| Orchestration | Docker Compose on Coolify | +| Visualisation | Grafana (provisioned via custom image) | +| Workflow automation | n8n | +| API source | Tracksolid Pro / Jimi IoT Open Platform (`eu-open.tracksolidpro.com/route/rest`) | +| Version control | Forgejo at `repo.rahamafresh.com` | + +--- + +## 3. Instance & Connection Parameters + +See `docs/CONNECTIONS.md` for the full shape. Summary: + +- **SSH:** `ssh -i ~/.ssh/id_ed25519 kianiadee@stage.rahamafresh.com` +- **DB name:** `tracksolid_db` · **DB user:** `postgres` (internal) · `tracksolid_owner` (app) · `grafana_ro` (read-only) +- **DB schema:** `tracksolid` (operational) · `infrastructure` · `dwh_gold` (aggregates) +- **Container naming:** Coolify appends a random suffix. Always resolve with: + ```bash + docker ps --filter name= --format "{{.Names}}" | head -1 + ``` + e.g. `docker ps --filter name=timescale_db --format "{{.Names}}" | head -1` +- **Env vars:** loaded from `.env` via `env_file` in `docker-compose.yaml`. See `docs/CONNECTIONS.md` for variable names. Never hardcode secrets. + +--- + +## 4. Codebase Map + +``` +ts_shared_rev.py # Shared: config, signing, DB pool, token cache, clean helpers +ingest_movement_rev.py # GPS positions, trips, parking, track-list (high-res trail), device sync +ingest_events_rev.py # Alarm events polling (fallback for webhook push) +webhook_receiver_rev.py # FastAPI push receiver: /pushobd /pushevent /pushtripreport etc. +sync_driver_audit.py # One-shot: API↔DB driver/IMEI gap report + full upsert +run_migrations.py # Applies SQL migrations in order at container startup +docker-compose.yaml # Services: timescale_db, ingest_movement, ingest_events, + # webhook_receiver, grafana +grafana/ # Grafana provisioning (baked into image) +n8n-workflows/ # n8n workflow exports +docs/ # Reference docs (connections, API, KPIs, project context) +02_tracksolid_full_schema_rev.sql # Full schema bootstrap +03..05_*.sql # Incremental migrations +01_BusinessAnalytics.md # SQL analytics library (reference before writing queries) +tracksolidApiDocumentation.md # API endpoint reference +260412_baseline_report.md # Latest fleet state snapshot +``` + +--- + +## 5. Database Schema — Key Tables + +```sql +tracksolid.devices -- Master device registry (63 devices, imei PK) +tracksolid.live_positions -- Current position per device (1 row per IMEI, upserted) +tracksolid.position_history -- All GPS fixes (hypertable, partitioned by gps_time) + -- source: 'poll' (60s sweep) | 'track_list' (30m high-res) +tracksolid.trips -- Trip summaries: distance_km, driving_time_s, avg/max speed +tracksolid.parking_events -- Stop events with duration and address +tracksolid.alarms -- Alarm events (alarm_type, alarm_name, alarm_time) +tracksolid.obd_readings -- OBD diagnostics (push only, awaiting webhook registration) +tracksolid.device_events -- Power on/off tamper events +tracksolid.ingestion_log -- API call audit trail per endpoint +dwh_gold.fact_daily_fleet_metrics -- Nightly ETL aggregates per vehicle per day +``` + +Full DDL: `02_tracksolid_full_schema_rev.sql` + migrations `03`–`05`. + +--- + +## 6. API Critical Facts + +**Always read `tracksolidApiDocumentation.md` before adding a new endpoint call.** + +| Fact | Detail | +|---|---| +| Auth | OAuth2 — token cached in `tracksolid.api_token_cache`, refreshed via `jimi.oauth.token.refresh` | +| Signing | MD5: `secret + sorted(k+v pairs) + secret` — see `build_sign()` in `ts_shared_rev.py` | +| Batch limit | Max 50 IMEIs per call for most endpoints | +| `distance` field | **Returns METRES, not km** despite docs. Always divide by 1000. (FIX-M16) | +| `driverName`/`driverPhone` | From `jimi.user.device.list` — will be NULL if not set in Tracksolid Pro UI | +| `alarm_type` field | API polling returns `alertTypeId`/`alarmTypeName` — NOT `alarmType`/`alarmName` (FIX-E06) | +| `durSecond` | Parking endpoint returns `durSecond`, not `seconds` (FIX-M13) | +| `jimi.device.track.mileage` | `startMileage`/`endMileage` are cumulative odometer in **metres** | +| Rate limit | Code 1006 — back off and retry with re-sign (handled in `api_post()`) | +| OBD data | Push only via `/pushobd` webhook — no polling endpoint exists | + +--- + +## 7. Fix History (do not regress) + +| Fix ID | File | What it fixed | +|---|---|---| +| FIX-M11 | `ingest_movement_rev.py` | Removed erroneous ×1000 on distance (was storing km as mm) | +| FIX-M13 | `ingest_movement_rev.py` | Parking: added `acc_type=0`, `account`; mapped `durSecond` | +| FIX-M14 | `ingest_movement_rev.py` | `poll_track_list()` — high-res GPS trail every 30m | +| FIX-M15 | `ingest_movement_rev.py` | `get_device_locations()` — on-demand precision refresh | +| FIX-M16 | `ingest_movement_rev.py` | `distance` from API is metres → divide by 1000 before storing | +| FIX-M17 | `ingest_movement_rev.py` | `sync_devices()` ON CONFLICT now updates all 26 fields (was 5) | +| FIX-E06 | `ingest_events_rev.py` | Alarm field mapping: `alertTypeId`/`alarmTypeName`/`alertTime` | +| BUG-02 | Migration 04 | Historical `distance_m` rows ÷1,000,000 → renamed to `distance_km` | + +--- + +## 8. Working Rules + +1. **No prod push without explicit user confirmation.** Always state what you are about to push and wait. +2. **Never rewrite a migration that is already applied.** Check `tracksolid.schema_migrations` first. Add a new numbered migration file for any schema change. +3. **Read before writing.** Before suggesting any code change, read the relevant source file. Before writing a query, check `01_BusinessAnalytics.md` for an existing pattern. +4. **Reuse shared utilities.** All DB access via `get_conn()`, all API calls via `api_post()`, all cleaning via `clean()` / `clean_num()` / `clean_int()` / `clean_ts()` in `ts_shared_rev.py`. Do not reinvent these. +5. **Resolve container names dynamically.** Never hardcode the Coolify suffix. Use `docker ps --filter name=`. +6. **SSH only when asked.** Default workflow is local code → commit → push. SSH into the instance only when explicitly asked to test or run something live. +7. **Secrets from env only.** Connection strings, API keys, and passwords live in `.env`. Reference variable names from `docs/CONNECTIONS.md`, never values. +8. **Two developers, one incoming.** Write code and docs that a second developer (mixed technical/operations background) can follow without prior context. + +--- + +## 9. Fleet State (as of 2026-04-12 baseline) + +| Metric | Value | +|---|---| +| Total registered devices | 63 (growing to 80) | +| Devices with GPS fix < 2h | 2 | +| Devices never reported | 44 | +| Driver names populated | 0 — must be set in Tracksolid Pro UI first | +| Cities active | Nairobi (primary), Mombasa (deploying), Kampala (1 device confirmed) | +| Uganda anomaly | X3-63282 at 0.196, 32.540 — under investigation | +| Service flags | KDK 829A GP (239,264 km), Belta KCU-647D (235,000 km) | + +Latest full snapshot: `260412_baseline_report.md` + +--- + +## 10. Open Items (update as resolved) + +| Priority | Item | +|---|---| +| HIGH | Assign driver names + vehicle numbers in Tracksolid Pro UI | +| HIGH | Register webhooks: `/pushobd` `/pushoil` `/pushtem` `/pushlbs` `/pushevent` | +| HIGH | Investigate X3-63282 in Kampala — legitimate or unauthorised? | +| MEDIUM | Set `fuel_100km` per vehicle type to activate fuel cost calculations | +| MEDIUM | Investigate 44 silent devices — SIM installed? Activated? | +| MEDIUM | Co-develop client KPI framework (see `docs/KPI_FRAMEWORK.md`) | +| LOW | Populate geofences — depot boundaries, city zones | +| LOW | Run nightly ETL: `SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1)` | diff --git a/docs/CONNECTIONS.md b/docs/CONNECTIONS.md new file mode 100644 index 0000000..3ed1748 --- /dev/null +++ b/docs/CONNECTIONS.md @@ -0,0 +1,111 @@ +# Connection Parameters Reference + +**No secrets are stored here. All values come from `.env` at runtime.** + +--- + +## SSH + +``` +Host: stage.rahamafresh.com +User: kianiadee +Key: ~/.ssh/id_ed25519 +``` + +```bash +ssh -i ~/.ssh/id_ed25519 kianiadee@stage.rahamafresh.com +``` + +--- + +## Database + +| Parameter | Value | +|---|---| +| Database | `tracksolid_db` | +| Host (internal) | `timescale_db` (Docker service name) | +| Port | `5432` | +| App user | `tracksolid_owner` | +| Read-only user | `grafana_ro` | +| Superuser | `postgres` | + +### `.env` variable names + +``` +POSTGRES_DB=tracksolid_db +POSTGRES_USER=... +POSTGRES_PASSWORD=... +DATABASE_URL=postgresql://tracksolid_owner:@timescale_db:5432/tracksolid_db +GRAFANA_DB_RO_PASSWORD=... +``` + +### Run a query from host + +```bash +DB=$(docker ps --filter name=timescale_db --format "{{.Names}}" | head -1) +docker exec $DB psql -U postgres -d tracksolid_db -c "SELECT COUNT(*) FROM tracksolid.devices;" +``` + +### Run a query file + +```bash +docker exec -i $DB psql -U postgres -d tracksolid_db < migration.sql +``` + +--- + +## Tracksolid Pro API + +| Parameter | Env var | +|---|---| +| App key | `TRACKSOLID_APP_KEY` | +| App secret | `TRACKSOLID_APP_SECRET` | +| User ID | `TRACKSOLID_USER_ID` | +| Target account | `TRACKSOLID_TARGET_ACCOUNT` (defaults to USER_ID) | +| Password MD5 | `TRACKSOLID_PWD_MD5` | +| Base URL | `TRACKSOLID_API_URL` (default: `https://eu-open.tracksolidpro.com/route/rest`) | + +--- + +## Container Name Resolution + +Coolify appends a random suffix to all container names. Never hardcode. Always resolve: + +```bash +# Pattern +docker ps --filter name= --format "{{.Names}}" | head -1 + +# Examples +docker ps --filter name=timescale_db --format "{{.Names}}" | head -1 +docker ps --filter name=ingest_movement --format "{{.Names}}" | head -1 +docker ps --filter name=webhook_receiver --format "{{.Names}}" | head -1 +docker ps --filter name=grafana --format "{{.Names}}" | head -1 +``` + +Current suffix (may change on redeploy): `bo3nov2ija7g8wn9b1g2paxs-19xxxxxxxxxx` + +--- + +## Forgejo + +``` +Host: https://repo.rahamafresh.com +Repo: kianiadee/tracksolid_timescale_grafana_prod +Remote: https://repo.rahamafresh.com/kianiadee/tracksolid_timescale_grafana_prod.git +``` + +--- + +## Grafana + +- Deployed as Docker service `grafana` +- Provisioning baked into image (datasources + dashboards via `grafana/Dockerfile`) +- Admin password: `GF_SECURITY_ADMIN_PASSWORD` from `.env` +- Default dashboard: NOC Fleet Dashboard + +--- + +## n8n + +- Deployed as separate Coolify service (`n8n-usoksgg8o40044g0cw08s8wc`) +- Workflows exported to `n8n-workflows/` diff --git a/docs/KPI_FRAMEWORK.md b/docs/KPI_FRAMEWORK.md new file mode 100644 index 0000000..2047cef --- /dev/null +++ b/docs/KPI_FRAMEWORK.md @@ -0,0 +1,101 @@ +# KPI Framework — Telco Field Service Fleet +## Fireside Communications · Co-developed with client + +> **Status:** Draft — pending client review and validation. +> Update this file after each client feedback session. Move KPIs from Proposed → Active → Retired as the programme matures. + +--- + +## How to Use This Document + +1. **Proposed** — KPI defined, not yet validated with client +2. **Active** — Client confirmed this matters; query written; Grafana panel exists or is in progress +3. **Baseline set** — Enough historical data exists to set a meaningful target +4. **Retired** — No longer tracked (document reason) + +Each active KPI should link to: +- The SQL query (or reference to `01_BusinessAnalytics.md`) +- The Grafana panel name/dashboard +- The refresh frequency +- The person who reviews it + +--- + +## KPI Status Register + +### Fleet Utilisation + +| KPI | Status | SQL ref | Grafana panel | Reviewed by | Cadence | +|---|---|---|---|---|---| +| Utilisation rate (%) | Proposed | `01_BusinessAnalytics.md §2.1` | — | — | Daily | +| Idle time % of shift | Proposed | `01_BusinessAnalytics.md §2.2` | — | — | Daily | +| Vehicles not moved today | Proposed | `01_BusinessAnalytics.md §2.3` | — | — | Daily | +| Fleet km today | Proposed | `01_BusinessAnalytics.md §5.1` | — | — | Daily | +| Fleet km this week | Proposed | `01_BusinessAnalytics.md §5.2` | — | — | Weekly | + +### Technician Productivity *(requires job system integration)* + +| KPI | Status | SQL ref | Grafana panel | Reviewed by | Cadence | +|---|---|---|---|---|---| +| Jobs completed per tech per day | Proposed | TBD | — | — | Daily | +| First-time fix rate | Proposed | TBD | — | — | Weekly | +| Mean time to arrive (MTTA) | Proposed | TBD | — | — | Weekly | +| Mean time to repair (MTTR) | Proposed | TBD | — | — | Weekly | +| SLA compliance rate | Proposed | TBD | — | — | Weekly | + +### Driver Behaviour + +| KPI | Status | SQL ref | Grafana panel | Reviewed by | Cadence | +|---|---|---|---|---|---| +| Speeding events per 100 km | Proposed | `01_BusinessAnalytics.md §3.1` | — | — | Weekly | +| Harsh driving index | Proposed | `01_BusinessAnalytics.md §3.2` | — | — | Weekly | +| Late starts (count per driver) | Proposed | `01_BusinessAnalytics.md §3.3` | — | — | Monthly | +| Early knock-off | Proposed | `01_BusinessAnalytics.md §3.3` | — | — | Monthly | +| After-hours movement | Proposed | `01_BusinessAnalytics.md §3.4` | — | — | Daily | + +### Asset Health & Cost + +| KPI | Status | SQL ref | Grafana panel | Reviewed by | Cadence | +|---|---|---|---|---|---| +| Estimated idle fuel cost (KES) | Proposed | `01_BusinessAnalytics.md §2.2` | — | — | Monthly | +| Vehicles at service threshold | Proposed | TBD | — | — | Weekly | +| Alarm rate per vehicle/week | Proposed | `01_BusinessAnalytics.md §6` | — | — | Weekly | +| GPS offline rate | Proposed | — | — | — | Daily | + +--- + +## Severity & Threshold Reference + +Adjust with client after first month of live data: + +| Metric | Green | Amber | Red | +|---|---|---|---| +| Fleet utilisation rate | > 60% | 40–60% | < 40% | +| Idle time % of shift | < 15% | 15–30% | > 30% | +| Speeding per 100 km | < 0.5 | 0.5–2.0 | > 2.0 | +| Harsh driving index | < 0.5 | 0.5–2.0 | > 2.0 | +| Late starts / month | 0–1 | 2–4 | ≥ 5 | +| Alarm rate / vehicle / week | 0–2 | 3–7 | > 7 | +| GPS offline rate | < 5% | 5–15% | > 15% | +| MTTA (minutes) | < 30 | 30–60 | > 60 | +| First-time fix rate | > 85% | 70–85% | < 70% | +| SLA compliance | > 95% | 85–95% | < 85% | + +--- + +## Client Feedback Log + +| Date | Session | Feedback | Action | +|---|---|---|---| +| — | Initial framework | Draft created | Awaiting first client review | + +--- + +## Next Review Checklist + +- [ ] Confirm shift hours (start, end, lunch, working days) +- [ ] Confirm SLA tiers (home vs business customer) +- [ ] Confirm which KPIs the ops manager wants on a daily digest +- [ ] Confirm reporting format (Grafana link, PDF, WhatsApp summary) +- [ ] Identify job management system / ticketing tool for MTTA/MTTR +- [ ] Confirm vehicle categories (motorcycle, van, 4WD) for per-type benchmarks diff --git a/docs/PROJECT_CONTEXT.md b/docs/PROJECT_CONTEXT.md new file mode 100644 index 0000000..4b4f1df --- /dev/null +++ b/docs/PROJECT_CONTEXT.md @@ -0,0 +1,131 @@ +# Project Context — Fireside Communications Fleet Intelligence + +## The Client + +A first-line technical support operation contracted by a large Kenyan/East African telco. The client manages field technicians who handle the full spectrum of last-mile broadband support: + +| Service Type | Description | +|---|---| +| New installations | Fibre/broadband installs at home and business premises | +| Fault resolution | LOS (Loss of Signal) troubleshooting, slow service investigations | +| Outside plant maintenance | Physical cable, cabinet, and pole infrastructure maintenance | +| Migrations | Customer plan or technology upgrades requiring a site visit | +| Business customer support | Prioritised SLA-driven support for commercial accounts | + +## Operational Geography + +| City | Status | Notes | +|---|---|---| +| Nairobi | Primary — fully operational | Main depot at Kikuyu Rd corridor (~-1.237, 36.727) | +| Mombasa | Deploying | Fleet being onboarded | +| Kampala, Uganda | 1 device confirmed | X3-63282 at 0.196, 32.540 — status under investigation | + +All three cities managed from a single Tracksolid Pro account and a single database instance. A `city` field or grouping by device group should be used for per-city analytics rather than separate schemas. + +## The Fleet + +- ~80 vehicles total (63 currently registered in Tracksolid Pro) +- Mix of motorcycles (courier/light inspection) and vans/4WDs (equipment and crew) +- Device models in use: AT4 (hardwired), JC400P (camera-capable), X3 (compact), GT06E (OBD) +- Vehicle identity (plate numbers, driver assignments) not yet populated in Tracksolid Pro — primary data quality gap + +## Data Quality Gaps (as of April 2026) + +| Gap | Impact | Resolution path | +|---|---|---| +| No driver names assigned | Reports show IMEIs instead of people | Assign in Tracksolid Pro UI → DB syncs nightly | +| No vehicle numbers populated | Cannot link vehicle to job/plate | Manual UPDATE or CSV import | +| 44 of 63 devices never reported GPS | Cannot track these vehicles | Verify SIM installation + activation | +| `fuel_100km` null for all devices | Fuel cost calculations inactive | Set by vehicle type via UPDATE | +| No geofences defined | Cannot alert on depot departures or route deviations | Define depot polygons + city zones | +| Webhooks not registered | OBD, fuel, temperature tables empty | Register in Tracksolid Pro account settings | + +--- + +## KPI Framework + +> This section is developed iteratively with the client. KPIs are grouped by operational domain. As client feedback arrives, move items from "Proposed" to "Active" and add the Grafana panel reference. + +### Domain 1 — Fleet Utilisation + +Measures whether vehicles are productively deployed during working hours. + +| KPI | Definition | Target | Status | +|---|---|---|---| +| Utilisation rate | Drive time / shift hours × 100 | > 60% | Proposed | +| Idle time % | Engine-on-stationary / total shift | < 15% | Proposed | +| Vehicles not moved today | COUNT where no trip recorded | 0 | Proposed | +| Fleet km per day | SUM(distance_km) across all trips | Baseline TBD | Proposed | + +### Domain 2 — Field Technician Productivity + +Measures output per technician per day. **Requires job management system integration or manual job log.** + +| KPI | Definition | Target | Status | +|---|---|---|---| +| Jobs completed per technician per day | Count of closed jobs | Baseline TBD | Proposed | +| First-time fix rate | Jobs resolved on first visit % | > 80% | Proposed | +| Mean time to arrive (MTTA) | Job assignment → vehicle on-site | < 45 min | Proposed | +| Mean time to repair (MTTR) | Job creation → job closed | < 2 hours | Proposed | +| SLA compliance rate | % jobs closed within SLA window | > 95% | Proposed | + +> Note: MTTA and MTTR require job timestamps from the telco's ticketing system. Integration point TBD. + +### Domain 3 — Driver Behaviour & Safety + +Measures driving quality. Feeds into insurance, safety, and coaching programmes. + +| KPI | Definition | Target | Status | +|---|---|---|---| +| Speeding events per 100 km | GPS fixes > 80 km/h / total km × 100 | < 0.5 | Proposed | +| Harsh driving index | Speed delta > 30 km/h in < 60s per 100 km | < 0.5 | Proposed | +| After-hours movement | Trips starting before 06:00 or after 20:00 EAT | 0 | Proposed | +| Late starts | First ignition after 07:45 EAT | < 2/month | Proposed | +| Early knock-off | Last trip ended before 17:00 EAT | < 2/month | Proposed | + +### Domain 4 — Route & Dispatch Efficiency + +Measures how well vehicles are matched to jobs geographically. + +| KPI | Definition | Target | Status | +|---|---|---|---| +| Avg distance per job | Total km / jobs completed | Baseline TBD | Proposed | +| Nearest available vehicle ETA | PostGIS dispatch query | < 30 min | Proposed | +| Return-to-depot rate | % trips ending at primary depot | Baseline TBD | Proposed | + +### Domain 5 — Asset Health & Cost + +Measures maintenance burden and fuel efficiency. + +| KPI | Definition | Target | Status | +|---|---|---|---| +| Estimated idle fuel cost (KES) | Idle hours × 0.8 L/h × KES 180/L | Minimise | Proposed | +| Vehicles approaching service interval | Odometer > threshold | 0 overdue | Proposed | +| Alarm rate per vehicle per week | COUNT(alarms) / 7 | < 2 | Proposed | +| GPS offline rate | Devices with fix age > 10 min / total | < 10% | Proposed | + +### Shift Schedule Assumptions + +Adjust these as confirmed with client: + +| Parameter | Assumed Value | +|---|---| +| Shift start | 07:30 EAT | +| Late threshold | After 07:45 EAT | +| Shift end | 17:00 EAT | +| After-hours | Before 06:00 or after 20:00 EAT | +| Working days | Monday–Saturday (confirm with client) | +| Shift length for utilisation | 10 hours | + +--- + +## Integration Roadmap + +| Integration | What it unlocks | Priority | +|---|---|---| +| Telco ticketing system (job timestamps) | MTTA, MTTR, first-time fix rate, jobs/day | HIGH | +| Tracksolid webhook registration | OBD, fuel, temperature, tamper events | HIGH | +| Driver assignment in Tracksolid Pro | All driver-attributed KPIs | HIGH | +| Geofence definition | Depot departure alerts, city zone coverage | MEDIUM | +| Fuel sensor webhook (`/pushoil`) | Actual fuel consumption vs estimated | MEDIUM | +| Temperature sensor (`/pushtem`) | Cold-chain compliance (if applicable) | LOW |