Commit graph

15 commits

Author SHA1 Message Date
david kiania
3015104f5b docs+tooling: OSM POI export workflow + Shell stations data; graded DQ report
Some checks failed
Static Analysis / static (push) Has been cancelled
Tests / test (push) Has been cancelled
- scripts/export_osm_pois.py: reproducible OSM .pbf -> GeoJSON+CSV exporter
  (amenity/brand filter; pyosmium via uv, no system deps).
- docs/OSM_POI_EXPORT.md: runbook (extract -> export -> FleetNow layer) with
  reference counts (1,794 fuel stations; Shell=232).
- shell_stations.geojson/.csv: the Shell export of record (232 pts, kenya-260605).
- docs/reports/260608_fleet_registry_data_quality.*: rewritten as a graded
  (Red/Amber/Yellow) action plan with owners.
- .gitignore: ignore *.osm.pbf (331MB, reproducible). CLAUDE.md: index the new docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-08 22:04:49 +03:00
kianiadee
94cbd2a85e docs: document FleetNow merged dashboard + read-API topology + FIX-D03
Some checks failed
Static Analysis / static (push) Has been cancelled
Tests / test (push) Has been cancelled
- New '3. Map dashboards & read-API' subsection: the three SPAs (liveposition,
  fleetintelligence, fleetnow), how dashboard_api is deployed (standalone bridge
  container, not Coolify), and that FleetNow lives in its own repo.
- FIX-D03: fleetnow CORS origin + the deploy-script strip/guard fixes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-06 10:09:48 +03:00
david kiania
9986d3b411 docs(claude): add pending Grafana redeploy to open items (post ops/dwh_gold purge)
Some checks are pending
Static Analysis / static (push) Waiting to run
Tests / test (push) Waiting to run
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-05 20:39:56 +03:00
david kiania
8c5a43f3b8 chore(db): purge unused ops + dwh_gold schemas
Some checks are pending
Static Analysis / static (push) Waiting to run
Tests / test (push) Waiting to run
Drop the dormant ops (workshop / tickets / dispatch / SLA / odometer)
and dwh_gold (nightly ETL aggregates) schemas plus their dependents —
features never implemented, no live writer or scheduled refresh.

- Prod DB (already applied): DROP SCHEMA ops/dwh_gold CASCADE, plus
  tracksolid.dispatch_log, v_sla_inflight, v_utilisation_daily.
- migrations/12_drop_ops.sql + 13_drop_dwh_gold.sql (forward, all
  IF EXISTS) registered in run_migrations.py for rebuild durability.
- grafana: removed 8 now-broken panels (In-flight SLA, Idle Cost,
  Utilisation Heatmap, Row 7 Field-Service SLAs) from daily_operations;
  panel count 21 -> 13.
- docs: scrubbed CLAUDE.md, PLATFORM_OVERVIEW.html (-19KB), DATA_FLOW.md;
  pre-drop seed snapshot in docs/reports/260605_ops_purge_backup.md.

The separate tracksolid_dwh server (31.97.44.246:5888) is unrelated
and untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-05 18:11:03 +03:00
david kiania
e060933c55 docs: record fleet-dashboard filter fix + v_trips self-refresh
Some checks are pending
Static Analysis / static (push) Waiting to run
Tests / test (push) Waiting to run
Update PLATFORM_OVERVIEW.html (§2 migration, §4 read-API, §5 refresh_log,
§7 ops notes) and CLAUDE.md §7 fix history (FIX-D01, FIX-D02) to reflect
the two 2026-06-05 fixes that closed out the n8n→fleetapi cutover:
form-urlencoded POST body parsing, and moving the reporting.v_trips
matview refresh from the retired n8n job into dashboard_api.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-05 16:58:20 +03:00
david kiania
26fa1a4dc5 docs: add Grafana dashboards appendix + link PLATFORM_OVERVIEW from CLAUDE.md
Some checks are pending
Static Analysis / static (push) Waiting to run
Tests / test (push) Waiting to run
Adds section 6 (Grafana dashboards) to PLATFORM_OVERVIEW.html, generated from
the provisioned dashboard JSON: every panel in the NOC Fleet (9 panels) and
Daily Operations (23 panels) dashboards with type and source view/table.
Renumbers Operational notes to section 7. Links the doc from the CLAUDE.md
codebase map.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-05 12:50:48 +03:00
david kiania
e5b0e192d8 chore(repo): reorganize tree into migrations/ data/ legacy/ docs/
Group root-level files (accreted from incremental changes) by purpose
without moving any deployment entrypoint or breaking imports:

- migrations/  : numbered SQL 02-10
- data/        : source CSVs
- legacy/      : superseded pre-_rev scripts + old pipeline notes (not deployed)
- docs/{manuals,reference,reports}/ : loose manuals, references, reports
- strip stray ** / *** prefixes from 5 doc filenames
- delete empty documents.txt / push_webhook.md

Reference updates so nothing breaks:
- run_migrations.py  -> /app/migrations/<file>
- run_migrations.sh  -> $SCRIPT_DIR/migrations
- import_drivers_csv.py -> data/<csv>
- docker-compose.yaml -> runbook path comment
- CLAUDE.md -> codebase map + inline doc references

Deployed Python (3 services + ts_shared_rev + run_migrations) and the
documented ops one-shots stay at root, preserving the flat-import layout
and all documented commands. Verified: py_compile clean across all modules,
every MIGRATIONS entry resolves under migrations/, CI-referenced paths
(tests/, mypy targets, db_audit) and the grafana build context intact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-01 02:27:30 +03:00
David Kiania
34f5fa1b9c feat(dwh): bronze pipeline migrations, runbook, and execution manual
DWH pipeline (new):
  - dwh/261001_dwh_control.sql — watermarks + per-run audit log schema
  - dwh/261002_bronze_constraints_audit.sql — ON CONFLICT key assertion
  - dwh/261003_dwh_roles.sql — dwh_owner / grafana_ro contract assertion
  - dwh/261004_dwh_observability_views.sql — v_table_freshness,
    v_recent_failures, v_watermark_lag (readable by grafana_ro)
  - docs/DWH_PIPELINE.md — operations runbook (setup, troubleshooting,
    manual re-run, back-fill, rotation)
  - DWH_Execution_Manual.md — reusable playbook for future data
    projects (extract → blob → load pattern, 7 design principles,
    snapshot-vs-incremental matrix, verification gates)
  - docs/superpowers/{specs,plans}/2026-04-24-n8n-dwh-bronze-pipeline-*
    — design spec + 27-task implementation plan

Security:
  - dwh/260423_dwh_ddl_v1.sql — redacted plaintext role passwords to
    'CHANGE_ME_BEFORE_APPLY' placeholders; added SECURITY header
    documenting generation + rotation flow

Docs:
  - CLAUDE.md — §3 adds tracksolid_dwh@31.97.44.246:5888 target,
    §4 adds dwh/ + docs/DWH_PIPELINE.md to codebase map, §5 adds
    bronze + dwh_control schema roll-up, §10 adds deploy task +
    password rotation follow-up

Also includes miscellaneous in-progress files accumulated on this
branch (workspace, analytics notes, vehicle CSVs, extract helpers,
renamed markdown archives).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-25 01:07:53 +03:00
David Kiania
fa110f4313 feat: [FIX-M19] multi-account ingest across fireside sub-accounts
Some checks failed
Static Analysis / static (push) Waiting to run
Tests / test (push) Waiting to run
Static Analysis / static (pull_request) Has been cancelled
Tests / test (pull_request) Has been cancelled
Fleet lives across three Tracksolid sub-accounts:
  fireside         —  63 devices
  Fireside@HQ      —  52 devices
  Fireside_MSA     —  41 devices

Previously sync_devices / poll_live_positions / poll_parking only
queried a single TARGET_ACCOUNT, so ~64% of the fleet was invisible to
the pipeline.

Changes:
  - ts_shared_rev.py: new TARGETS list (env TRACKSOLID_TARGETS,
    comma-separated; falls back to the single TARGET_ACCOUNT).
  - ts_shared_rev.py: new get_active_imeis_by_target() helper that
    groups active IMEIs by their stored account so parking calls can
    pass the right account param per batch.
  - ingest_movement_rev.py: sync_devices and poll_live_positions loop
    over every target and dedupe by IMEI before upserting. poll_parking
    loops over imeis_by_target so each batch carries the matching
    account.
  - CLAUDE.md: FIX-M19 entry.

Requires new env var TRACKSOLID_TARGETS="fireside,Fireside@HQ,Fireside_MSA"
on the ingest services in Coolify.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-24 10:43:07 +03:00
David Kiania
417627675e fix: [FIX-M18] pull driverName/vehicleNumber/sim from detail endpoint
Some checks failed
Static Analysis / static (push) Has been cancelled
Tests / test (push) Has been cancelled
Static Analysis / static (pull_request) Has been cancelled
Tests / test (pull_request) Has been cancelled
jimi.user.device.list returns null for vehicleName, vehicleNumber,
driverName, driverPhone, and sim even after those fields are set via
jimi.open.device.update — the values only surface through
jimi.track.device.detail. sync_devices() now reads from dtl first with
d as fallback, which unblocks backfill of the 144 CSV-driven updates
pushed on 2026-04-22.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-22 18:21:25 +03:00
David Kiania
778686e7ce docs: CLAUDE.md audit — add backup sidecar, missing files, update open items
Some checks failed
Static Analysis / static (push) Waiting to run
Tests / test (push) Waiting to run
Static Analysis / static (pull_request) Has been cancelled
Tests / test (pull_request) Has been cancelled
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-21 16:01:38 +03:00
David Kiania
85d02c81a5 feat: Daily Operations dashboard + tracksolid analytics views
Some checks failed
Static Analysis / static (push) Has been cancelled
Tests / test (push) Has been cancelled
Static Analysis / static (pull_request) Has been cancelled
Tests / test (pull_request) Has been cancelled
Add a second Grafana dashboard focused on daily operational KPIs and live
dispatch, keeping the NOC Live dashboard untouched.

- grafana/provisioning/dashboards-json/daily_operations_dashboard.json
  New dashboard covering §7 Blueprint Panels 3-8 and the §4 dispatch lens:
  freshness banner, today-at-a-glance stat row, active vehicles map,
  currently-idle table, vehicles-not-moved-today, per-vehicle daily KPI
  roll-up, driver behaviour leaderboard, distance trend, alarm frequency,
  idle cost MTD, utilisation heatmap, SLA row (collapsed, data-gated).

- 07_analytics_views.sql
  Nine views in tracksolid.* wrapping the BA-file [DASHBOARD]-tagged
  queries. Each view carries COMMENT ON VIEW with its spec section.
  SELECT granted to grafana_ro. Smoke-tested against live DB.

- run_migrations.py
  Register 06 and 07 in MIGRATIONS list with idempotent seed checks so
  future fresh deploys apply them correctly.

- CLAUDE.md
  Retire the tracksolid_2 schema references (schema no longer exists);
  §9 Fleet State dated 2026-04-19 with correct pipeline status (running,
  875 runs/24h, 0 failures) and accurate position_history row counts
  (hypertable stats don't show in pg_stat_user_tables).

- docs/superpowers/specs/2026-04-19-daily-operations-dashboard-design.md
  Design spec covering architecture, views, panel layout, deployment,
  rollback, and known data gaps.
2026-04-19 13:44:18 +03:00
David Kiania
4371a0d6e6 docs: CLAUDE.md audit — add commands section, fix stale DB access note
Some checks failed
Static Analysis / static (push) Waiting to run
Tests / test (push) Waiting to run
Static Analysis / static (pull_request) Has been cancelled
Tests / test (pull_request) Has been cancelled
Add §0 Commands (uv, pytest, ruff, mypy, docker exec query pattern).
Fix §3 DB access: DATABASE_URL is internal-only since fix 152fce8.
Add docs/superpowers/ to codebase map.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 00:27:59 +03:00
David Kiania
244112154a docs: update CLAUDE.md with session learnings (18 Apr 2026)
Some checks failed
Static Analysis / static (push) Waiting to run
Tests / test (push) Waiting to run
Static Analysis / static (pull_request) Has been cancelled
Tests / test (pull_request) Has been cancelled
- §3: note tracksolid_2 as live schema, tracksolid as empty target;
  add DB direct access tip (31.97.44.246:5888, leading space in .env)
- §4: add import_drivers_csv.py and migration 06 to codebase map
- §5: document tracksolid_2 live tables with column differences
  (assigned_team vs cost_centre, city vs assigned_city); add ops.*
- §8: add rule 9 (Forgejo API auth via keychain) and rule 10
  (always check active schema before querying)
- §9: update fleet state — pipeline stopped Apr 6, CSV fleet pending,
  0 driver names, 19 stale positions
- §10: replace driver-name manual item with deploy + CSV import tasks

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-18 12:26:21 +03:00
David Kiania
f9834564ab Add CLAUDE.md and project docs for structured Claude project
CLAUDE.md: cached context file covering project identity, tech stack,
codebase map, schema quick-ref, API gotchas, fix history, working rules,
fleet state, and open items. Structured for maximum cache efficiency —
stable content first, dynamic state at the end.

docs/CONNECTIONS.md: connection parameter shapes (no secrets) for SSH,
DB, API, container resolution, Forgejo, Grafana, n8n.

docs/PROJECT_CONTEXT.md: client business context (telco field service,
3 cities, service types), data quality gaps, KPI framework by domain,
integration roadmap.

docs/KPI_FRAMEWORK.md: living KPI register with status tracking,
thresholds, client feedback log, and review checklist. To be co-developed
with client iteratively.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-12 20:59:15 +03:00