Self-host MapLibre; de-dupe position_history
Some checks are pending
build / lint-test (push) Waiting to run
build / build-push (push) Blocked by required conditions

- Vendor maplibre-gl 4.7.1 (js+css) and serve from /vendor instead of the unpkg CDN — no external dependency/SRI gap for the core map.

- Projector skips duplicate (imei, occurred_at) history rows via NOT EXISTS (parked devices re-report the same gpsTime each poll); migration 23 dedupes existing rows and adds a unique index.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
kianiadee 2026-05-29 03:41:34 +03:00
parent 558f095392
commit 84e9421b4d
5 changed files with 106 additions and 3 deletions

View file

@ -223,16 +223,26 @@ async def _project_one(
),
)
# De-dupe at write time: a parked device re-reports the same gpsTime on
# every poll, which would otherwise pile identical (imei, occurred_at) rows
# into the history hypertable and inflate the trip "fix count". Single
# writer + the (imei, occurred_at) index make the NOT EXISTS check cheap,
# and it needs no unique constraint — so it can't fail if deployed before
# the dedup migration.
await cur.execute(
"""
INSERT INTO state.position_history (
vehicle_id, imei, occurred_at, geom, speed_kmh, direction_deg,
acc_state, altitude_m, satellites, source, parser_version,
mc_type, current_mileage_km, gps_signal, pos_type
) VALUES (
)
SELECT
%s, %s, %s, ST_SetSRID(ST_GeomFromText(%s), 4326),
%s, %s, %s, %s, %s, %s, %s,
%s, %s, %s, %s
WHERE NOT EXISTS (
SELECT 1 FROM state.position_history p
WHERE p.imei = %s AND p.occurred_at = %s
)
""",
(
@ -248,6 +258,7 @@ async def _project_one(
payload.get("current_mileage_km"),
payload.get("gps_signal"),
payload.get("pos_type"),
imei, occurred_at,
),
)
return True

View file

@ -0,0 +1,32 @@
-- migrate:up
--
-- One-time cleanup + guard for state.position_history duplicates.
--
-- A parked device re-reports the same gpsTime on every 60s poll, so before the
-- projector's write-time NOT EXISTS guard (shipped alongside this migration)
-- the history hypertable accumulated identical (imei, occurred_at) rows —
-- bloating storage and inflating the "fix count" shown in the trip dock.
--
-- Step 1 delete duplicates, keeping the lowest history_id per
-- (imei, occurred_at).
-- Step 2 add a unique index so duplicates can never reappear. It includes
-- occurred_at (the hypertable partition column), which TimescaleDB
-- requires for a unique index on a hypertable.
--
-- NOTE: the DELETE scans the whole hypertable; run it in a quiet window. The
-- migration is independent of the projector deploy order — the code's
-- NOT EXISTS guard needs no constraint, so applying this before/after a
-- redeploy is both safe.
DELETE FROM state.position_history a
USING state.position_history b
WHERE a.imei = b.imei
AND a.occurred_at = b.occurred_at
AND a.history_id > b.history_id;
CREATE UNIQUE INDEX IF NOT EXISTS position_history_imei_occurred_uq
ON state.position_history (imei, occurred_at);
-- migrate:down
DROP INDEX IF EXISTS state.position_history_imei_occurred_uq;

View file

@ -4,7 +4,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Live Fleet · rahamafresh</title>
<link rel="stylesheet" href="https://unpkg.com/maplibre-gl@4.7.1/dist/maplibre-gl.css" />
<link rel="stylesheet" href="/vendor/maplibre-gl.css" />
<style>
:root {
--bg: #0f172a;
@ -235,7 +235,7 @@
</aside>
</div>
<script src="https://unpkg.com/maplibre-gl@4.7.1/dist/maplibre-gl.js"></script>
<script src="/vendor/maplibre-gl.js"></script>
<script type="module">
import {
authClient, apiFetch, initMap, renderView,

1
web/vendor/maplibre-gl.css vendored Normal file

File diff suppressed because one or more lines are too long

59
web/vendor/maplibre-gl.js vendored Normal file

File diff suppressed because one or more lines are too long