5f5d71d500
Split the INC-only loader into a dataset-agnostic engine (pipeline.py, renamed from import_tickets.py) parameterized by a Dataset config, with thin per-type entrypoints inc/import_inc.py and crq/import_crq.py. CRQ shares INC's identical 32-column source schema and CDC change stream, so the engine is fully shared. - pipeline.py: Dataset config (name/table/prefixes/key_regex/post_apply); INC keeps the capture_history post-apply hook, CRQ has none yet. geocode_locations now unions tickets.crq (geocoding is cross-dataset: one gazetteer/budget). - crq/import_crq.py: drains automations/crq/changes/ from isptickets into tickets.crq (data layer + map; SLA/dashboard/history deferred). - migrations/13_crq_columns.sql: CRQ mirror of 03 — typed STORED generated columns + indexes on tickets.crq (reuses tickets.eat_ts()). - Deployment: Dockerfile/run_ingest.sh run both via `python -m`; pyproject packages inc/crq. Docs (README, implementation, deployment-and-operations, n8n export ref, phase-1) updated for the split + the one-time CRQ seed runbook. tickets.crq already exists (mig 01, LIKE tickets.inc) and is unioned into reporting.fn_tickets_for_map + resolve_ticket_geoms, so CRQ appears on the existing Tickets map once seeded. Verified locally: ruff-clean new files, engine lists/parses both streams against live S3 (crq=52 files, inc unaffected). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
34 lines
1.4 KiB
Bash
Executable file
34 lines
1.4 KiB
Bash
Executable file
#!/usr/bin/env bash
|
||
# run_ingest.sh — fleettickets · INC + CRQ ingest wrapper for cron (plain host/VM).
|
||
#
|
||
# Loads env from the local .env (DATABASE_URL + RUSTFS_* + GEOCODER_*) and drains
|
||
# both ticket change streams with --apply (watermark skip-if-unchanged + per-file
|
||
# archive are built in, so a run with no new files is a cheap no-op).
|
||
#
|
||
# Install on the instance (every 20 min, 06:00–20:40 EAT):
|
||
# */20 6-20 * * * /opt/fleettickets/run_ingest.sh >> /var/log/fleettickets.log 2>&1
|
||
# Ensure the crontab runs in the Africa/Nairobi timezone (CRON_TZ=Africa/Nairobi or
|
||
# the host/container TZ), since the export filenames and the schedule are EAT.
|
||
#
|
||
# On Coolify the two ingests run as separate Scheduled Tasks instead (see Dockerfile
|
||
# + docs/deployment-and-operations.md); this wrapper is the plain-host fallback.
|
||
set -euo pipefail
|
||
|
||
cd "$(dirname "$0")"
|
||
|
||
# Load .env if present (KEY=VALUE lines); never commit the real .env.
|
||
if [ -f .env ]; then
|
||
set -a
|
||
# shellcheck disable=SC1091
|
||
. ./.env
|
||
set +a
|
||
fi
|
||
|
||
# Prefer the project venv if it exists, else the python on PATH (e.g. in-container).
|
||
PY="python"
|
||
[ -x ".venv/bin/python" ] && PY=".venv/bin/python"
|
||
|
||
# Run from the repo root (cwd above) so `-m inc.import_inc` / `-m crq.import_crq`
|
||
# resolve the packages alongside pipeline.py + shared.py.
|
||
"$PY" -m inc.import_inc --from-bucket --apply
|
||
"$PY" -m crq.import_crq --from-bucket --apply
|