mqqt-scrubber/docs/WORKPLAN.md

Work Plan

Goal

Build a simple Go daemon that runs continuously, subscribes to MQTT topics from home automation devices with Tasmota as the primary source, normalizes payloads, and stores the resulting time-series data in InfluxDB v3.

Milestones

Milestone 1: Ingestion skeleton

Estimate: 1 to 2 days

• create runnable Go service with config loading and structured logging
• connect to MQTT with reconnect support
• subscribe to tele/+/SENSOR and tele/+/STATE
• enqueue incoming messages without blocking the MQTT callback path
• batch and write records to InfluxDB v3
• implement graceful shutdown with final flush

Definition of done:

• service can run unattended for several hours
• on restart it reconnects and resumes subscriptions
• valid payloads reach InfluxDB

Milestone 2: Tasmota normalization

Estimate: 1 to 2 days

• collect sample payloads from real devices
• normalize nested Tasmota JSON into stable field keys
• derive a schema for measurement, tags, and fields
• support SENSOR and STATE payload families cleanly
• add parser unit tests for the most common device payloads

Definition of done:

• representative payloads from real devices parse consistently
• field names stay stable across restarts and devices
• parse failures are visible in logs and do not crash the daemon

Milestone 3: Operational hardening

Estimate: 1 to 2 days

• expose counters for received, parsed, written, dropped, and failed messages
• improve retry behavior and failure logging for Influx writes
• ensure bounded buffering and predictable memory use
• add sample deployment instructions for systemd or Docker
• document configuration and failure modes

Definition of done:

• service behavior under broker or Influx outages is understood and documented
• logs are enough to diagnose the common failure modes
• memory and buffering remain bounded under load

Milestone 4: Broader topic support

Estimate: 2 to 4 days

• add more Tasmota topics if needed, for example stat/+/STATUS10
• add device-specific enrichments only when justified by real payloads
• optionally separate measurements by domain such as energy, climate, and state
• add integration tests with a local MQTT broker and InfluxDB instance

Definition of done:

• new topic families are added without destabilizing the core pipeline
• schema remains low-cardinality and query-friendly

Backlog

• persistent local spool for outage tolerance
• dead-letter handling for invalid payloads
• health endpoint or Prometheus metrics
• config reload without restart
• per-topic parser routing for non-Tasmota devices
• retention and schema review once real data volume is known

Delivery order

1. Make the daemon run reliably with a narrow scope.
2. Lock in the Tasmota schema from real samples.
3. Add visibility and failure handling.
4. Broaden topic coverage only after the core path is stable.