Cleanup — Housekeeping & Datapoint Cleaner
Core capabilities
- Identify stale data points, orphaned sensors, and unused tag definitions based on configurable inactivity windows.
- Safe archival to compressed snapshot stores (with CRC/SHA verification).
- Dry‑run and report mode with full change diffs and audit logs for compliance.
- Normalization and deduplication pipelines for tag metadata and historical series.
- Integration with backup and retention policies; supports tiered archival (S3, MinIO, on‑prem NAS).
How Cleanup works
Cleanup analyses database records and time‑series meta information using configurable rules. It generates a change plan (human‑readable diff) and an auditable runbook. Administrators should always review the plan in dry‑run mode before applying changes. Archival moves data to a chosen cold store and replaces live records with compact pointers for recovery.
- Scan phase — collect candidates based on inactivity thresholds, tag ownership and retention rules.
- Plan phase — generate human‑readable change plan with impact assessment and space gains.
- Review & Approval — export plan as JSON/HTML for audit and sign‑off.
- Apply phase — archive selected records, update indexes, and emit verification checksums.
- Post‑apply verification — run integrity checks and report results to monitoring and SIEM.
Configuration highlights
- Rule engine: regex and metadata conditions, owner whitelists, tag classification.
- Inactivity window: configurable per tag class (hours/days/months).
- Archival backends: S3, MinIO, NFS, CIFS — configurable lifecycle policies.
- Retention policies: soft‑delete grace windows and automatic pointer creation.
- Compliance: immutable archival support (WORM) for regulated environments.
Dry‑run & Auditability
Every Cleanup run produces:
- Full change plan (JSON + HTML) summarizing records, sizes and expected space reclaimed.
- Checksums (SHA‑256) for archived bundles and a verification manifest.
- Audit log entries with operator, timestamp and approval token (supports OIDC SSO integration).
- Revert plan enabling safe restoration from archive within defined SLAs.
Integrations & Notifications
Cleanup can emit events to monitoring stacks and ticketing systems:
- Prometheus metrics and Grafana dashboard panels for run status and reclaimed space.
- Webhook notifications to ticketing (Jira, ServiceNow) and collaboration channels (Slack, MS Teams).
- Automatic creation of incident tickets for failed verification or archive storage issues.
Safe operations & Best practices
- Always run in dry‑run mode for new rules and schedule apply runs during maintenance windows.
- Use owner whitelists to prevent accidental cleanup of business‑critical tags.
- Test restore procedures regularly — archived data must be recoverable within SLAs.
- Keep multiple retention tiers and stagger purges to avoid large simultaneous restores.
Example CLI usage
# Run a dry‑run with rule set cleanupctl scan --config /etc/cleanup/config.yaml --rules /etc/cleanup/rules.d/cleanup-rules.yaml --mode dry-run --output plan-2026-02-06.json Apply the approved plan cleanupctl apply --plan plan-2026-02-06.json --approve-token ABC123 --mode apply Verify archived bundle cleanupctl verify --archive s3://archives/scada/2026-02-06-bundle.tar.gz --checksum 9f2a...abcd
Deployment & Support
Cleanup ships as a CLI and a scheduled service with YAML configuration. We provide templates for scheduled housekeeping, audit export hooks, and runbooks as part of our service plans.
audit export hooks, and runbooks as part of our service plans.
Download Cleanup Service PlansContact Ops
Abil’I.T. — Cleanup
Contact: info@abilit.eu