Rate Watcher — Rate‑Limit, Throttling & Anomaly Detection

A focused service for detecting abnormal request/operation rates, enforcing rate limits and triggering automated or human responses. Designed to protect APIs, probes and critical endpoints from bursts, floods and slow‑burn anomalies.

How Rate Watcher works

Rate Watcher ingests request events or aggregated counters (from gateway logs, sidecars or Prometheus) and evaluates them against configured rules and learned baselines. It supports:

• Multiple keying strategies (API key, client IP, user ID, route).
• Sliding window and fixed bucket algorithms for short and long windows (e.g., 1s, 10s, 1m, 1h).
• Exemptions and whitelists for internal systems and critical clients.
• Rate‑limit policies with tiered actions and automatic cooldown timers.
• Behavioral anomaly detection that compares current rate to historic baseline and seasonality profile.

Example policies & configuration

Use policy templates to quickly apply standard protections and tune per service.

# Example policy (YAML) policies: - id: api_public_default key: api_key windows: - window: 1s limit: 20 action: soft-throttle - window: 1m limit: 1000 action: monitor baseline_learning: 14d exempt_clients: ["internal-service-1"] 

Adaptive policy example: start in monitor mode for 7 days to learn baseline, then enforce soft‑throttle if burst profile exceeds 3× normal.

Integration & enforcement

• API Gateway: use Rate Watcher as an external policy engine (RP calls) or as a sidecar for local enforcement.
• WAF & LB: push decisions (block/throttle) to WAF rules or LB ACLs for immediate action.
• Watchdog: when abnormal sustained rates are detected, trigger Watchdog maintenance mode or automated scaling playbook.
• Incident systems: open tickets in ServiceNow / PagerDuty when P0 thresholds hit.

Metrics & dashboards

Expose these metrics to Prometheus for dashboards and alerting:

• ratewatcher_requests_total{policy,key,action}
• ratewatcher_throttled_total{policy,key}
• ratewatcher_blocked_total{policy,key}
• ratewatcher_baseline_deviation{policy,window}
• ratewatcher_policy_eval_duration_seconds{policy}

Suggested Grafana panels: top throttled clients, policy hit heatmap, baseline vs actual rate overlays, throttling impact on latency.

Alerting & runbooks

Define alert severities for burst vs sustained anomalies. Example rules:

• P0 — sustained blocked rate > X% of total traffic for > 5m → page on‑call, trigger Watchdog.
• P1 — burst above 10× baseline for specific key → open ticket to investigate client misbehaviour.
• P2 — repeated soft‑throttle events for non‑exempt client → notify team via Slack/email.

Operational best practices

• Start in monitor mode to collect 7–14 days of baseline data before enforcing hard limits.
• Maintain owner whitelists and emergency bypass tokens for critical traffic.
• Provide clear client feedback headers (Retry‑After, X‑RateLimit‑Reset) and API docs about limits.
• Automate cooldown and auto‑unblock after verification windows to reduce manual toil.
• Periodically review and tune policies based on seasonal patterns (business hours, campaigns).

Security & compliance

Keep audit trails of enforcement decisions and store them in tamper‑evident logs for investigations and compliance. Mask PII in logs and follow retention rules.

Deployment & scaling

• Scale horizontally: shard keyspace by hash (e.g., key → shard) and use consistent hashing for sticky routing.
• Use local caching for ultra‑low latency enforcement and eventual consistency to the central policy store.
• Provide high‑availability policy store (etcd/consul) and replicate learning models across nodes.

CLI examples

# Scan incoming logs and push counters to Rate Watcher ratewatcherctl ingest --source /var/log/gateway/access.log --format nginx --policy-map /etc/ratewatcher/policies.yaml
Evaluate a specific key against policies (dry-run)

ratewatcherctl eval --policy api_public_default --key "api-key-123" --window 1m --count 1500
Apply a temporary block on a client

ratewatcherctl action block --key "bad-client-ip" --duration 3600 --reason "sustained flood"

Abil’I.T. — Rate Watcher

Contact: ops@abilit.eu