Skip to content

PRO Release Notes

Release notes for Baldur PRO — the licensed tier. The open-source core has its own changelog in the baldur-framework repository. Each entry describes an operator-visible capability; a PRO license is required.

The format follows Keep a Changelog.

[Unreleased]

Added

  • Runtime Config editor — a web console panel to view and change runtime-tunable settings (retry attempts, circuit-breaker thresholds, DLQ limits, and more) from the browser, with a per-section apply-strategy selector, a current → proposed diff confirmation before applying, an out-of-range clamp notice, and an audit-status badge. Replaces hand-crafting PUT calls with curl or editing env vars and restarting.
  • Runtime-config REST surface — read responses now include a per-section version, and PUT /config/{section} accepts an optional expected_version for optimistic-concurrency control. A stale write returns HTTP 409 with the expected/actual versions and the current config so the client can merge and retry; omitting expected_version preserves the previous unconditional write behavior.
  • DLQ auto-replay arming surface — you can now see at a glance whether automatic replay on circuit-breaker recovery is actually armed, and if not, which prerequisite is missing. A new baldur_dlq_auto_replay_armed gauge, an auto_replay block on GET /dlq/cleanup/stats, and a Web Console DLQ-panel badge name the first missing link (PRO/entitlement absent, on-recovery replay disabled, no Celery worker on the dlq_processing queue, unconfigured service→failure-type map, or no registered replay handler), plus a baldur_dlq_replay_dispatch_total{outcome} counter for each recovery dispatch — so a self-healing loop that would otherwise drain nothing is visible before an incident rather than after.
  • Optional reason on a manual DLQ replay (POST /dlq/replay) and retry (POST /dlq/{id}/retry), with matching Web Console inputs; when provided it is recorded in the entry's resolution note and the audit trail.
  • DLQ trace continuity — a failed operation's trace is now linked to its later replay, so "original failure → DLQ capture → replay" reads as one connected story in your tracing tools instead of disconnected traces. The failing request's trace is captured onto the entry and, on every replay path (targeted retry, batch, automatic on-recovery sweep, force-redrive), surfaced on the replay's log line, folded into its audit record, and — when OpenTelemetry is active — attached as a span link (with a searchable baldur.dlq.origin_trace_id attribute) on a dlq.replay span back to the original failure. The link is additive: the replay keeps its own trigger trace, and the origin is attached alongside. Always on and fail-open — a linkage failure never affects a store or replay.

Changed

  • Replay-automation environment variables and config fields renamed for clarity so each name says what it tunes: BALDUR_REPLAY_AUTOMATION_TRACK1_ENABLED → BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_ENABLED, BALDUR_REPLAY_AUTOMATION_TRACK1_MAX_ITEMS → BALDUR_REPLAY_AUTOMATION_ON_RECOVERY_MAX_ITEMS, BALDUR_REPLAY_AUTOMATION_TRACK2_MAX_ITEMS → BALDUR_REPLAY_AUTOMATION_ADAPTIVE_INITIAL_ITEMS, BALDUR_REPLAY_AUTOMATION_TRACK3_ENABLED → BALDUR_REPLAY_AUTOMATION_TRAFFIC_AWARE_ENABLED, and BALDUR_REPLAY_AUTOMATION_TRACK3_MAX_ITEMS → BALDUR_REPLAY_AUTOMATION_TRAFFIC_AWARE_MAX_ITEMS, along with the matching REST config fields. Defaults are unchanged. Breaking: a stored runtime-config override or environment variable set under an old TRACK-ordinal name is ignored and reverts to the field default — re-set it under the new name.

Removed

  • track2_enabled runtime-config field and the scheduled-batch DLQ replay it advertised. The periodic beat job behind it was inert — it dispatched an incompatible task signature to a queue no worker consumes — and the config field had no backing setting, so setting it never had any effect. Automatic replay on circuit-breaker recovery and manual/operator batch replay are unaffected and remain the DLQ drain paths.
  • HedgingResultValidator / ResultMismatchRecord — removed from the Hedging service. No hedging execution path ever constructed or invoked the validator, so the hedged-result divergence detection it implied never ran; removing it makes the hedging surface honest.

Fixed

  • Error Budget Gate alerts no longer suppress one another. All three gate alert types (fail-open activation, rate-limit exceeded, circuit-breaker open) previously shared one notification-hub cooldown window, so a warning-level fail-open page could swallow a later critical circuit-open page for the rest of the window. Each alert type now cools down under its own hub key, aligned one-to-one with the gate's local per-type cooldown — a missed page stays diagnosable from either layer's suppression log. The gate's default cooldown also now follows the alert_cooldown_seconds setting instead of a hardcoded 300 seconds.
  • Error-budget fail-safe alerts now cool down per component instead of globally. One component's fail-safe alert no longer silences a different component's within the shared window; two threads racing the same alert can no longer both page; and a failed send no longer starts a cooldown that would suppress the immediate retry.
  • Recovery resume retries now apply jitter to their exponential backoff, so simultaneously failed recoveries no longer retry in lockstep. The schedule's base and cap are tunable via BALDUR_RECOVERY_TASKS_RESUME_BACKOFF_BASE_SECONDS / BALDUR_RECOVERY_TASKS_RESUME_BACKOFF_MAX_SECONDS (previously hardcoded 30 s doubling to a 300 s cap).
  • Error-budget-gate configuration changes now survive a restart. A gate config change made through the REST API or console returned success and took effect in memory, but was written under a store key that nothing read back on boot, so the change silently reverted on the next worker restart. The gate now persists to its own runtime-config section — with version tracking and an audit record — so a tuned threshold, cache TTL, or fail-open policy is retained across restarts.
  • Canary config rollouts now actually take effect. A started or promoted canary rollout previously wrote its config change only to per-cluster cache keys that nothing read, so on the single-deployment setup the running process never saw the change — stage health checks evaluated the unchanged system and reported success, and a rollback reverted nothing. Rollout start, stage promotion, and rollback now apply through the runtime-config manager (the same in-process surface as a console config edit), so the change takes effect, is audited, and is faithfully reverted on rollback — including a rollback of the first-ever change on a domain, which restores the values captured at create time. Creating a rollout for an unknown config type or field is now rejected with a clear error naming the valid vocabulary instead of silently applying nothing.
  • A runtime-config edit made while a canary rollout is in progress can no longer be silently clobbered. A rollout holds a per-section lock for its lifetime and reverts the whole section to its pre-rollout snapshot on rollback — so an editor/REST edit to that section mid-rollout would previously apply and then be silently overwritten by the rollback. Such an edit is now rejected with HTTP 409 (error_code: ROLLOUT_CONFLICT, naming the owning rollout) instead; the Web Console editor also shows a locked badge on the section and disables its Apply. The rollout's own start/promote/rollback still apply — only foreign edits are blocked. Enforcement covers every config writer (editor, SLO, governance, drift, chaos-safety, full/section resets, delayed/graceful scheduled changes, and the chaos-component config persists), and the create-time snapshot is now captured under the lock so it cannot miss a concurrent edit. The escape hatch is to roll back or cancel the rollout, or wait for it to finish.
  • Gradual recovery from Emergency Mode no longer reports itself as an unhealthy background worker. Shortly after a gradual recovery started, its worker was misjudged as stalled — turning the Meta-Watchdog daemon_workers health red for the entire recovery, and paging on-call where an escalation channel is configured — even while the recovery was progressing normally. The worker's health is now judged against its actual step cadence, so a healthy recovery stays healthy while a genuinely hung recovery is still detected.
  • Delayed and graceful runtime-config changes now actually apply. Editing a section whose default apply strategy is delayed (circuit breaker, DLQ, retry, idempotency, security) and clicking Apply previously returned "scheduled" but the value never changed and no clamp or audit was recorded; the change now takes effect after its delay, with the out-of-range clamp surfaced and the change audited.
  • Config history rollback — rolling back a saved configuration version now applies correctly for every section (retry, circuit breaker, DLQ, and the rest), instead of failing for every Pydantic-backed section and silently leaving the configuration unchanged.
  • Daily Report — the Slack digest now delivers the full multi-section report instead of a body cut off at 500 characters and padded with unreadable raw-metadata fields; sections for features not in the current release are no longer shown, and operator-defined custom metrics render under their own heading instead of trailing the error list.
  • Audit trail — a delayed or graceful configuration change is now attributed to the operator who requested it, instead of being recorded as an internal worker. The "who changed what" record is now accurate for every configuration change regardless of apply strategy.
  • Audit trail — an audit record that the central store rejects during recovery (e.g. one oversized or invalid entry the store refuses while accepting its batch neighbors) is no longer silently lost. The background sync worker now holds its position at the failed entry, so its local write-ahead-log file is retained and the entry is retried every cycle instead of being skipped and the file deleted. A persistently-rejected entry raises a CRITICAL audit_sync_worker.cursor_stalled alert and sets a new baldur_wal_sync_cursor_stalled gauge (1 while stalled, 0 once it lands) so the operator can fix the rejection at its source; the entry is never auto-discarded.
  • Audit trail — audit records buffered locally during a central-store outage are now delivered correctly when the store recovers. On recovery replay the background sync worker previously handed the store a raw internal record instead of a proper audit entry, which — depending on the configured store — silently discarded the record (hash-chain store), left the backlog permanently undrained (stdout / database store), or wrote it in the wrong on-disk shape (file store). Replayed records are now delivered as real audit entries with their original action and timestamp preserved, so a compliance trail survives a central outage intact.
  • Concurrent runtime-config edits are no longer silently lost. Previously, two operators (or pods) editing the same configuration section within one read→write window resolved last-writer-wins with no signal — the first operator's change vanished. Edits are now guarded by an optimistic-concurrency version: a stale edit is rejected with HTTP 409 (carrying the current version and values to retry against) instead of overwriting a fresher change, and a scheduled/rollback apply that races an unrelated change auto-retries rather than dropping. A pending (delayed/graceful) change created while another change is being applied is also no longer dropped.
  • Traffic-aware DLQ replay is now deliverable. Its scheduled job was routed to a dlq queue that nothing registers on and no worker consumes, so even with the feature enabled it drained nothing; it now targets the canonical dlq_processing queue that the DLQ worker serves. The feature is off by default, so default deployments are unaffected.
  • DLQ auto-replay on circuit-breaker recovery no longer goes silently inert. When it cannot run — no Celery worker consuming the dlq_processing queue, the service_failure_type_map unconfigured, or no replay handler registered for the domain — the recovery now logs a WARNING naming the remediation and the arming surface reports disarmed, instead of a DEBUG-level skip that left operators believing auto-replay was on. The prerequisites for closing the loop are now documented in the DLQ + Replay guide.
  • Replay provenance is now trigger-accurate. Every successful replay was recorded with resolution type auto_replay regardless of how it started, so an operator's console/API replay was indistinguishable from a system-automatic one in the compliance record. Replays now record their actual trigger (manual, circuit-close auto-replay, traffic-aware, throttle-aware, scheduled batch, or DLQ consumer) and the acting principal in the audit trail.
  • BALDUR_REPLAY_AUTOMATION_* settings (on-recovery enable / max-items, and the service→failure-type map) are now honored even when the runtime-config service is not present, instead of falling back to hardcoded defaults that ignored the environment variables.
  • The Web Console Runtime Config editor's remaining settings domains now take effect. Editing the sla, security, idempotency, notification, forensic, or metrics section previously persisted and displayed the new value, but no running consumer read it — so the edit was inert even after a worker restart. Every consumer of these domains now reads the same layered-settings surface the console writes, so an edit is observed on the next read (idempotency within a ~30s per-process cache; a few construction-captured fields — e.g. notification channel targets, idempotency cache-TTLs — still take effect after a worker restart, noted in the runtime-config-change runbook). Separately, 15 editor widgets that had no backing consumer at all — the forensic collection/masking toggles and the security request-counting/failed-login detection fields — were removed from the editor; their BALDUR_* settings and env vars remain.

[1.0.0] - 2026-06-23

The inaugural PRO release. Entries require a PRO license.

Added

  • Bulkhead — isolates each dependency in its own fixed slice of capacity.
  • Dead Letter Queue + Replay — captures operations that fail on a downed dependency with the context needed to re-run them, and replays the backlog once it recovers.
  • Emergency Mode — sheds non-critical traffic in deliberate steps under stress and holds until the system stabilizes.
  • Adaptive Throttle — caps admitted requests and moves the cap up or down automatically as the service speeds up or slows down.
  • Canary Recovery — rolls a configuration change out to a small slice of the fleet first and automatically restores the previous config if the rollout degrades or stalls.
  • Governance — a safety gate every automated recovery action must pass before running, recording why each action was allowed or blocked.
  • Meta-Watchdog — monitors Baldur's own healing subsystems and pages a human if they stall or go quiet.
  • Daily Report — rolls a day of self-healing activity (recoveries, circuit-breaker trips, errors) into a single digest, formatted for Slack.
  • Unified Notification — one hub that routes, deduplicates, rate-limits, and records every alert Baldur raises, delivering to Slack.
  • Audit Trail — tamper-evident, append-only record of every configuration change and healing decision (who, what, when, why) with masked client IPs.