bert.hausmans
ddacf9363e
Initial observability architecture document. Skeleton with §3
(\$dontReport exception list) as the only concrete section. Other
sections are structured placeholders for WS-7 sessie 1 decisions:
- §1 Logging strategy (log levels, criteria)
- §2 Sentry decisions (SDK config, sample rates, breadcrumbs,
release tagging)
- §3 \$dontReport exceptions (concrete) — three classes that are
expected business outcomes, not bugs:
* PublishGuardViolationException (422 publish-time)
* PurposeRequirementsNotMetException (422)
* IdempotencyConflictException (409)
With explicit out-of-scope rationale for the three runtime
pipeline exceptions that DO go to Sentry (PersonProvisioning /
PurposeSubjectResolution / FormBindingApplicator) — engineering
needs cross-org visibility into systemic patterns even when
org admins handle individual failures via the WS-6 admin UI.
- §4 Structured logging conventions (key naming tree)
- §5 Metrics (counters, histograms)
- §6 Alerting rules (thresholds, routing)
- §7 Dashboards (panel layout)
The skeleton ensures WS-7 starts from a clear scope; the concrete
\$dontReport list closes a real Sentry-noise gap immediately
(PublishGuardViolationException etc. should never have hit Sentry).
RFC-WS-6.md §9 cross-references the new doc and adds an
Observability follow-up row.
Refs: WS-6 sessie 3b Task 5, WS-7 (forward)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
122 lines
4.6 KiB
Markdown
122 lines
4.6 KiB
Markdown
# ARCH-OBSERVABILITY
|
|
|
|
> Crewli's observability architecture — logging, monitoring, alerting,
|
|
> metrics.
|
|
>
|
|
> Status: SKELETON. Section §3 (`$dontReport`) is concrete; other
|
|
> sections are structured placeholders for WS-7 sessie 1 decisions.
|
|
|
|
## Document history
|
|
|
|
- 2026-04-28 — v0.1 — Initial skeleton (WS-6 sessie 3b). Only §3
|
|
concrete; remainder placeholdered for WS-7.
|
|
|
|
## §1 — Logging strategy
|
|
|
|
[WS-7: define log levels with explicit criteria. Example questions to
|
|
answer in WS-7 sessie 1:
|
|
|
|
- When does code use `Log::error` vs `Log::warning` vs `Log::info`?
|
|
- Are unhandled exceptions automatically `error`?
|
|
- Is `Log::debug` allowed in production, or stripped in deploy?
|
|
- How do structured payload conventions tie to log keys (see §4)?
|
|
]
|
|
|
|
## §2 — Sentry decisions
|
|
|
|
[WS-7: Sentry SDK install + configuration decisions. Skeleton:
|
|
|
|
- Which environments report to Sentry? (dev / staging / production)
|
|
- Sample rate per environment?
|
|
- Source map upload to Sentry CI?
|
|
- User context injection (auth user ID + organisation ID, opt-in
|
|
redaction for PII)?
|
|
- Breadcrumbs strategy (which events generate breadcrumbs)?
|
|
- Release tagging convention (commit SHA? semver? both?)?
|
|
]
|
|
|
|
## §3 — `$dontReport` exceptions (concrete)
|
|
|
|
The following exception classes are **expected business outcomes**,
|
|
not bugs. They are caught and handled in the application; reporting
|
|
them to Sentry would generate noise that drowns the signal.
|
|
|
|
When the Sentry SDK lands (WS-7), add the following classes to
|
|
Laravel's `app/Exceptions/Handler.php` `$dontReport` array:
|
|
|
|
| Class | Reason |
|
|
|---|---|
|
|
| `\App\Exceptions\FormBuilder\PublishGuardViolationException` | Publish-time validation: schema fails a guard. Returned as 422 with field-level errors. Not a system bug. |
|
|
| `\App\Exceptions\FormBuilder\PurposeRequirementsNotMetException` | Schema lacks required bindings for its purpose. Returned as 422. Not a system bug. |
|
|
| `\App\Exceptions\FormBuilder\IdempotencyConflictException` | Duplicate idempotency key on submission. Returned as 409. Not a system bug. |
|
|
|
|
**Out of scope for `$dontReport` (these DO go to Sentry):**
|
|
|
|
- `App\Exceptions\FormBuilder\PersonProvisioningException` — runtime
|
|
failure during the apply pipeline. Caught by
|
|
`ApplyBindingsOnFormSubmit` and recorded as
|
|
`FormSubmissionActionFailure`, but the engineering team needs
|
|
visibility into recurring patterns across orgs.
|
|
- `App\Exceptions\FormBuilder\PurposeSubjectResolutionException` —
|
|
runtime resolution failure (no portal token, no auth user, etc.).
|
|
Same dual-handling rationale: action-failures table for
|
|
org-admin operational handling; Sentry for engineering visibility.
|
|
- `App\Exceptions\FormBuilder\FormBindingApplicatorException` —
|
|
runtime applicator failure (no_transaction, no_schema,
|
|
unknown_purpose). These should never happen in production; if they
|
|
do, they're systemic bugs — Sentry is the correct destination.
|
|
|
|
The dual recording (Sentry + `form_submission_action_failures` table)
|
|
is intentional: org admins fix specific failures via the WS-6 admin
|
|
UI; engineering identifies systemic issues across all orgs via
|
|
Sentry's aggregation.
|
|
|
|
## §4 — Structured logging conventions
|
|
|
|
[WS-7: log key naming convention. Skeleton:
|
|
|
|
- Hierarchical dot-separated namespace tree
|
|
- Existing examples to align with:
|
|
- `form-builder.apply.transaction_rolled_back`
|
|
- `form-builder.identity-match.no_person_subject_post_apply`
|
|
- `form-webhook.delivery.exception`
|
|
|
|
Define the tree formally so future code discovers the right namespace
|
|
deterministically.]
|
|
|
|
## §5 — Metrics
|
|
|
|
[WS-7: which counters / histograms / gauges? Namespace?
|
|
Statsd / Prometheus / OTel flavour? At minimum, candidate metrics:
|
|
|
|
- `form_submissions_total` (counter, tagged by purpose)
|
|
- `form_submission_apply_status` (counter, tagged by status)
|
|
- `form_failures_open` (gauge per org)
|
|
- `retry_attempts_total` (counter, tagged by outcome)
|
|
- `apply_pipeline_duration_seconds` (histogram)
|
|
]
|
|
|
|
## §6 — Alerting rules
|
|
|
|
[WS-7: which thresholds trigger alerts? Where (Slack? PagerDuty?
|
|
Email?). At minimum, candidate alerts:
|
|
|
|
- "Open failures > X for > Y hours"
|
|
- "Apply pipeline error rate > X% in 1h window"
|
|
- "no_transaction guard fired" (immediate alert; should never happen
|
|
in production)
|
|
- "Webhook dead-letter rate > X%"
|
|
]
|
|
|
|
## §7 — Dashboards
|
|
|
|
[WS-7: Grafana / Cloudwatch / similar. Panel layout, widget types,
|
|
default time ranges. Skeleton later.]
|
|
|
|
## Related docs
|
|
|
|
- `RFC-WS-6.md` — WS-6 binding pipeline design (the failures observed
|
|
and recorded by §3's classes originate here)
|
|
- `ARCH-BINDINGS.md` — apply pipeline architecture
|
|
- `ARCH-FORM-BUILDER.md` — form-builder runtime including webhooks
|