crewli/dev-docs/form-builder-migration-playbook.md

# Form Builder — Migration Playbook

> Operator runbook for migrating legacy `registration_form_fields` /
> `person_field_values` / `registration_field_templates` data into the new
> `form_*` structure. Companion to `/dev-docs/ARCH-FORM-BUILDER.md` §11.

---

## Status (post-S1)

**Legacy tables retained intentionally.** The tables `registration_form_fields`,
`person_field_values`, and `registration_field_templates` remain in the
schema through end of S1. They are dropped atomically in the first S2
commit together with the removal of legacy controllers, services, requests,
resources, policies, and routes. DevSeeder and FormBuilderDevSeeder no
longer write to them; they hold zero rows in dev but the schema is preserved
for environments with real legacy data that will be migrated via
`forms:migrate-legacy-data`.

Until S2 lands, the legacy registration UI remains functional for any
environment that depends on it.

---

## When to use this playbook

Run this when promoting an environment that holds real legacy
registration data into the new form-builder schema. For development
machines starting from `migrate:fresh --seed`, no migration is needed —
DevSeeder produces new-structure data directly.

---

## Pre-flight audit

Before touching any data, confirm the working state matches expectations.

```bash
# 1. Working tree clean?
git status

# 2. On the expected branch?
git log -1 --oneline

# 3. Database backup exists? (production / staging only — your call for dev)

# 4. Verify the new tables are present and the legacy tables still hold data
php artisan db:show | grep -E "form_|registration_form_fields|person_field_values|registration_field_templates"
```

If the legacy tables are absent, the migration command will detect this
and exit cleanly without doing work — there is nothing to migrate.

---

## Step 1 — Dry run

Always start with a dry run. It writes nothing and prints the same summary
table the real run would produce.

```bash
php artisan forms:migrate-legacy-data --dry-run
```

**What to look for:**
- The per-organisation log lines list every event that will get a new
  schema and the number of fields/submissions per event.
- The summary table shows projected counts for `form_schemas`,
  `form_fields`, `form_submissions`, `form_values`, `form_templates`.
- Exit code is `0`.

If anything looks surprising — wildly off counts, organisations missing,
templates duplicated — STOP and investigate. Do not proceed to step 2 until
the dry-run output is what you expect.

---

## Step 2 — Run the migration

```bash
php artisan forms:migrate-legacy-data
```

**What happens:**
- Per organisation, inside a transaction (one bad org rolls back only itself).
- Per event in that org: one `form_schemas` row + N `form_fields` rows.
- Per person with `person_field_values` for that event: one `form_submissions`
  row (status = `submitted` if person.status ∈ applied/approved/no_show
  else `draft`) + one `form_values` row per legacy value.
- `FormValueObserver` populates typed columns and the
  `form_value_options` pivot during the inserts (per ARCH §7.2).
- All `registration_field_templates` rows are wrapped in the ARCH §4.6.1
  `schema_snapshot` shape and inserted as `form_templates`.
- The whole run is wrapped in `App\Support\ActivityLog::suppressed(...)`
  so the activity log isn't flooded by hundreds of `logSchemaChange` /
  `logFieldChange` entries from the bulk inserts.
- After migration completes, verification runs automatically. Final
  exit code is whatever `forms:verify-data-integrity` returns.

**Idempotent:** if a `form_schemas` row already exists for `(org, event,
event_registration)`, that event is skipped silently. Safe to re-run.

---

## Step 3 — Verify

Verification runs automatically at the end of step 2. To re-run it
standalone (or to verify state without touching data):

```bash
php artisan forms:verify-data-integrity
```

Or, equivalently:

```bash
php artisan forms:migrate-legacy-data --verify-only
```

**Nine checks:**
1. Schema coherence (purpose enum, custom_purpose_slug ⇄ purpose=custom, public_token uniqueness)
2. Field coherence (FK valid, field_type ∈ FormFieldType ∪ custom registry, is_filterable matches type, slug unique within schema, binding JSON valid against `config/form_binding.php`)
3. Submission coherence (subject_type/subject_id consistency, subject_type ∈ `config/form_subjects.php`, status ⇄ submitted_at)
4. Value coherence (orphans, unique pair, length, multi-value sanity, value_indexed only set when field.is_filterable)
5. User profile coherence (every user has one profile; reliability_score ∈ [0, 5])
6. Data migration counts (skipped when legacy tables absent)
7. Orphan records (pivot rows, section statuses)
8. Section/schema relation consistency
9. (`--strict`) Strict reachability: value.field.schema must equal value.submission.schema

Exit code `0` means all checks passed. Anything else is a real failure
worth investigating before proceeding.

---

## Step 4 — Spot-check a sample

After verification is green, eyeball a sample submission to confirm the
shape is right.

```bash
php artisan tinker
>>> use App\Models\FormBuilder\FormSubmission;
>>> $sub = FormSubmission::with(['schema', 'values.field'])->latest()->first();
>>> $sub->schema->name;
>>> $sub->values->map(fn($v) => [$v->field->slug => $v->value]);
```

Compare against the legacy view of the same person's data in
`person_field_values` to confirm nothing was silently lost.

---

## Rollback

The data-migration command does not have a `down()` step. If a migration
went wrong, the recovery path is:

1. Restore from backup (production / staging).
2. For dev: `php artisan migrate:fresh --seed` rebuilds everything from scratch.

The legacy tables are preserved during the migration — they are not
modified. So even after running `forms:migrate-legacy-data`, the legacy
data is still there (until S2 drops the tables). This means a production
data-migration error can be recovered by truncating the new form_* tables
and re-running with the underlying legacy data still intact.

---

## Common gotchas

- **`registration_form_fields` table absent:** the command detects this
  and exits 0 silently. Expected behaviour for fresh environments.
- **PII heuristic miss:** the heuristic in §11.2.1 of the ARCH covers
  obvious patterns (email, phone, geboort, allergie, …) but is not
  exhaustive. After migration, organisers can fine-tune `is_pii` per
  field via the form-builder UI (S5).
- **`is_published` derived from event status:** the command considers an
  event "published" when its status is one of registration_open / buildup
  / showday / teardown / closed. If the legacy registration was open on
  events with other statuses, adjust `is_published` manually after migration.
- **Custom field types:** the command maps the 10 legacy field types
  (text/textarea/select/multiselect/checkbox/radio/boolean/number/tag_picker/heading)
  into the new uppercase enum values. If your legacy data has values
  outside this set, the command will set them to `TEXT` and the verifier
  will flag them as invalid `field_type`.