Eleven days in Wichita: what a DAL-A dry run really costs

Eleven days, six engineers, one conference room and a rule that nobody was allowed to defend their own work. It is the most expensive thing we do voluntarily, and we have never regretted it.

A stage-of-involvement review is where a certification authority looks at your evidence and decides whether it supports the claim you are making. On a DAL-A programme there are four of them. Failing one does not end a programme, but it does mean re-work, a re-review, and a schedule that moves in months rather than weeks.

Before every one of ours, we run the whole thing internally first.

How the rehearsal is set up

Three of our engineers play the authority. They are chosen specifically because they did not work on the artefacts under review, and for the duration they are not on the programme — they report to our head of quality, not to the programme manager. Their job is to fail us.

The other three present the evidence exactly as they would to the real authority: same pack, same order, same level of preparation. No warm-up, no pre-agreed list of questions.

The rule that makes it work: the people who wrote the artefact are not allowed to explain it. If the evidence does not stand up on its own, it does not stand up.

That last rule is the whole exercise. In a real review the author is often in the room, and it is very easy for a gap in the documentation to be papered over by somebody who knows the answer. The authority does not get that person for the next twenty years. Neither does the maintainer.

Nine findings

Across eleven days the internal panel raised nine findings. Seven were what you would expect and want: traceability links that pointed at a superseded requirement revision, two test cases whose pass criteria were written loosely enough to be arguable, a tool qualification record missing its version stamp. All real, all cheap to fix at that point, all embarrassing in front of an authority.

One was procedural and genuinely useful — our review records did not consistently show who had performed the independence check, only that it had been performed. Easy to fix, and exactly the kind of thing that turns into a systemic finding if an auditor pulls three examples and gets three different formats.

And then the ninth

The ninth finding was raised on day eight by an engineer who had joined the company four months earlier, and it was raised because she did not yet know what everybody else had stopped seeing.

She asked why a particular low-level requirement existed. It described a behaviour in the lane isolation logic — a specific timing relationship — and it traced correctly up to a high-level requirement and down to two test cases. Everything about it was structurally correct.

Nobody in the room could explain what it was for.

It traced. It was tested. It passed. And not one of us could say what it was actually protecting against.Marcus Adeyemi · Certification Manager

It took two days to reconstruct. The requirement had been written in 2019 to accommodate a sensor with a particular startup transient. That sensor had been replaced in a hardware revision in 2022. The requirement had survived because it traced cleanly and its tests passed, and because a requirement that passes its tests attracts no attention.

It was not harmful. But it constrained the isolation timing in a way that no longer had a reason, and we would have been defending it to an authority as though it did. The honest answer — that it was a fossil — is not one you want to give live, because the immediate next question is how many others there are.

So we went looking

We audited every low-level requirement in the lane isolation and voting logic against a single question: what happens if this is not true? Two hundred and forty requirements. Three more fossils, all from the same hardware revision, all structurally perfect and semantically empty.

Removing them took five weeks including re-verification. Doing that work after a finding at a real review would have taken considerably longer, under scrutiny, with the authority reasonably wondering what else was in there.

What it costs

Eleven days of six engineers, plus the five weeks of remediation. On a programme of that size, roughly 3% of the certification budget.

We have run this on every DAL-A programme since 2016 and it has never once found nothing. The findings get smaller each time, which we take as the process working rather than as a reason to stop.

The argument for it is not really about audit outcomes. It is that a requirement nobody can justify is a requirement that will be maintained, tested and defended for twenty years by people who also cannot justify it. The review is just when you find out.