What the regulations Say - and the QP Code of Practice

The EU AI Act and the EudraLex Annex 22 (draft) are clear: Human oversight must be real, effective, and documented, not symbolic. The HITL (Human-in-the-loop) must understand AI, warts and all. The UK QP Code of Practice places ultimate legal responsibility on the QP for product certification. That responsibility cannot be delegated to an algorithm. Oversight means being able to challenge, override and document decisions, not rubber-stamp outputs you don’t understand.

If a QP is branded the “HITL" without access to the underlying data, validation evidence, or performance metrics, the company is not protecting compliance - it’s creating regulatory exposure and patient risk.

QP concerns: Lucy had many but here's her top three:

• Accountability without control: She is legally accountable for release decisions but has no documented authority or tools to verify how AI outputs are generated.
• Data integrity risk: AI-enabled equipment (e.g. autoclaves) generate outputs that feed into release. Without audit trails and validation, how can she prove compliance with ALCOA+?
• False sense of security: Management treats “HITL” as a safety net, but without proper training, processes, and challenge mechanisms, the human becomes a fig leaf, not a safeguard. 

Five recommendations to mitigate risk? This is what we came up with:

1.    Define intended use & limits of AI: Each application must have a documented statement of intended use, risk assessment, and validation evidence. AI should assist - not decide - on GMP-critical outcomes.

2.    Mandate transparent oversight: QPs must have access to performance data, drift monitoring and audit trails. Oversight means being able to challenge and override, not just approve.

3.    Competency framework for HITL roles: QPs and QA staff need training in AI literacy: understanding bias, drift and validation principles. Competence must be assessed, not assumed.

4.    Establish governance systems: Create a cross-functional AI oversight board (QA, IT, Engineering) that reviews AI lifecycle management: validation, retraining, and performance monitoring.

5.    Codify “no rubber stamping” in SOPs: Procedures must explicitly require independent evidence review, second-person challenge for borderline cases, and documented rationale for decisions.

And Finally

If regulators walked in tomorrow and asked:

“Show me how your QP provides meaningful oversight of your AI-enabled systems”

would your company pass the test, or would your QP be left holding the liability bag? Human-in-the-Loop is not a slogan. It’s a system of controls, training, and governance. Without it, “HITL” becomes a trap: one that places QPs at unacceptable personal and professional risk.