ABSTRACT  

Artificial Intelligence (AI) in healthcare and medical research is here. Large, well-characterized and representative datasets are the foundations of safe and effective AI systems, including AI-enabled medical devices. Fairness in the ways that clinical data are collected, analyzed and shared to train AI models used. In medical devices is consequential for the safety and efficacy of those devices. Regulators, however, do not explicitly consider issues of fairness in evaluating the rigor of clinical evidence used to substantiate device safety and efficacy as part of the regulatory approval process for medical devices. Other ethics and compliance oversight bodies—including institutional review boards (IRB) and data access committees (DAC)—work upstream to ensure ethical data collection and use practices during device development and validation. However, IRB and DAC reviews are rarely, if ever, made available to regulators during pre-market approval. In this paper, we argue for why regulatory approval bodies should be concerned with fairness at the level of training data supporting AI-enabled devices, and how they could integrate fairness assessment into their regulatory decisions. We discuss the opportunities and operational barriers of three possible models for a fairness “check-point” in the regulatory approval process for AI-enabled medical devices. These models build on the extant literature in fair AI and which regulatory bodies could feasibly integrate into existing device application review and approval processes.