Considering an FDA-type Agency for AI

Governing an emerging technology that we still have a lot to learn about isn’t exactly the political challenge lawmakers are necessarily most excited about. Algorithms are beyond the scope of our existing criminal law and tort regulatory systems that we use to establish product liability. They pose unique regulatory problems that current governance structures are ill-equipped to deal with.

Algorithms and AI are challenging to define, primarily due to the diverse methods that can be used to develop these technologies. The go-to definition for artificial intelligence is some variation of an “automated decision system.” During the EU AI Act negotiations, one of the major hurdles was developing a clear definition of AI. Policymakers had to balance between remaining vague to ensure new developments were captured, or risk missing the intended effect due to specificity as the technology rapidly develops. The socio-technical context AI operates in versus the precision required in legislation lends to a struggle between adopting human-based definitions that naturally accommodate progress (e.g. “ability to perform tasks that typically require human intelligence”) and technical capability-based definitions.

As AI increasingly has the potential to be consequential in our lives, stakeholders must be able to explain how the system has been able to deduce its conclusions based on the inputs and data it was trained upon. Much of this technology is plagued by the “black box” problem, where we do not know what happens between the input and output. Having such opacity when AI is used in high-stakes contexts means that AI models and their developers can unknowingly rely on biases and invalid correlations to make predictions and recommendations. Although the technical research community is actively figuring out explainable AI, we are still far from it. Not understanding how models work in detail means it would be difficult to correct erroneous outputs and govern them appropriately.

Another governance problem that algorithms pose is the question of responsibility. Since AI is capable of autonomy, what responsibility do they bear for their actions and decisions? Our criminal and tort legal systems operate on liability and accountability, which are difficult to establish in digital agents. These are just a few of the many questions we need to ask ourselves as we approach governing AI.

This emerging technology is above traditional boundaries, making it challenging to regulate in an era of policy fragmentation and sectoral regulation. As a result, one approach to its governance is “soft law”.

Soft law approaches are decentralized governance mechanisms characterised by their cooperative and voluntary nature, as well as their lack of legal mechanisms for ensuring compliance. There are various types, including codes of practice, principles, strategies and recommendations, standards, certifications, voluntary moratoriums and (some) treaties. For instance, in 2019, the Organisation for Economic Co-operation and Development (OECD) introduced a set of non-binding principles that, ideally, should be incorporated into the AI development process.

Soft law is well-suited for shaping norms, but it struggles to ensure accountability. Despite its optimistic outlook, it cannot be the sole replacement for traditional regulation.

In the realm of traditional regulation, the European Union recently implemented the AI Act. This is one of the first comprehensive AI legislation out there, and it aims to hold malicious actors accountable and prevent the harm that AI can elicit. It takes a risk-based approach, imposing stricter requirements on ‘high-risk’ AI systems. The rules came into force a few days ago (Aug 1, 2024), with implicated companies having to comply with the first set of rules in the next 6 months.

The EU Act is grounded in human rights and enshrines private right of action so that in the age of automation, citizens can still enforce their rights. However, as it usually is with legislation, the implementation is a lengthy process, likely putting many companies in a state of paralysis as they comprehend the legal and bureaucratic complexities.

More recently, the concept of AI Safety Institutes has emerged. This state-backed entity was introduced during the Bletchley Summit in 2023 when the United Kingdom announced the creation of a specialised agency to tackle safety in AI frontier models. Since then, other countries have announced that they will establish their own AI Safety Institute, and cooperate as part of an international network (as signed at the Seoul Summit). It is too early to comment on how this approach fairs, but it is a step in the right direction to increase government capacity on AI safety.

The common thread between the aforementioned governance mechanisms is that they are typically tort-based and consent-based approaches. When it comes to rapidly developing technologies, enforcement of laws might be too belated. Investigations and enforcement only come after harm has already been done, or there is suspicion of wrong-doing. Criminal law and tort regulatory systems are not enough for complex algorithms.

Given the explosive nature and ubiquity of AI, an ex-ante model could be used. Before an AI model is disseminated, the burden of proof can be shifted to the developers. They must be able to demonstrate safety and establish transparency. Policy thinkers have proposed the establishment of an agency similar to the FDA to carry out the auditing tasks required for an ex-ante approach.

Firstly, how does the FDA work? The U.S. Food and Drug Administration (FDA) regulates medical drugs and devices through risk-based criteria. The higher the risk level, the more extensive the evaluations and monitoring will be at various steps of the development and deployment process.

The agency has extensive auditing powers, with statutory powers to access detailed data about the drug or device (e.g. clinical trial data). If an entity does not comply with regulatory requirements, the FDA has the authority to issue fines and sanctions. Throughout the stages of development, the FDA has direct and continuous engagement with medical developers until the product goes to market.

Broadly speaking, these are the stages of the FDA oversight process (using the example of a medical drug):

1. Basic research — discovery and development
2. Pre-clinical research — the drug is tested on animals to understand toxicity, efficacy and safety. The developer then submits an Investigational New Drug (IND) Application to the FDA.
3. Clinical research — once the IND Application is approved, there are three phases of clinical trials.

Phase I: testing the drug on a small group of people (<100) to learn about safety and identify side effects

Phase II: a larger group of people (<500) to determine effectiveness and further study the drug.

Phase III: a large group of people (<5000) to confirm effectiveness, monitor side effects and compare with similar treatments.

4. FDA approval — the agency reviews the data from the clinical trial and, depending on the data.

To achieve FDA-style oversight of artificial intelligence models, a dedicated government agency would likely need to be created. Fortunately, AI Safety Institutes have recently emerged as an entity with this speciality, so perhaps this form of oversight could be added to their remit.

Similarly to the FDA, this agency would also be in continuous engagement with the developer of frontier AI models across the development lifecycle. Approval gates would be introduced at various stages of development (pre-market) so that harm, validity, and reliability of the model are established before moving on to the next stage.

In the design stage, disclosure requirements could be imposed on developers of frontier models to share information on what datasets are being used. Biased or inappropriate datasets result in models that are unable to accomplish the objectives they were designed for. By engaging the developer with an institute about their data, the necessary steps can be taken to ensure reliable and appropriate sets of consensual data are being used in development.

Once training runs are complete, a pre-market approval gate should be introduced. This would require the developer to share them with the agency, which will determine if the model’s proof of safety, efficacy and validity is sufficient to move on to deployment. During this step, sandbox testing and audits from third-party experts can occur. After receiving feedback and approval from the agency, the developer can fine-tune the model before deployment.

Post-market monitoring is also crucial, and infrastructure would need to be created to establish a reporting tool for adverse events and complaints. The agency could have the authority to suspend models if vulnerabilities or harms are too severe, requiring the developer to solve the issues or retract the model from use.

Labelling food and drugs is also part of the FDA’s remit. This AI agency could introduce labelling requirements for frontier models and their third-party integrations. As labels for foods and drugs help the user make informed choices about what they can consume and what side effects can occur, developers could create labels that highlight the dataset used to train the model, the intended purpose, and the typical vulnerabilities/false results the model can create.

Considerations of the intended audience, complexity of information and amount of information provided need to be discussed within a working group, before such a requirement is imposed.

The idea of an FDA-type agency has been discussed previously by the Ada Lovelace Institute and other scholars. However, implementing a labelling requirement is an aspect I have not come across before. I have very briefly touched upon it here and aim to continue diving into this concept to discover whether it is a feasible policy. What is clear is that ex-ante approaches to governing AI need to be explored as a mechanism to ensure responsible innovation.