Enhancing regulatory affairs in the market placing of new medical devices: how LLMs like ChatGPT may support and simplify processes

Introduction

The medical device industry is facing increasing regulatory complexity following the implementation of Regulation (EU) 2017/745 (Medical Device Regulation, MDR), which requires strict regulatory compliance to demonstrate the efficacy and safety of medical devices that are to be made available on the market or are already on the market, including the CE marking process. This process needs significant human and time resources, not only for the drafting of detailed documentation and the execution of thorough clinical evaluations but also for conducting post-market surveillance and other management processes (1).

Simultaneously, the advent of Artificial Intelligence (AI) is revolutionizing various fields, offering opportunities for efficiency and innovation. Among AI technologies, Large Language Models (LLMs) like ChatGPT promise automation and simplification of complex processes.

This article explores the potential of LLMs in supporting regulatory affairs in the medical device field by critically evaluating their effectiveness, emphasizing prompt engineering, and highlighting the need for user training to ensure a performant approach to AI.

Current regulatory scenario

More than three years have passed since the European Regulation 2017/745 came into force. In the initial months, medical device stakeholders had little certainty and many doubts about how to comply with the new regulatory framework. Today, all agree that it requires increasingly stringent and impactful requirements to demonstrate and ensure the efficacy and safety of medical devices (1-3, 6).

Primarily, manufacturers of medical devices were required to verify the state of the art of their existing products on the market to fill gaps resulting from incomplete technical and clinical data characterizing the device. Even more so for newly designed medical devices, the main activity was understanding how to apply the MDR to enable the CE marking of the product.

The requirements dictated by Regulation 2017/745 have been raised to such an extent that they sometimes necessitate a new classification of the device based on rules and risk classes, a new drafting of technical documentation to fully meet Annex II and Annex III of the MDR, and an adequacy check of the clinical evaluation of the device, often resulting in substantial revision due to a lack of performance, safety, and benefit-risk evidence and data. This includes risk analysis, usability of the device, and post-market surveillance process that must be actively conducted on the market by the manufacturer to verify whether the device achieves the intended use defined (1-7).

This significant effort, multiplied by the number of medical devices in the manufacturer’s portfolio, has a strong impact in terms of time, resources, and consequently money, especially for small and medium-sized enterprises (SMEs) with limited investment capacity.

There is no doubt that the acquisition of skills, whether through internal resources or external qualified experts, is a source of optimization. Being competent, aware, and having high-level expertise improves timelines and makes processes, including the CE marking of a medical device, more linear and logical, thus shorter in reaching the set goal.

Certainly, other methods, tools, and resources can complement the manufacturer or stakeholders in the field to facilitate the market placing or maintenance of a medical device. New methods are under evaluation and study in terms of applicability.

AI and regulatory affairs: a synergistic approach

Faced with the complexity of MDR requirements, AI, particularly LLMs, can play a role in supporting the implementation of regulatory processes by enhancing their efficiency and compliance wherever possible (8).

The main activities required from the manufacturer for the market placing of a medical device can be identified as follows:

1. Drafting the technical documentation of the medical device: the generation and review of technical documentation can be time-consuming (1). ChatGPT could assist in drafting detailed descriptions of devices, risk management plans, general safety and performance requirements checklist, and technical reports (8).
2. Generating risk analysis of the device (1): LLMs could support manufacturers in identifying risks associated with the medical device and in pinpointing risk mitigation measures (8).
3. Drafting and approval of the clinical evaluation of the medical device (2,4,5): LLMs could aid in collecting clinical data from scientific literature to generate clinical evaluation reports. They could summarize findings and highlight key points (8).
4. Conducting usability verification of the medical device (7): ChatGPT could generate a usability test based on precise prompts that direct it to evaluate the specific use of the medical device (8).
5. Implementation of post-market surveillance processes (1,6): LLMs could assist in creating survey questionnaires as a method for collecting efficacy and safety data on the medical device directly from market users (8).

Tests have been conducted to evaluate the use of AI tools (ChatGPT, LLMs) in some of the aforementioned regulatory activities to assess their ability to function as regulatory expertise, support it, or, conversely, where they may fail to provide satisfactory performance.

The LLMs: what ChatGPT is and how it works

ChatGPT is an advanced language model developed by OpenAI, capable of understanding and generating natural text, facilitating many activities traditionally performed by specialists. ChatGPT is based on the Generative Pre-trained Transformer (GPT), a class of language models that uses deep neural networks to understand and generate text (9). The power of GPT lies in its architecture, which leverages language transformations in geometry and vector calculation to process information efficiently and flexibly (Fig. 1).

Figure 1 illustrates the basis of GPT’s functioning: the ability to model natural language into geometric representations. Each word or phrase is converted into vectors, that is, points in a multidimensional space that capture the meaning and relationships between terms. This process, known as embedding, allows the model to manipulate and analyze language mathematically (Fig. 2). For example, words with similar meanings tend to be close in vector space, while words with different meanings are further apart. This approach enables GPT to understand the context and nuances of human language, improving the precision of generated responses.

The attention mechanism

A key component of GPT is the attention mechanism: it allows the model to focus on specific parts of the input text while generating a response. Attention helps identify and prioritize relevant information, enhancing the coherence and relevance of responses. The mechanism operates through a series of weights that determine the importance of each word relative to others. These weights are dynamically calculated during text processing, allowing GPT to adapt to different situations and linguistic contexts (Fig. 3). Figure 3 shows the SoftMax function, which underpins the mathematical encoding mechanism of attention.

As shown in Figure 3, Self-Attention is a fundamental mechanism in Transformer models that allows the model to weigh the importance of each word in a sequence relative to the others during natural language processing. Here is a brief explanation of the vectors involved:

The self-attention mechanism allows the model to capture complex and dynamic relationships between words in a sequence, enhancing context understanding compared to other traditional methods (9,10).

Differences from rule-based systems

Old natural language processing systems were based on predefined rules manually created by developers. These systems worked well for specific tasks but were rigid and difficult to adapt to new contexts. The rules needed continuous updating and expansion to handle new scenarios, requiring significant human effort.

In contrast, ChatGPT uses machine learning to extract knowledge from large amounts of textual data. This approach allows the model to generalize better and manage a variety of tasks without the need for continuous manual intervention (10). The flexibility and continuous learning capability of GPT make it possible to support many activities in regulatory affairs, ranging from document drafting to clinical data research and analysis.

Token consumption: optimization, limits, and methods

One of the crucial aspects of using ChatGPT is managing token consumption. Each request to GPT is processed based on a number of tokens, where a token can be a character, a word, or part of a word. GPT models have a maximum token limit they can handle in a single request, usually ranging from 2048 to 4096 tokens.

When processing large volumes of documents, the token limit can become a significant issue. If the text exceeds the token limit, responses may be partial or incomplete.

There are various methods to optimize token consumption and mitigate associated problems. Some of these include:

1. Text segmentation: Divide the document into smaller segments that fall within the token limit. Each segment can be processed separately and then combined to obtain the complete response.
2. Optimized prompting: Create more concise and specific prompts to reduce the number of tokens used in the request. This requires strategic formulation of questions to maximize efficiency.
3. Iteration and synthesis: Use partial responses to iteratively build a complete answer. This method involves requesting multiple successive responses and synthesizing the obtained information.
4. Using more advanced models: Adopt newer and more advanced versions of GPT models that may have a higher token capacity or better processing efficiency.

Methodology for evaluating ChatGPT

To assess the comprehensiveness and reliability of LLMs like ChatGPT in processes, it is essential to use a structured and rigorous methodology. This approach involves comparing the outputs generated by ChatGPT with those produced by industry experts in selected activities.

Training and optimization

Based on the results, develop a training program for users that includes:

Prompt engineering and output quality

Prompt engineering, or the art of creating precise and detailed input requests for language models like ChatGPT, is a crucial element that directly impacts the quality of the generated outputs. Good prompt engineering can make the difference between a vague, irrelevant output and a precise, useful response. Below are the fundamental principles of prompt engineering and how it can be optimized to improve output quality.

Fundamental principles of prompt engineering

Constraints and guidelines

− Setting specific constraints and guidelines can improve the accuracy of responses. For example, if concise answers are required, specify this clearly in the prompt. “Provide a brief description (maximum 150 words) of the main features of the medical device XYZ.”

Optimization of prompts for token consumption

Optimizing token consumption is a critical aspect, especially when dealing with large volumes of documents. Here are some strategies to optimize prompts:

Evaluation of output quality

To evaluate the quality of outputs generated by optimized prompts, it is important to use the following criteria:

1. Relevance: Is the output relevant to the request?
2. Completeness: Does the output cover all required aspects?
3. Clarity: Is the output clear and understandable?
4. Accuracy: Is the output correct and free of errors?
5. Comprehensiveness: Does the output provide an exhaustive answer?

AI applied to regulatory affairs: two examples based on MDR requirements

Prompt

“Write a complete survey that aims at collecting all necessary information regarding safety and performance for patients using a medical device such as a vaginal cream according to its intended use, that is, a gynecological cream with a barrier effect that helps restore the physiological pH of the vagina and allows the recovery of optimal conditions for the proliferation of natural lactobacilli. The survey needs to report an evaluation of the efficacy of the medical device in treating signs and symptoms before and after the application of the medical device.”

The survey questionnaire generated by ChatGPT begins by asking the user of the medical device about their age and general health conditions. Then, it requests information on which symptoms and their intensity the patient experienced before and after using the gynecological cream, as well as any manifestation of adverse events or new symptoms.

The survey concludes with a satisfaction judgment on the effectiveness of the medical device and whether it should be recommended to others. It is evident that AI was able to support the generation of a survey for the device of interest, covering most of the effectiveness and safety topics required for collecting clinical data on the medical device in the PMS process.

However, there is a lack of quantitative characterization of the severity level of the symptoms addressed by the cream, as ChatGPT provides, in this case, a descriptive-qualitative severity attribution of the symptoms based on a single prompt (Fig. 4), thus not allowing for statistical analysis of the collected data.

Regarding the safety questions, ChatGPT correctly suggests that the user should indicate any adverse events or new symptoms that may have occurred during or after treatment with the gynecological cream (Fig. 5).

Based on the material produced, the manufacturer of the medical device must review the survey suggested by AI through a detailed literary and statistical analysis to make the questionnaire suitable for generating the expected clinical performance and safety data. The AI output can, therefore, be given an overall rating of “Good” (Fig. 6), considering that in terms of relevance, it is pertinent to the request; in terms of completeness, it lacks some aspects as mentioned above, but it is clear, understandable, and sufficiently comprehensive.

Usability

With the MDR, usability has become an essential requirement for obtaining CE marking of a medical device. The manufacturer must ensure that the device is effective and safe, and that the risks associated with its use are acceptable when compared to clinical benefits. The manufacturer must assess potential risks and errors that may occur during the intended use of the medical device and reasonably foreseeable misuse. For this reason, usability, which aims to protect the patient, has become an important element whose application allows for easy and appropriate use of the product in terms of user interface and confirmation of the effectiveness of labeling concerning readability and comprehension.

Based on the current state of the art, usability is governed by the IEC 62366 Standard “Application of usability engineering to medical devices,” which states that the manufacturer’s definition of usability engineering leads to the establishment of requirements that, if correctly implemented, can increase the likelihood that users will use the device without errors.

To evaluate, analyze, and improve the usability of the medical device, the manufacturer performs a usability test, which consists of a series of questions directed at the user about the ease of use of the product based on the provided instructions for use (IFU) (1,7).

A usability test developed and submitted to users of a medical device, such as a polyurethane breast implant, is one of the case studies used to apply AI to the regulatory usability requirement of a medical device.

The breast implant acts as a temporary replacement for removed tissue and supports the growth of autologous tissue in patients. The device, intended for use in adult female subjects, is used in patients who wish to undergo conservative and reconstructive breast surgery to restore the natural appearance of the breast.

The device is intended for use by qualified medical-surgical personnel who are provided with a usability questionnaire. The questionnaire starts by asking about the clarity of consulting the package labeling, as well as the ease of opening, and continues by investigating the convenience and ease of extraction from the packaging and hydration of the device. It includes questions on the usability of the breast implant during surgery in terms of positioning, securing, and covering the device, as well as during potential removal or explant.

The questionnaire includes a section dedicated to collecting feedback on the clarity of the information provided by the manufacturer to the user regarding packaging and sterilization, materiovigilance, storage, disposal, warnings, precautions, adverse effects, residual risks, and complaints.

The questionnaire concludes with a request for an overall satisfaction rating on the clarity of the IFU and suggestions for their improvement.

Following the creation of the usability questionnaire that meets regulatory requirements, the case study continued with the generation using AI, which was given the prompt to develop a usability test for a breast implant.

Prompt (1st)

“I am attaching the file, which is the IFU of a medical device. Can you write a usability questionnaire, that is, a series of questions that allow us to evaluate the likelihood of human error or misuse of the device due to a lack of clarity in the IFU or difficulty in using the device according to the instructions provided by the manufacturer?”

To provide ChatGPT with adequate context to guide the generation process, the IFU document of the device was attached as a knowledge element to the prompt, requesting the creation of a usability questionnaire.

It is essential to share the IFU with the AI, considering that the objective of the usability test is to verify the clarity and comprehension of the instructions provided to the user of the specific device, in order to revise them if these lead to errors and, consequently, misuse during the test.

The usability questionnaire generated by ChatGPT remains generic without delving into the expected specifics of the medical device’s use. It formulates questions on the clarity and completeness of the instructions (Fig. 7), as well as on the ease of use of the device, without asking the user to effectively verify the feasibility, in a clear and understandable manner, of all the steps described in the IFU of the device.

The AI suggests additional questions, including the frequency of potential errors, the presence or absence of a troubleshooting section in the instructions, and an overall satisfaction rating.

Therefore, in response to generic and unfocused answers, it was necessary to specify the need with a second prompt.

Prompt (2nd)

“Can you write the questionnaire with a more specific approach, that is, include questions that follow the step-by-step use of the device to objectively determine if the IFU are generally clear enough to prevent misuse?”

In response to this prompt, ChatGPT was able to pose questions related to the preparation for use, including the identification of all device and packaging components, and aspects of the actual use of the device, such as the clarity of each step described in the instructions and the ease of execution of the main operation intended with the device (Fig. 8). The general usability evaluation questions and possible improvement suggestions remain the same as those from the first prompt.

The inclusion of the second prompt resulted in the loss of some responses obtained with the first prompt and the introduction of new ones. Even if the integration of the two outputs were considered, the expected result would not be achieved, namely usability that is compliant and comprehensive according to regulatory requirements.

It was decided to continue querying ChatGPT by providing additional knowledge, specifically an explanation from a regulatory expert on what a usability test for a medical device is and its purpose.

Prompt (3rd)

“Please be much more specific and detailed and delve into the individual functionalities of the device. Before placing the device on the market, the medical device manufacturer must ensure that the IFU provided with the device are clear to the user. To do this, the manufacturer subjects the device to a test by selecting 15 volunteers, who are the actual users of the device. These can be ordinary people or professionals in the case of devices that can only be used by them. Performing a usability test means providing the user with both the device’s IFU or the user manual, as it is provided when the device is marketed, and a questionnaire that must detail each step of using the device, from consulting the packaging, opening the packaging, setting up the device, using the device for its intended function, to storing the device, and performing maintenance if applicable. Therefore, when generating the questions, they must cover all phases, not simply asking the user if the described phase is clear, but whether that step described in the instructions is performed as described and achieves the expected result. For example, in the case of the device in question, it is indicated that the device must be rehydrated before use, and instructions are provided on how to proceed with this rehydration. It is necessary to ask the user if the steps indicated for rehydration are clear, simple, and, most importantly, allow the achievement of the final desired volume of the device. Therefore, the questionnaire questions should not only aim to ask if the IFU are understandable but whether the user’s understanding is objective, meaning that the understanding is not based on subjectivity and user interpretation, but the result obtained by performing the step is objective for all users. Are you able to formulate a questionnaire that follows these guidelines I have provided? Use the attached file (IFU) as a reference.”

Despite being provided with guidelines for designing a usability test, ChatGPT merely added questions to those already generated with the 2nd Prompt, focusing on the ease of opening the packaging and the rehydration phase of the device, with or without achieving the desired final volume of the medical device (Fig. 9).

Therefore, the analysis of what ChatGPT produced, whether from a single prompt, a prompt supplemented with knowledge, or a combination of responses to individual prompts, leads to the conclusion that ChatGPT provides insufficient support to regulatory affairs in this case (Fig. 10) in terms of relevance, completeness, clarity, accuracy, and comprehensiveness.

Conclusions

The introduction of ChatGPT in the management of regulatory affairs for medical devices represents a significant technological advancement. Thanks to its ability to understand and generate natural language, combined with a sophisticated architecture based on geometric transformations and attention mechanisms, ChatGPT offers a wide range of applications that can complement complex processes, improving operational efficiency and reducing the workload of human operators. This approach translates into support capabilities in regulatory compliance activities, document and report drafting, questionnaire and test creation, data analysis, and scientific information management, increasing productivity and reducing response times.

Challenges and optimization of token consumption

Despite significant advantages, managing token consumption represents a crucial challenge, especially in regulatory contexts where information can be complex and detailed. ChatGPT operates on a model based on token counting, where each word and punctuation mark contribute to the computational cost of generating responses. This can become an issue when long or detailed responses are required, risking reaching the token limits and thus obtaining incomplete responses.

Adopting optimization methods and specific training is therefore essential to fully exploit the potential of ChatGPT. Prompt engineering is fundamental in this context, as it determines the model’s effectiveness in understanding and responding to requests. Well-designed prompts not only improve the quality of outputs but also optimize token consumption, reducing the risk of fragmented responses and enhancing the relevance of the provided information.

Continuous improvement through prompt engineering

The practice and refinement of prompt engineering techniques are iterative activities that enable increasingly precise and targeted responses in relevant contexts. Developing detailed prompts, analyzing generated outputs, and training users on how to effectively interact with ChatGPT are key steps to maximize the value of AI in regulatory workflows.

Using context-specific prompts and in-depth knowledge of the model allows requests to be calibrated to extract the system’s full potential. For example, for a more technical query, it may be necessary to provide ChatGPT with basic information and context to ensure a comprehensive response, while in other situations, a direct and concise question may be sufficient to achieve the desired result.

Adapting interaction strategies

The case studies clearly show that interaction with ChatGPT must be adapted based on the topic to be developed and the task to be performed. The user is called upon to establish a symbiotic relationship with the AI, gradually understanding the logic of interaction and information exchange. This approach requires continuous commitment to refining interrogation techniques, with the goal of obtaining high-quality outputs and reducing operational times: a simple single prompt may suffice for some generic requests, while other situations may require a sequence of complex prompts, accompanied by detailed technical information, possibly drawn from experts in the field. This approach allows ChatGPT to operate more comprehensively, making the best use of the available information.

Evolution of the model and access to information

The effectiveness of integrating ChatGPT into regulatory processes is closely linked to the continuous evolution of the model, the availability of up-to-date information from the web, and the development of increasingly advanced releases and algorithms. The AI’s access policies to informational databases, the expansion of its capabilities through integration with new data sources, and the continuous updating of its knowledge are crucial factors for maintaining high performance levels.

In conclusion, the use of ChatGPT in regulatory affairs for medical devices not only represents a powerful tool to support decision-making processes but also requires a strategic and adaptable approach from users. Continuous training, prompt optimization, and adaptation to technological evolutions are essential elements to maximize the benefits of this technology, enhancing operational effectiveness and ensuring a more efficient and responsive management of regulatory affairs.

Disclosures

Conflict of interest: All authors declare no conflict of interest.

Financial support: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.