Independent assessment of the NESTcc Test-Cases: RAND interim report provides insights from RWE generation for medical devices
As part of its MDUFA IV commitments, the National Evaluation System for health Technology Coordinating Center (NESTcc) contracted with The RAND Corporation (RAND) to develop an independent assessment of NESTcc’s 21 pilot projects (Test-Cases), to evaluate the strengths, limitations and use of real-world evidence (RWE) for regulatory decision-making. The final report of this assessment will be published by September 2022 following the completion of all 21 Test-Cases.
To share insights into the generation of RWE for medical devices gleaned to date, RAND has developed an interim report for the independent assessment, covering the 14 Test-Cases completed through August 2021. The Test-Cases explore the feasibility for medical device ecosystem stakeholders to work with RWD sources and NESTcc’s initial set of Network Collaborators, as well as help to identify areas where NESTcc could play a role in creating efficiencies in RWE generation.
The interim report provides an overview of the Test-Cases, key lessons, and stakeholder views of the evidence based on interviews with submitting organizations, Network Collaborators and FDA representatives who participated in Test-Cases.
Download the interim report
The full interim report of the RAND independent assessment of the NESTcc Test-Cases is available below.
Independent Assessment Summary
The interim report summarizes the findings of the initial assessment to date:
NESTcc Test-Cases represent one of the largest initiatives undertaken to systematically assess the potential for RWD to support regulatory decision-making. The first 14 Test-Cases highlighted promising methods and applications of RWD to generate evidence on uncommon uses of devices, develop IVD tests and create objective performance criteria to support efforts to benchmark device performance. Interviewees overwhelmingly viewed the Test-Cases as valuable in helping to identify lessons about the use of RWD that could generalize across devices and clinical conditions. Interviewees identified several keys to success for conducting research with RWD and highlighted a range of benefits to stakeholders—especially health care providers— such as using RWE to improve medical device purchasing decisions.
Despite these benefits, the Test-Cases also revealed several challenges facing manufacturers who seek to use RWE for regulatory purposes, including reliably capturing UDIs, and ensuring adequate sample sizes, sufficient follow-up, and valid measures of patient and device characteristics and endpoints. Nevertheless, stakeholders expressed optimism that many of these issues could be overcome in time.
Keys to Success
In undertaking their efforts, RAND recognized themes across interviews with Test-Case participants that led to successful RWE generation, including:
- Using manufacturer data to support research site selection and data collection.
- Using existing registries to support data collection and sharing.
- Leveraging CDMs to automate the extraction of EHR data.
- Employing natural language processing and other forms of text matching to extract data.
- Demonstrating concordance across different RWD to build confidence in results.
Data and Research Challenges
Themes also emerged relating to challenges of using RWD for medical devices:
- Variation in data formats and data generation practices across network collaborators.
- Extracting data elements from RWD.
- Limited long-term follow-up of study endpoints.
- Lack of existing computable phenotypes.
- Data sharing and data privacy.
Opportunities to Improve the Use of RWD
Ways to improve the use of RWD in medical device evaluation also became evident, and NESTcc continues to explore avenues with Network Collaborators and other stakeholders to harness these opportunities for future efforts. These include:
- Expanding capture of UDIs in hospital and health system EHRs.
- Developing libraries of computable phenotypes and endpoints.
- Expanding use and breadth of CDMs.
- Identifying opportunities for effectively deploying NLP.
- Creating more linkages between EHR and other data.
NESTcc appreciates the contributions of RAND and of all the participants in the Test-Cases as we advance the generation of RWE for the benefit of stakeholders across the medical device ecosystem.
