Final Report on Lessons from the NESTcc Test-Cases

RAND Independent Assessment of NESTcc Test-Cases

As part of its MDUFA IV commitments, the National Evaluation System for health Technology Coordinating Center (NESTcc) contracted with The RAND Corporation (RAND) to develop an independent assessment of NESTcc’s pilot projects (Test-Cases) to evaluate the strengths, limitations and appropriate use of real-world evidence (RWE) for informing pre-market decision making for multiple device types  

RAND has developed a final report for the independent assessment, covering 18 of the 21 Test-Cases and builds on the interim report published in December 2021, which described lessons learned from the first 14 Test-Cases. The remaining three Test-Cases were terminated early due to various reasons and thus were not included in the assessment. The Test-Cases explore the feasibility for medical device ecosystem stakeholders to work with real-world data (RWD) sources and NESTcc’s Network Collaborators, as well as help to identify areas where NESTcc could play a role in creating efficiencies in RWE generation. 

The final report provides an overview of the Test-Cases, key lessons learned and stakeholder views on the strength of the evidence generated during the Test-Cases and identified opportunities to support industry stakeholders in generating robust RWE to support regulatory decisions. The full report of the RAND independent assessment of the NESTcc Test-Cases is available below.

Independent Assessment Summary

The final report summarizes the findings of the independent assessment: 

NESTcc Test-Cases represent one of the largest initiatives undertaken to systematically assess the potential for RWD to support regulatory decision-making. The Test-Cases highlighted promising methods and applications of RWD to generate evidence on uncommon uses of devices, develop in-vitro diagnostic (IVD) tests and create objective performance criteria to support efforts to benchmark device performance.  

Interviewees overwhelmingly viewed the Test-Cases as valuable in identifying lessons about the use of RWD across devices and clinical conditions. Interviewees also identified several keys to success for conducting research with RWD and highlighted a range of benefits to stakeholders—especially health care providers— such as using RWE to improve medical device purchasing decisions. 

Despite these benefits, the Test-Cases also revealed several challenges facing manufacturers who seek to use RWE for regulatory purposes, including reliably capturing unique device identifiers (UDIs), and ensuring adequate sample sizes, sufficient follow-up, and valid measures of patient and device characteristics and endpoints. Nevertheless, stakeholders expressed optimism that many of these issues could be overcome in time. The final report identifies several opportunities to overcome these challenges, including longer-term objectives such as more systematic capture of UDIs in structured fields and expanded data linkages. 

Keys to Success

In undertaking their efforts, RAND recognized themes across interviews with Test-Case participants that led to successful RWE generation, including: 

  • Utilizing manufacturer data to support research site selection and data collection 
  • Using supply chain data to identify use of medical devices 
  • Managing existing registries to support data collection and sharing 
  • Leveraging Common Data Models (CDM) to automate the extraction of Electronic Health Record (EHR) data 
  • Employing natural language processing (NLP) and other forms of text matching to extract data
  • Demonstrating concordance across different RWD to build confidence in results 

Data and Research Challenges

Themes also emerged relating to challenges of using RWD for medical devices, such as: 

  • Identifying devices in the absence of unique device identifiers 
  • Capturing key elements and their appropriate meaning in structured data 
  • Assessing the reliability of diagnosis and procedure information in structured data 
  • Extracting information from unstructured RWD 
  • Availability of long-term study endpoints 
  • Inadequate sample sizes 
  • Lack of computable phenotypes 
  • Limits to data sharing as a result of privacy and proprietary concerns 
  • Understanding patient financial burden 

Opportunities to Improve the Use of RWD

Ways to improve the use of RWD in medical device evaluation also became evident, and NESTcc continues to explore avenues with Network Collaborators and other stakeholders to harness these opportunities for future efforts. These include: 

  • Expand capture of UDIs in hospital and health system EHRs 
  • Expand use of medical device supply chain data to address lack of UDIs in EHRs 
  • Expand prospective data collection in EHRs to support medical device research and other health system needs 
  • Develop libraries of computable phenotypes and endpoints 
  • Expand use and breadth of CDMs 
  • Identify opportunities for effectively deploying NLP 
  • Create more linkages between EHR and other data 

Moving Forward

With this report from RAND Corporation, NESTcc has identified and will continue to identify areas where efficiencies in the system can be created, and innovative solutions can be developed to overcome the existing challenges in generating RWE in medical device regulatory decision-making and beyond. Our efforts thus far have been made possible under MDUFA IV. These efforts, combined with the unparalleled expertise of our community of Network Collaborators uniquely position NESTcc to demonstrate the promise of RWD moving forward. NESTcc is grateful to those who have and continue to collaborate in the drive to fulfill the promise of RWD. While you are on this site, we encourage you to explore and learn more about NESTcc and our efforts. Please view our work on Research Methods and Data Quality and our work on UDI adoption.

Continuous improvement of the NESTcc system will require the collaboration of our extended community of stakeholders, including regulators, payors, strategic partners, solution providers, medical device companies and patients. If you are interested in learning how you can join our efforts to accelerate patients’ access to safe and effective medical technology, please join our Collaborative Community.

Download the final report

The full report of the RAND independent assessment of the NESTcc Test-Cases is available below.

NESTcc appreciates the contributions of RAND and of all the participants in the Test-Cases as we advance the generation of RWE for the benefit of stakeholders across the medical device ecosystem.