How to perform a clinical evaluation of medical devices – Part 1 – Overview and sample of activities

So after reviewing some concepts for a Clinical Evaluation training I gave this week, some people asked me about sources of literature on how to perform a clinical evaluation.

I notice that a lot of people does not seem to not that a clinical evaluation is nothing more than a specific application of a systematic review. Guidance such as MEDDEV 2.7.1 Rev 4 – Clinical Evaluation do give some guidance on the way to perform a clinical evaluation, however, it’s too much focused on the regulatory aspects, so they are really not enough to understand how to perform a clinical evaluation.

My suggestion of literature to understand how to perform a clinical evaluation/systematic review is as follows, in order:

1 – Summing Up – An interesting book as it provides a overview of systematic reviews without going into so much technical detail.

2 – Assembling the Pieces of a Systematic Review: A Guide for Librarians – Provides a very good overview of the process of systematic review, particularly from the point of view of the search and information specialist of the clinical evaluation team – the librarian.

3 – Systematic Approaches to a Successful Literature Review – Provides an in-depth technical overview of several aspects of critical analysis.

4 – Users’ Guides to the Medical Literature – Focused on the critical appraisal of the collected studies using Critical Appraisal Worksheets and other tools.

Also, I created the following list of sample activities required for a clinical evaluation, and will use this list in the next parts of this series.

1 – Clinical evaluation planning (MEDDEV 2.7 / 1 revision 4 item 7)

1.1 – Initial Meeting

1.1.1 – Clarification and agreement of concepts and assumptions (scope, results, team, project and data management, etc.)

1.1.2 – Schedule agreement (based on examples 6, 9, and 12 months of book 

Systematic Approaches to a Successful Literature Review)

1.1.3 – Definition of the research question (basis for the whole process)

1.1.3.1 – Define the type of research question and analyze feasibility

– Effectiveness of an intervention (treatment / therapy / policy)

– Harm

– Cause / Risk Factors

– Screening / Diagnosis

– Prognosis

– Prevention

– Experience / perceptions of patient / consumer / participant

– Service Delivery

– Cost-effectiveness

1.1.3.2 – Structure the research question using framework (suggestion – PICO or variations (PICO +, PICOC, PICOS, PICOT, PICO specific to diagnostic tests)

1.1.3.3 – Initiate definition of eligibility criteria (inclusion / exclusion)

1.1.3.3.1 – Perform preliminary searches to identify and calibrate criteria

1.1.3.3.2 – Define study types to consider depending on question type

1.1.3.4 – Final feasibility analysis

1.1.3.4.1 – Use criteria to determine if the research question is appropriate (e.g. FINER, TREAD, RETREAT)

1.1.4 – Define what information should be collected (use for example PRISMA as a basis)

1.2 – Team Planning and Data Management

1.2.1 – Data Management

1.2.1.1 – Define which bibliographic or reference management software to use

1.2.1.2 – Define which data extraction forms to use

1.2.1.3 – Define which data management software to use (database, spreadsheets, etc.)

1.2.2 – Definition of team

1.2.2.1 – Define the roles, qualifications and activities

– Clinical or content expert (remember that to reduce bias, you must use at least 2 reviewers)

– Expert in research methods and design

– Librarian or search expert

– Data Management Expert

– Statistician

1.2.2.3 – Define how each role will act in the clinical evaluation stages

– Plan – research question wording and protocol

– Data identification in the literature

– Rating – Screening and Appreciation

– Coding and Explanation 

– Data Extraction and Analysis

– Summary – Reporting and dissemination

1.2.2.4 – Include regulatory aspects (MEDDEV 2.7 / 1 revision 4 item 7)

1.2.2.4.1 – Include device description (MEDDEV 2.7 / 1 revision 4 A3)

1.2.2.4.2 – Define equivalence (if applicable) (MEDDEV 2.7 / 1 revision 4 A1)

2 – Identification of studies/data

2.1 – Search in databases (MEDDEV 2.7 / 1 revision 4 item 8.2)

2.1.1 – Define how the 4 types of searches will be done

– Preliminary Search

– Comprehensive database search

– Manual Search

– Contact with experts

2.1.2 – Identify what information will be documented for each of the search types

2.1.3 – Choice of databases (MEDDEV 2.7 / 1 revision 4 item A4)

2.1.3.1 – Define criteria for the accuracy evaluation of search results

2.1.3.2 – Define criteria for the search results recall evaluation

2.1.3.3 – Define additional criteria for the search results evaluation

2.1.3.4 – Initial identification of databases

2.1.3.4.1 – Listing of databases based on research question

2.1.3.4.2 – Verify if the research question topics are covered by each identified database

2.1.3.4.3 – Evaluate which materials are indexed by each identified database

2.1.3.4.4 – Evaluate which are the best platforms of each database and define which one to use

2.1.3.4.5 – Assess how each database will impact accuracy, recall and additional criteria

2.1.3.4.6 – Formal definition of which databases to use, with justifications for choosing each one

2.1.3.5 – Search strategy design (MEDDEV 2.7 / 1 revision 4 item A5)

2.1.3.5.1 – Translate the search query into a search plan based on the query structure (PICO, etc.)

2.1.3.5.2 – Define the concepts related to research question

2.1.3.5.3 – Define search terms for each concept

2.1.3.5.3.1 – Term harvesting

2.1.3.5.3.1.1 – Objective term harvesting (extraction) for natural language

2.1.3.5.3.1.1.1 – Search for test articles (which meet the eligibility criteria)

2.1.3.5.3.1.1.2 – Check how each article is cited in each database

2.1.3.5.3.1.1.3 – Identify the terms used for each research question concept and whether each article covers all concepts

2.1.3.5.3.1.1.4 – Investigate when the article does not cover all concepts and define related action (modify research question, change concepts, etc.)

2.1.3.5.3.1.2 – Conceptual term harvesting (localization) for natural language

2.1.3.5.3.1.3 – Objective term harvesting (extraction) for controlled vocabulary

2.1.3.5.3.1.4 – Conceptual term harvesting (localization) controlled vocabulary

2.1.3.5.3.1.4.1 – Identify the citation records of each article in each database

2.1.3.5.3.1.4.2 – Analyze the indexing of each article and identify the search terms

2.1.3.5.3.1.4.3- Perform independent searches on thesauri of each database to verify completeness of terms

2.1.3.5.2 – Perform independent searches for each article in each database to verify completeness of terms

2.1.3.5.4 – Refine Search Strategy

2.1.3.5.4.1 – Define floating subtitles (check rule in each database)

2.1.3.5.4.2 – Identify synonyms of natural language

2.1.3.5.4.3 – Define Truncation

2.1.3.5.4.4 – Defining wildcards

2.1.3.5.4.5 – Setting Limits

2.1.3.5.4.7 – Create and Validate Filters

2.1.3.5.4.8 – Define Boolean Operators

2.1.3.5.4.9 – Controlling the Use of NOT

2.1.3.5.4.10 – Perform additional searches for each article for each of the refinements to validate the refinement

2.1.3.5.5 – Critically analyze search strategies with the PRESS tool – Peer Review of Electronic Search Strategies

2.1.3.5.6 – Perform the searches, recording information as planned

2.2 – Searches beyond databases (MEDDEV 2.7 / 1 revision 4 item 8.1)

2.2.1 – Define additional search sources (MEDDEV 2.7 / 1 revision 4 item 8.1)

– Data generated and maintained by the manufacturer

– Search Records

– Clinical Trial Records

– Advertising / Contact

– Manual Search

– Search by quote

– Gray literature

– Event Annals

– Dissertations / Theses

– Internet Search

– Government, IGOs, NGOs

2.2.2 – Define what to document in each data source of each search beyond databases (see MECIR or PRISMA)

2.2.3 Perform Searches, recording information as planned

2.2.4 – Check if the results are indexed

2.2.5 – Determine why an article was not retrieved in a search

2.2.6 – Determine Terms

2.2.7 – Redo database search with additional terms

2.2.7 – Identify missing results

2.2.8 – Update database search records

2.3 – Search Evaluation

2.3.1 – Use the capture-mark-recapture (CRM) method

2.3.2 – Identify additional questions for evaluation (eg, when comparing complementary databases, how many new and unique relevant citations were found in the last two searches?)

2.3.3 – Finish recording evaluation

3 – Study selection

3.1 – Reviewers (minimum 2) de-duplicate citations

3.2 – Perform pilot test of eligibility criteria

3.3 – Screen titles and abstracts identified by searches using the eligibility criteria

3.4 – Get full-text articles from all relevant potential studies

3.5 – Select full-text articles for inclusion in systematic review using eligibility criteria

3.6 – Reporting results of the PRISMA declaration-based selection process

3.7 – Verify degree of agreement between reviewers (eg using Gwet kappa AC1)

4 – Critical appraisal (MEDDEV 2.7 / 1 revision 4 item 9)

4.1 – Identify the type of study / dataset (MEDDEV 2.7 / 1 revision 4 item 9.1)

– Effectiveness of an intervention (treatment / therapy / policy)

– Harm

– Cause / Risk Factors

– Screening / Diagnosis

– Prognosis

– Prevention

– Experience / perceptions of patient / consumer / participant

– Service Delivery

– Cost-effectiveness

4.2 – Define a valid appraisal tool that is appropriate for the study in question (Critical appraisal worksheet) (MEDDEV 2.7 / 1 revision 4 item 9.1 and A6)

4.3 – Sample studies / datasets (MEDDEV 2.7 / 1 revision 4 item 9.1)

4.4 – Reviewers validate the tool from sample studies / datasets (MEDDEV 2.7 / 1 revision 4 item 9.1)

4.5 – Reviewers apply the critical appraisal worksheet for each study / dataset (MEDDEV 2.7 / 1 revision 4 item 9.3.1)

4.6 – Reviewers determine relevance of each studies / datasets for clinical evaluation

(MEDDEV 2.7 / 1 revision 4 item 9.3.2)

4.7 – Reviewers weigh the contribution of each studies / datasets (MEDDEV 2.7 / 1 revision 4 item 9.3.3)

4.7.1 – Reviewers record / tabulate data and summarize critical appraisal results

4.8 – Data Collection and Summary

4.8.1 – Synthesis Planning (MEDDEV 2.7 / 1 revision 4 item 10)

4.8.1.1 – Define the type of synthesis (qualitative or quantitative)

4.8.2 – Define Planning-Based Data Elements

4.8.2.1 – Define if coding will be open or categorical

4.8.3 – Develop methods for data collection

4.8.3.1 – Develop coders

4.8.3.2 – Define conflict resolution, including vague or missing data

4.8.3.2 – Define Tool (eg Systematic Review Data Repository (SRDR))

4.8.4 – Develop Data Collection Forms

4.8.5 – Synthesizing the Data

4.8.5.1 – General Overview

4.8.5.2 – Summary of data related to safety requirements conformity assessment (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.1)

4.8.5.3 – Summary of data related to conformity assessment with acceptable benefit / risk profile requirement (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.2)

4.8.5.4 – Summary of data related to the assessment of compliance with performance requirements (MDD ER3 / AIMDD ER2) (MEDDEV 2.7 / 1 revision 4 A7.3)

4.8.5.5 – Summary of data related to conformity assessment with requirement for acceptability of undesirable side effects (MDD ER6 / AIMDD ER5) (MEDDEV 2.7 / 1 revision 4 A7.4)

4.8.6 – Explain the results and findings

4.8.5.2 – Analysis of safety related conformity assessment data (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.1)

4.8.5.3 – Analysis of data related to conformity assessment with acceptable benefit / risk profile requirement (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.2)

4.8.5.4 – Analysis of performance related conformity assessment data (MDD ER3 / AIMDD ER2) (MEDDEV 2.7 / 1 revision 4 A7.3)

4.8.5.5 – Analysis of data related to conformity assessment with requirement for acceptance of undesirable side effects (MDD ER6 / AIMDD ER5) (MEDDEV 2.7 / 1 revision 4 A7.4)

4.9 – Summary

4.9.1 – Write CER (MEDDEV 2.7 / 1 revision 4 item 11, A9)

4.9.2 – Use PRISMA as a basis

4.10 – Critical analysis of clinical evaluation

4.10.1 – Use tool like AMSTAR 2

4.10.2 – Evaluate also following MEDDEV 2.7 / 1 revision 4 A10)

4.11 – PMCF Planning (as part of PMS)