Press "Enter" to skip to content

How to perform a clinical evaluation of medical devices – Part 1 – Overview and sample of activities

So after reviewing some concepts for a Clinical Evaluation training I gave this week, some people asked me about sources of literature on how to perform a clinical evaluation.

I notice that a lot of people does not seem to not that a clinical evaluation is nothing more than a specific application of a systematic review. Guidance such as MEDDEV 2.7.1 Rev 4 – Clinical Evaluation do give some guidance on the way to perform a clinical evaluation, however, it’s too much focused on the regulatory aspects, so they are really not enough to understand how to perform a clinical evaluation.

My suggestion of literature to understand how to perform a clinical evaluation/systematic review is as follows, in order:

1 – Summing Up – An interesting book as it provides a overview of systematic reviews without going into so much technical detail.

2 – Assembling the Pieces of a Systematic Review: A Guide for Librarians – Provides a very good overview of the process of systematic review, particularly from the point of view of the search and information specialist of the clinical evaluation team – the librarian.

3 – Systematic Approaches to a Successful Literature Review – Provides an in-depth technical overview of several aspects of critical analysis.

4 – Users’ Guides to the Medical Literature – Focused on the critical appraisal of the collected studies using Critical Appraisal Worksheets and other tools.

Also, I created the following list of sample activities required for a clinical evaluation, and will use this list in the next parts of this series.

1 – Clinical evaluation planning (MEDDEV 2.7 / 1 revision 4 item 7)

1.1 – Initial Meeting

1.1.1 – Clarification and agreement of concepts and assumptions (scope, results, team, project and data management, etc.)

1.1.2 – Schedule agreement (based on examples 6, 9, and 12 months of book 

Systematic Approaches to a Successful Literature Review)

1.1.3 – Definition of the research question (basis for the whole process)

1.1.3.1 – Define the type of research question and analyze feasibility

– Effectiveness of an intervention (treatment / therapy / policy)

– Harm

– Cause / Risk Factors

– Screening / Diagnosis

– Prognosis

– Prevention

– Experience / perceptions of patient / consumer / participant

– Service Delivery

– Cost-effectiveness

1.1.3.2 – Structure the research question using framework (suggestion – PICO or variations (PICO +, PICOC, PICOS, PICOT, PICO specific to diagnostic tests)

1.1.3.3 – Initiate definition of eligibility criteria (inclusion / exclusion)

1.1.3.3.1 – Perform preliminary searches to identify and calibrate criteria

1.1.3.3.2 – Define study types to consider depending on question type

1.1.3.4 – Final feasibility analysis

1.1.3.4.1 – Use criteria to determine if the research question is appropriate (e.g. FINER, TREAD, RETREAT)

1.1.4 – Define what information should be collected (use for example PRISMA as a basis)

1.2 – Team Planning and Data Management

1.2.1 – Data Management

1.2.1.1 – Define which bibliographic or reference management software to use

1.2.1.2 – Define which data extraction forms to use

1.2.1.3 – Define which data management software to use (database, spreadsheets, etc.)

1.2.2 – Definition of team

1.2.2.1 – Define the roles, qualifications and activities

– Clinical or content expert (remember that to reduce bias, you must use at least 2 reviewers)

– Expert in research methods and design

– Librarian or search expert

– Data Management Expert

– Statistician

1.2.2.3 – Define how each role will act in the clinical evaluation stages

– Plan – research question wording and protocol

– Data identification in the literature

– Rating – Screening and Appreciation

– Coding and Explanation 

– Data Extraction and Analysis

– Summary – Reporting and dissemination

1.2.2.4 – Include regulatory aspects (MEDDEV 2.7 / 1 revision 4 item 7)

1.2.2.4.1 – Include device description (MEDDEV 2.7 / 1 revision 4 A3)

1.2.2.4.2 – Define equivalence (if applicable) (MEDDEV 2.7 / 1 revision 4 A1)

2 – Identification of studies/data

2.1 – Search in databases (MEDDEV 2.7 / 1 revision 4 item 8.2)

2.1.1 – Define how the 4 types of searches will be done

– Preliminary Search

– Comprehensive database search

– Manual Search

– Contact with experts

2.1.2 – Identify what information will be documented for each of the search types

2.1.3 – Choice of databases (MEDDEV 2.7 / 1 revision 4 item A4)

2.1.3.1 – Define criteria for the accuracy evaluation of search results

2.1.3.2 – Define criteria for the search results recall evaluation

2.1.3.3 – Define additional criteria for the search results evaluation

2.1.3.4 – Initial identification of databases

2.1.3.4.1 – Listing of databases based on research question

2.1.3.4.2 – Verify if the research question topics are covered by each identified database

2.1.3.4.3 – Evaluate which materials are indexed by each identified database

2.1.3.4.4 – Evaluate which are the best platforms of each database and define which one to use

2.1.3.4.5 – Assess how each database will impact accuracy, recall and additional criteria

2.1.3.4.6 – Formal definition of which databases to use, with justifications for choosing each one

2.1.3.5 – Search strategy design (MEDDEV 2.7 / 1 revision 4 item A5)

2.1.3.5.1 – Translate the search query into a search plan based on the query structure (PICO, etc.)

2.1.3.5.2 – Define the concepts related to research question

2.1.3.5.3 – Define search terms for each concept

2.1.3.5.3.1 – Term harvesting

2.1.3.5.3.1.1 – Objective term harvesting (extraction) for natural language

2.1.3.5.3.1.1.1 – Search for test articles (which meet the eligibility criteria)

2.1.3.5.3.1.1.2 – Check how each article is cited in each database

2.1.3.5.3.1.1.3 – Identify the terms used for each research question concept and whether each article covers all concepts

2.1.3.5.3.1.1.4 – Investigate when the article does not cover all concepts and define related action (modify research question, change concepts, etc.)

2.1.3.5.3.1.2 – Conceptual term harvesting (localization) for natural language

2.1.3.5.3.1.3 – Objective term harvesting (extraction) for controlled vocabulary

2.1.3.5.3.1.4 – Conceptual term harvesting (localization) controlled vocabulary

2.1.3.5.3.1.4.1 – Identify the citation records of each article in each database

2.1.3.5.3.1.4.2 – Analyze the indexing of each article and identify the search terms

2.1.3.5.3.1.4.3- Perform independent searches on thesauri of each database to verify completeness of terms

2.1.3.5.2 – Perform independent searches for each article in each database to verify completeness of terms

2.1.3.5.4 – Refine Search Strategy

2.1.3.5.4.1 – Define floating subtitles (check rule in each database)

2.1.3.5.4.2 – Identify synonyms of natural language

2.1.3.5.4.3 – Define Truncation

2.1.3.5.4.4 – Defining wildcards

2.1.3.5.4.5 – Setting Limits

2.1.3.5.4.7 – Create and Validate Filters

2.1.3.5.4.8 – Define Boolean Operators

2.1.3.5.4.9 – Controlling the Use of NOT

2.1.3.5.4.10 – Perform additional searches for each article for each of the refinements to validate the refinement

2.1.3.5.5 – Critically analyze search strategies with the PRESS tool – Peer Review of Electronic Search Strategies

2.1.3.5.6 – Perform the searches, recording information as planned

2.2 – Searches beyond databases (MEDDEV 2.7 / 1 revision 4 item 8.1)

2.2.1 – Define additional search sources (MEDDEV 2.7 / 1 revision 4 item 8.1)

– Data generated and maintained by the manufacturer

– Search Records

– Clinical Trial Records

– Advertising / Contact

– Manual Search

– Search by quote

– Gray literature

– Event Annals

– Dissertations / Theses

– Internet Search

– Government, IGOs, NGOs

2.2.2 – Define what to document in each data source of each search beyond databases (see MECIR or PRISMA)

2.2.3 Perform Searches, recording information as planned

2.2.4 – Check if the results are indexed

2.2.5 – Determine why an article was not retrieved in a search

2.2.6 – Determine Terms

2.2.7 – Redo database search with additional terms

2.2.7 – Identify missing results

2.2.8 – Update database search records

2.3 – Search Evaluation

2.3.1 – Use the capture-mark-recapture (CRM) method

2.3.2 – Identify additional questions for evaluation (eg, when comparing complementary databases, how many new and unique relevant citations were found in the last two searches?)

2.3.3 – Finish recording evaluation

3 – Study selection

3.1 – Reviewers (minimum 2) de-duplicate citations

3.2 – Perform pilot test of eligibility criteria

3.3 – Screen titles and abstracts identified by searches using the eligibility criteria

3.4 – Get full-text articles from all relevant potential studies

3.5 – Select full-text articles for inclusion in systematic review using eligibility criteria

3.6 – Reporting results of the PRISMA declaration-based selection process

3.7 – Verify degree of agreement between reviewers (eg using Gwet kappa AC1)

4 – Critical appraisal (MEDDEV 2.7 / 1 revision 4 item 9)

4.1 – Identify the type of study / dataset (MEDDEV 2.7 / 1 revision 4 item 9.1)

– Effectiveness of an intervention (treatment / therapy / policy)

– Harm

– Cause / Risk Factors

– Screening / Diagnosis

– Prognosis

– Prevention

– Experience / perceptions of patient / consumer / participant

– Service Delivery

– Cost-effectiveness

4.2 – Define a valid appraisal tool that is appropriate for the study in question (Critical appraisal worksheet) (MEDDEV 2.7 / 1 revision 4 item 9.1 and A6)

4.3 – Sample studies / datasets (MEDDEV 2.7 / 1 revision 4 item 9.1)

4.4 – Reviewers validate the tool from sample studies / datasets (MEDDEV 2.7 / 1 revision 4 item 9.1)

4.5 – Reviewers apply the critical appraisal worksheet for each study / dataset (MEDDEV 2.7 / 1 revision 4 item 9.3.1)

4.6 – Reviewers determine relevance of each studies / datasets for clinical evaluation

(MEDDEV 2.7 / 1 revision 4 item 9.3.2)

4.7 – Reviewers weigh the contribution of each studies / datasets (MEDDEV 2.7 / 1 revision 4 item 9.3.3)

4.7.1 – Reviewers record / tabulate data and summarize critical appraisal results

4.8 – Data Collection and Summary

4.8.1 – Synthesis Planning (MEDDEV 2.7 / 1 revision 4 item 10)

4.8.1.1 – Define the type of synthesis (qualitative or quantitative)

4.8.2 – Define Planning-Based Data Elements

4.8.2.1 – Define if coding will be open or categorical

4.8.3 – Develop methods for data collection

4.8.3.1 – Develop coders

4.8.3.2 – Define conflict resolution, including vague or missing data

4.8.3.2 – Define Tool (eg Systematic Review Data Repository (SRDR))

4.8.4 – Develop Data Collection Forms

4.8.5 – Synthesizing the Data

4.8.5.1 – General Overview

4.8.5.2 – Summary of data related to safety requirements conformity assessment (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.1)

4.8.5.3 – Summary of data related to conformity assessment with acceptable benefit / risk profile requirement (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.2)

4.8.5.4 – Summary of data related to the assessment of compliance with performance requirements (MDD ER3 / AIMDD ER2) (MEDDEV 2.7 / 1 revision 4 A7.3)

4.8.5.5 – Summary of data related to conformity assessment with requirement for acceptability of undesirable side effects (MDD ER6 / AIMDD ER5) (MEDDEV 2.7 / 1 revision 4 A7.4)

4.8.6 – Explain the results and findings

4.8.5.2 – Analysis of safety related conformity assessment data (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.1)

4.8.5.3 – Analysis of data related to conformity assessment with acceptable benefit / risk profile requirement (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.2)

4.8.5.4 – Analysis of performance related conformity assessment data (MDD ER3 / AIMDD ER2) (MEDDEV 2.7 / 1 revision 4 A7.3)

4.8.5.5 – Analysis of data related to conformity assessment with requirement for acceptance of undesirable side effects (MDD ER6 / AIMDD ER5) (MEDDEV 2.7 / 1 revision 4 A7.4)

4.9 – Summary

4.9.1 – Write CER (MEDDEV 2.7 / 1 revision 4 item 11, A9)

4.9.2 – Use PRISMA as a basis

4.10 – Critical analysis of clinical evaluation

4.10.1 – Use tool like AMSTAR 2

4.10.2 – Evaluate also following MEDDEV 2.7 / 1 revision 4 A10)

4.11 – PMCF Planning (as part of PMS)

3 Comments

  1. Annie Annie September 4, 2020

    Hi, Marcelo,

    I am Annie. My company is an Orthopedic products manufacturers.There is a question confusing me. It is about the Stage 3-analysis of the the clinical data. About how to classify and summary the clinical data to conformity with the four parts: safety, performance, acceptability of undesirable side-effect, and acceptable benefit / risk profile requirement?

    • Marcelo Antunes Marcelo Antunes Post author | September 16, 2020

      Hello, you need to define this in the planning stage, based on the device design. You do not wait to define it when doing the stage 3.

Leave a Reply

Your email address will not be published. Required fields are marked *