So after reviewing some concepts for a Clinical Evaluation training I gave this week, some people asked me about sources of literature on how to perform a clinical evaluation.
I notice that a lot of people does not seem to not that a clinical evaluation is nothing more than a specific application of a systematic review. Guidance such as MEDDEV 2.7.1 Rev 4 – Clinical Evaluation do give some guidance on the way to perform a clinical evaluation, however, it’s too much focused on the regulatory aspects, so they are really not enough to understand how to perform a clinical evaluation.
My suggestion of literature to understand how to perform a clinical evaluation/systematic review is as follows, in order:

1 – Summing Up – An interesting book as it provides a overview of systematic reviews without going into so much technical detail.
2 – Assembling the Pieces of a Systematic Review: A Guide for Librarians – Provides a very good overview of the process of systematic review, particularly from the point of view of the search and information specialist of the clinical evaluation team – the librarian.
3 – Systematic Approaches to a Successful Literature Review – Provides an in-depth technical overview of several aspects of critical analysis.
4 – Users’ Guides to the Medical Literature – Focused on the critical appraisal of the collected studies using Critical Appraisal Worksheets and other tools.
Also, I created the following list of sample activities required for a clinical evaluation, and will use this list in the next parts of this series.
1 – Clinical evaluation planning (MEDDEV 2.7 / 1 revision 4 item 7)
1.1 – Initial Meeting
1.1.1 – Clarification and agreement of concepts and assumptions (scope, results, team, project and data management, etc.)
1.1.2 – Schedule agreement (based on examples 6, 9, and 12 months of book
Systematic Approaches to a Successful Literature Review)
1.1.3 – Definition of the research question (basis for the whole process)
1.1.3.1 – Define the type of research question and analyze feasibility
– Effectiveness of an intervention (treatment / therapy / policy)
– Harm
– Cause / Risk Factors
– Screening / Diagnosis
– Prognosis
– Prevention
– Experience / perceptions of patient / consumer / participant
– Service Delivery
– Cost-effectiveness
1.1.3.2 – Structure the research question using framework (suggestion – PICO or variations (PICO +, PICOC, PICOS, PICOT, PICO specific to diagnostic tests)
1.1.3.3 – Initiate definition of eligibility criteria (inclusion / exclusion)
1.1.3.3.1 – Perform preliminary searches to identify and calibrate criteria
1.1.3.3.2 – Define study types to consider depending on question type
1.1.3.4 – Final feasibility analysis
1.1.3.4.1 – Use criteria to determine if the research question is appropriate (e.g. FINER, TREAD, RETREAT)
1.1.4 – Define what information should be collected (use for example PRISMA as a basis)
1.2 – Team Planning and Data Management
1.2.1 – Data Management
1.2.1.1 – Define which bibliographic or reference management software to use
1.2.1.2 – Define which data extraction forms to use
1.2.1.3 – Define which data management software to use (database, spreadsheets, etc.)
1.2.2 – Definition of team
1.2.2.1 – Define the roles, qualifications and activities
– Clinical or content expert (remember that to reduce bias, you must use at least 2 reviewers)
– Expert in research methods and design
– Librarian or search expert
– Data Management Expert
– Statistician
1.2.2.3 – Define how each role will act in the clinical evaluation stages
– Plan – research question wording and protocol
– Data identification in the literature
– Rating – Screening and Appreciation
– Coding and Explanation
– Data Extraction and Analysis
– Summary – Reporting and dissemination
1.2.2.4 – Include regulatory aspects (MEDDEV 2.7 / 1 revision 4 item 7)
1.2.2.4.1 – Include device description (MEDDEV 2.7 / 1 revision 4 A3)
1.2.2.4.2 – Define equivalence (if applicable) (MEDDEV 2.7 / 1 revision 4 A1)
2 – Identification of studies/data
2.1 – Search in databases (MEDDEV 2.7 / 1 revision 4 item 8.2)
2.1.1 – Define how the 4 types of searches will be done
– Preliminary Search
– Comprehensive database search
– Manual Search
– Contact with experts
2.1.2 – Identify what information will be documented for each of the search types
2.1.3 – Choice of databases (MEDDEV 2.7 / 1 revision 4 item A4)
2.1.3.1 – Define criteria for the accuracy evaluation of search results
2.1.3.2 – Define criteria for the search results recall evaluation
2.1.3.3 – Define additional criteria for the search results evaluation
2.1.3.4 – Initial identification of databases
2.1.3.4.1 – Listing of databases based on research question
2.1.3.4.2 – Verify if the research question topics are covered by each identified database
2.1.3.4.3 – Evaluate which materials are indexed by each identified database
2.1.3.4.4 – Evaluate which are the best platforms of each database and define which one to use
2.1.3.4.5 – Assess how each database will impact accuracy, recall and additional criteria
2.1.3.4.6 – Formal definition of which databases to use, with justifications for choosing each one
2.1.3.5 – Search strategy design (MEDDEV 2.7 / 1 revision 4 item A5)
2.1.3.5.1 – Translate the search query into a search plan based on the query structure (PICO, etc.)
2.1.3.5.2 – Define the concepts related to research question
2.1.3.5.3 – Define search terms for each concept
2.1.3.5.3.1 – Term harvesting
2.1.3.5.3.1.1 – Objective term harvesting (extraction) for natural language
2.1.3.5.3.1.1.1 – Search for test articles (which meet the eligibility criteria)
2.1.3.5.3.1.1.2 – Check how each article is cited in each database
2.1.3.5.3.1.1.3 – Identify the terms used for each research question concept and whether each article covers all concepts
2.1.3.5.3.1.1.4 – Investigate when the article does not cover all concepts and define related action (modify research question, change concepts, etc.)
2.1.3.5.3.1.2 – Conceptual term harvesting (localization) for natural language
2.1.3.5.3.1.3 – Objective term harvesting (extraction) for controlled vocabulary
2.1.3.5.3.1.4 – Conceptual term harvesting (localization) controlled vocabulary
2.1.3.5.3.1.4.1 – Identify the citation records of each article in each database
2.1.3.5.3.1.4.2 – Analyze the indexing of each article and identify the search terms
2.1.3.5.3.1.4.3- Perform independent searches on thesauri of each database to verify completeness of terms
2.1.3.5.2 – Perform independent searches for each article in each database to verify completeness of terms
2.1.3.5.4 – Refine Search Strategy
2.1.3.5.4.1 – Define floating subtitles (check rule in each database)
2.1.3.5.4.2 – Identify synonyms of natural language
2.1.3.5.4.3 – Define Truncation
2.1.3.5.4.4 – Defining wildcards
2.1.3.5.4.5 – Setting Limits
2.1.3.5.4.7 – Create and Validate Filters
2.1.3.5.4.8 – Define Boolean Operators
2.1.3.5.4.9 – Controlling the Use of NOT
2.1.3.5.4.10 – Perform additional searches for each article for each of the refinements to validate the refinement
2.1.3.5.5 – Critically analyze search strategies with the PRESS tool – Peer Review of Electronic Search Strategies
2.1.3.5.6 – Perform the searches, recording information as planned
2.2 – Searches beyond databases (MEDDEV 2.7 / 1 revision 4 item 8.1)
2.2.1 – Define additional search sources (MEDDEV 2.7 / 1 revision 4 item 8.1)
– Data generated and maintained by the manufacturer
– Search Records
– Clinical Trial Records
– Advertising / Contact
– Manual Search
– Search by quote
– Gray literature
– Event Annals
– Dissertations / Theses
– Internet Search
– Government, IGOs, NGOs
2.2.2 – Define what to document in each data source of each search beyond databases (see MECIR or PRISMA)
2.2.3 Perform Searches, recording information as planned
2.2.4 – Check if the results are indexed
2.2.5 – Determine why an article was not retrieved in a search
2.2.6 – Determine Terms
2.2.7 – Redo database search with additional terms
2.2.7 – Identify missing results
2.2.8 – Update database search records
2.3 – Search Evaluation
2.3.1 – Use the capture-mark-recapture (CRM) method
2.3.2 – Identify additional questions for evaluation (eg, when comparing complementary databases, how many new and unique relevant citations were found in the last two searches?)
2.3.3 – Finish recording evaluation
3 – Study selection
3.1 – Reviewers (minimum 2) de-duplicate citations
3.2 – Perform pilot test of eligibility criteria
3.3 – Screen titles and abstracts identified by searches using the eligibility criteria
3.4 – Get full-text articles from all relevant potential studies
3.5 – Select full-text articles for inclusion in systematic review using eligibility criteria
3.6 – Reporting results of the PRISMA declaration-based selection process
3.7 – Verify degree of agreement between reviewers (eg using Gwet kappa AC1)
4 – Critical appraisal (MEDDEV 2.7 / 1 revision 4 item 9)
4.1 – Identify the type of study / dataset (MEDDEV 2.7 / 1 revision 4 item 9.1)
– Effectiveness of an intervention (treatment / therapy / policy)
– Harm
– Cause / Risk Factors
– Screening / Diagnosis
– Prognosis
– Prevention
– Experience / perceptions of patient / consumer / participant
– Service Delivery
– Cost-effectiveness
4.2 – Define a valid appraisal tool that is appropriate for the study in question (Critical appraisal worksheet) (MEDDEV 2.7 / 1 revision 4 item 9.1 and A6)
4.3 – Sample studies / datasets (MEDDEV 2.7 / 1 revision 4 item 9.1)
4.4 – Reviewers validate the tool from sample studies / datasets (MEDDEV 2.7 / 1 revision 4 item 9.1)
4.5 – Reviewers apply the critical appraisal worksheet for each study / dataset (MEDDEV 2.7 / 1 revision 4 item 9.3.1)
4.6 – Reviewers determine relevance of each studies / datasets for clinical evaluation
(MEDDEV 2.7 / 1 revision 4 item 9.3.2)
4.7 – Reviewers weigh the contribution of each studies / datasets (MEDDEV 2.7 / 1 revision 4 item 9.3.3)
4.7.1 – Reviewers record / tabulate data and summarize critical appraisal results
4.8 – Data Collection and Summary
4.8.1 – Synthesis Planning (MEDDEV 2.7 / 1 revision 4 item 10)
4.8.1.1 – Define the type of synthesis (qualitative or quantitative)
4.8.2 – Define Planning-Based Data Elements
4.8.2.1 – Define if coding will be open or categorical
4.8.3 – Develop methods for data collection
4.8.3.1 – Develop coders
4.8.3.2 – Define conflict resolution, including vague or missing data
4.8.3.2 – Define Tool (eg Systematic Review Data Repository (SRDR))
4.8.4 – Develop Data Collection Forms
4.8.5 – Synthesizing the Data
4.8.5.1 – General Overview
4.8.5.2 – Summary of data related to safety requirements conformity assessment (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.1)
4.8.5.3 – Summary of data related to conformity assessment with acceptable benefit / risk profile requirement (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.2)
4.8.5.4 – Summary of data related to the assessment of compliance with performance requirements (MDD ER3 / AIMDD ER2) (MEDDEV 2.7 / 1 revision 4 A7.3)
4.8.5.5 – Summary of data related to conformity assessment with requirement for acceptability of undesirable side effects (MDD ER6 / AIMDD ER5) (MEDDEV 2.7 / 1 revision 4 A7.4)
4.8.6 – Explain the results and findings
4.8.5.2 – Analysis of safety related conformity assessment data (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.1)
4.8.5.3 – Analysis of data related to conformity assessment with acceptable benefit / risk profile requirement (MDD ER1 / AIMDD ER1) (MEDDEV 2.7 / 1 revision 4 A7.2)
4.8.5.4 – Analysis of performance related conformity assessment data (MDD ER3 / AIMDD ER2) (MEDDEV 2.7 / 1 revision 4 A7.3)
4.8.5.5 – Analysis of data related to conformity assessment with requirement for acceptance of undesirable side effects (MDD ER6 / AIMDD ER5) (MEDDEV 2.7 / 1 revision 4 A7.4)
4.9 – Summary
4.9.1 – Write CER (MEDDEV 2.7 / 1 revision 4 item 11, A9)
4.9.2 – Use PRISMA as a basis
4.10 – Critical analysis of clinical evaluation
4.10.1 – Use tool like AMSTAR 2
4.10.2 – Evaluate also following MEDDEV 2.7 / 1 revision 4 A10)
4.11 – PMCF Planning (as part of PMS)
[…] a clinical evaluation of medical devices – Part 1 – Overview and sample of activities – http://www.medicaldevice.expert/europe/european-commission/medical-device-regulation/how-to-perform-… to have a better understanding of the concepts so the understanding of this part is […]
Hi, Marcelo,
I am Annie. My company is an Orthopedic products manufacturers.There is a question confusing me. It is about the Stage 3-analysis of the the clinical data. About how to classify and summary the clinical data to conformity with the four parts: safety, performance, acceptability of undesirable side-effect, and acceptable benefit / risk profile requirement?
Hello, you need to define this in the planning stage, based on the device design. You do not wait to define it when doing the stage 3.