M.E.S.S.

HARM
THE MAKING EBM SIMPLE
SERIES
Goals
At the completion of this presentation
the learner will be able to:
  Evaluate the validity, results and

applicability of a harm study

  Develop a “clinical bottom line” for a

harm study
Principles

  This is not rocket science

  This is not neurosurgery
  It is not even statistics

  It is basic math and simple concepts

Harm Studies
  Interventional studies
  Randomized clinical trials

  Observation studies
  Cohort – prospective or
  retrospective Case – Control
 Randomized Controlled Trial
P I C O

Intervention

n Random Efficacy Control

Allocation HARM

Exclusion
Criteria
Observational Studies
  Simpler to conduct than interventional studies
  Less time, effort, expense
  Less experimenter control
  Influence of confounding variables
  Greater bias potential = weaker strength of evidence
  Only choice if intervention not possible
  Unethical
  Not feasible (resources – patients, money)
Cohort Studies

  A group of subjects (cohort)

  Ideal for prognosis / harm studies

  Descriptive – incidence/prevalence of outcome

  Analytic – magnitude of the association

between exposures & outcome

Cohort - Orientation to Time

PAST PRESENT FUTURE

Prospective Exposure Harm

Outcome

Exposure Harm Retrospective

Prospective Cohort Study

PAST PRESENT FUTURE

Prospective Exposure Outcome

Chronic
Prospective Steroids Growth
Y/N
Retrospective Cohort Study

PAST PRESENT FUTURE

Exposure Outcome Retrospective

Febrile
Seizures Epilepsy Retrospective Y/N
Cohort Study - Strengths
  Limit risks to human subjects
  Not an intervention
  Only design when intervention not
feasible / ethical
  Can establish temporal relationship
  Causality vs Association
  Measurement of incidence/prevalence
  Calculate relative risk or hazard ratio
Cohort Study - Weaknesses
  Inefficient for rare outcomes
  Large sample size required
  Difficult to obtain controls if exposure is common
  Prospective – long follow-up – increase expense

  Potential biases

  Loss to follow-up (prospective)

  Detection bias – exposed group followed
more closely (prospective)
  Recall and record bias (retrospective)
Cohort Study - Comparison
Prospective
  Exposure in the present
  Outcome in the future
  More expensive and
longer (follow-up)
  Can match for
confounders
  Detection bias
Retrospective
  Exposure in the past
  Outcome in the present
  Less expensive
and faster
  Confounders
unmatched
  Recall bias
Case-Control Study
  Two study groups
  Cases – with outcome
  Controls – without outcome
  Retrospective
  Ideal for very rare outcomes

  Odds ratio – magnitude of association

between exposure and outcome

Case-Control Study

PAST PRESENT FUTURE

Outcome
Case

Exposure Retrospective
Y/N

Outcome
Control
 Case-Control Study - Example

PAST PRESENT FUTURE

Intussusception
Rotavirus
Vaccine Retrospective
Y/N
No
Intussusception

Murphy NEJM 344(8) Feb 2001

Case-Control - Strengths
  Limit risks to human subjects
  Not an intervention
  Only design when intervention not
feasible / ethical
  May be only feasible method for very
rare outcomes
  Smaller sample required,

inexpensive, fast
Case-Control - Weaknesses

 

 
 
 
  Outcomes
  Does not yield a true prevalence or incidence
  Ratio Outcome:No Outcome determined
by investigator
  Odds ratio only measure of association

  Potential bias - retrospective

Sampling bias – unequal distribution of potential
confounders
Recall bias - inadequate/biased memory of exposure
Record bias - data not available
Survivor bias – early deaths lost
Patient Scenario – Pneumonia
60 year old man with 3 day history of cough, fever and
shortness of breath with exertion. No chest pain. No
pertinent past medical history

HR 100, BP 120/70, RR 16, 02 Sat 98% on room air.

Alert, well appearing without tachypnea or increased work
of breathing. Breath sounds decreased, rales at the left
lung base. Remainder of exam normal

Elect to treat his pneumonia with Azithromycin. As you

hand him his prescription he recognizes the name of the
medication and states that he in concerned because he
read on the internet that azithromycin may increase the
risk of abnormal heart rhythms and death.
Azithromycin and CV Death

PAST PRESENT FUTURE

Azithromycin
Cardiovascular
No ABx Retrospective
Other ABx

Ray,W NEJM 366(20) May 2012

Harm Study – PICO Outline
P I C O

Exposure
Inclusion Harm

Outcome
No
Exposure

Exclusion
Azithromycin and CV Death
P I C O

Azithromycin
Adults
Death
Medicaid
Database
No Antibiotic
Amoxicillin CV
Non-CV
High Levofloxacin
Risk
Death
Ray,W NEJM 366(20) May 2012
 MESS SERIES DISCLAIMER
  The study discussed in this presentation is used only
to illustrate critical appraisal concepts
  The validity, results and applicability of the study
are presented only in part
  The charts and figures in the presentation are not
those found in the article
  This presentation should not be used to guide clinical
decision making
Critical Appraisal Process
  Clinical question – PICO format
  Search for available evidence

 
Assess
  Validity concerns – major, minor, none
  Primary results – clinically/statistically significant
  Applicability – to your patient or population
  Potential impact – change current practice?
Harm - PICO Question

P Does this patient/population with an ……….

I Exposure when compared to ………..

C No exposure have an increased risk of…….

O Harm? (negative outcome)

Azithromycin Study – PICO Question

P Does an adult patient receiving ………..

I Azithromycin when compared to …………

C No Antibiotics, Amoxicillin, Ciprofloxacin or

Levofloxacin have an increased risk of ……..

O Cardiovascular deaths?
Validity Criteria – Harm - Cohort
Aside from the exposure of interest did the
exposed and control groups start and finish
with the same risk for the outcome?

  Were the circumstances and methods

for detecting the outcome similar?
  Was follow-up sufficiently complete?
Validity – Equal Risk Factors?
  YES
  Used propensity matching

  Same purpose as randomization in a RCT

  Calculated from 153 covariates
  Included 22 possible factors including
a summary score for risk of CV death
  Risk factors very similar in each group
Demographics – Table 1
Cohort Group Size (N) CV Risk Score
No Antibiotic 1,3191,180 9.2
Amoxicillin 1,348,672 9.5
Ciprofloxacin 246,626 10.3
Levofloxacin 193,906 10.6
Azithromycin 347,795 9.3
Validity Criteria – Harm - Cohort
Aside from the exposure of interest did the
exposed and control groups start and finish
with the same risk for the outcome?
  Were patients similar for prognostic factors

that are known to be associated with the

outcome? (or were adjustments made using
statistical methods?)

  Was follow-up sufficiently complete?

Validity – Detection Methods?
  YES
  Medicaid database linked to

  Death certificates
  Statewide hospital discharge database
  Medicaid pharmacy files
Validity Criteria – Harm - Cohort
Aside from the exposure of interest did the
exposed and control groups start and finish
with the same risk for the outcome?
  Were patients similar for prognostic factors

that are known to be associated with the

outcome? (or were adjustments made using
statistical methods?)
  Were the circumstances and methods

for detecting the outcome similar?

Validity – Follow-up Complete?
  UNCLEAR
  Did not report if patients who started an

antibiotic course were not followed for

the full ten days
  Did not measure or report compliance

with the antibiotic course

Validity Concerns?
  None

  Major

  Which direction would the validity

concerns effect the results?
  Increased CV risk score in flouroquinolones –
Overestimate risk ciprofloxacin, levofloxacin
  Non-compliance – underestimate ABx risk
Results Criteria - Harm

  How precise is the estimate of the risk?

Outcome - Frequency
Prevalence
# with outcome at a point in time
# at risk for outcome at a point in time

Prevalence serious bacterial infection febrile infants = 10%

Incidence
# with outcome over a period of time
# at risk for outcome over a period of time

Incidence of death from ALL = 10% in the first year

Outcome - Strength of Association

  Relative risk – incidence (over time interval)

  Hazard ratio – incidence (at point in time)

  Odds ratio – neither – odds

  Standardized mortality ratio - prevalence

 Risk vs Odds

RISK ODDS
Similar
# times occurred # times occurred
# times could occur # times did not occur
Ace of spades 1/52 1/51
Any ace 4/52 4/48
Any red card 26/52 26/26
Not Ace of spades 51/52 51/1

Different
 Relative Risk vs Odds Ratio
RR =Risk of outcome
Intervention Risk of outcome
Control
Risk of outcome Exposed
Risk of outcome Not Exposed

OR = Odds of outcome Intervention

Odds of outcome Control
Odds of outcome Exposed
Odds of outcome Not Exposed
Interpreting RR & OR

RR Risk (exposed) / Risk (control)

> 1 Risk (exposed) > Risk (not exposed) =
1 Risk (exposed) = Risk (not exposed) < 1
Risk (exposed) < Risk (not exposed)

OR Odds (exposed) / Odds (not exposed)

> 1 Odds (exposed) > Odds (not exposed) =
1 Odds (exposed) = Odds (not exposed) < 1
Odds (exposed) < Odds (not exposed)
 Calculating RR & OR

OUTCOME
YES NO

YES
Exposure
NO

150 250 400

Risk (E) = 50/200 = 0.25 RR = R (E) = 0.25 = 0.5

Risk (NE) = 100/200 = 0.5 R (NE) 0.5
0
Odds (E) = 50/150 = 0.33 OR = O (E) = 0.33 = 0.33
Odds (NE) = 100/100 = 1 O (NE) 1
When to Use RR & OR?
Relative Risk Odds Ratio
Cohort Study Case-Control Study
RCT Logistic Regression
Meta-analysis
Hazard Ratio
Incidence - expressed over a time period

HR = incidence exposed
incidence not exposed

= hazard rate exposed

hazard rate not exposed
Interpreting Hazard Ratios

HR* HR (exposed) / HR (not exposed)

> 1 HR (exposed) > HR (not exposed) =
1 HR (exposed) = HR (not exposed) < 1
HR (exposed) < HR (not exposed)

HR* = Hazard Ratio

HR = Hazard Rate
Results – Magnitude?
Death
Total
CV
Non-CV
*Hazard
English Translation – HR
Hazard ratio (azithromycin/no antibiotic)
for CV death over 5 days = 2.88

A patient in the exposed group was ___ (HR) times more

likely to have an ______ (outcome) over time period X
then a patient in the not exposed group

A patient in the Azithromycin group was 2.88 more times

more likely to have a CV death over 5 days then a patient
in the no antibiotic group
Who is at Greatest Risk?

Excess deaths*/million - 5 day course

CV Risk Score *Azithromycin – Amoxicillin
th 245 95% CI (63-576)
10 decile
th 45 95% CI (11-105)
6-9 decile
th 9 95% CI (2-21)
1-5 decile
Results Criteria - Harm
  How strong is the association
between exposure and outcome?
Results – Precision?
Death No ABx Amoxicillin
Total 1 0.86
(0.58-1.28)
CV 1 0.95
(0.55-1.63)
Non-CV 1 0.76
(0.42-1.37)
Hazard Ratio (95% Confidence Interval)
HR - Statistical Significance
Hazard ratio (azithromycin/no antibiotic) for CV
death over 5 days = 2.88 95% CI (1.79 – 4.63)

  Hazard ratio = numerator/denominator

  If the two groups are the same then the HR will be 1
  A confidence interval for a hazard ratio that includes
1 is not statistically significant

  The confidence interval for the 2.88 hazard ratio

above does not include 1 so the hazard ratio is
statistically significant
Why do I need both an
absolute and relative
measure of the results?
Results – Hazard Rates (5days)
Death No ABx Amoxicillin Azithromycin
Total 52.6
CV 31.5
Non-CV 21.0
Hazard rate = number deaths / million courses of antibiotics
Relative vs Absolute Magnitude
  Relative – Hazard ratio (azithromycin/no
antibiotic) for CV death over 5 days HR
= 2.88 95% CI (1.79 – 4.63)

  Absolute – Hazard rate (death/million)

  No antibiotics – 29.8/1,000,000 = 0.00003
  Azithromycin – 85.2/1,000,000 = 0.000085
Absolute Risk Difference
  Hazard rates (CV death/million courses)
  No antibiotics – 29.8/1,000,000 = 0.00003
  Azithromycin – 85.2/1,000,000 = 0.000085

  Absolute risk difference (increase)

= Hazard rate (not exposed – exposed)
= Hazard rate (no antibiotic - azithromycin)
= 0.00003 – 0.000085 = - 0.000055 = - 0.0055%
English Translation – ARD
ARD (Azithro – No ABx) = - 0.0055%

A patient in the exposure group was ___%

less(ARD(+)) / more(ARD(-)) likely to have a
______ (outcome) then a patient in
the non-exposure group

A patient in the Azithromycin group was

0.0055% more likely to have a CV Death then a
patient in the No antibiotics group
Evidence of Causality
Association ≠ Causality
  Biologic plausibility
  Strength of association - magnitude

  Dose-response relationship

  Temporal sequence - predictor precedes

  outcome Consistency - results reproduced

 Kaplan-Meier Survival Curves

Survival
Percent Survival

Time

Not Exposed

Exposed

Years
 Temporal Relationship?

1-5 days 6-10 days

100
CV Deaths/Million Courses
50

Azithromycin

No Antibiotic
0

0 5 10
Days Since Prescription Filled
Applicability Criteria – Harm

  Was follow-up sufficiently long?

  Is the exposure similar to what might

occur in my patient?
  What is the magnitude of the risk?

  Are there any benefits that are known to

be associated with exposure?

Applicability – Patients Similar?
  UNCLEAR
  Demographic data presented

included only medications used,

coexisting condition and recent illness
  75% women - potentially limiting

generalizing the results to men

  Dependent on the CV risk profile in

your population
Applicability Criteria - Harm
  Were the study patients similar to
the patients in my practice?

  Is the exposure similar to what might

occur in my patient?
  What is the magnitude of the risk?

  Are there any benefits that are known to

be associated with exposure?

Applicability – Follow-up Duration?

  YES
  Short term cardiovascular death related

to the potential pro-arrhythmic effects of

azithromycin
  Data reported for 10 days after

the initiation of antibiotics

Applicability Criteria - Harm
  Were the study patients similar to
the patients in my practice?
  Was follow-up sufficiently long?

Data reported for 10 days after the initiation of antibiotics

  What is the magnitude of the risk?

  Are there any benefits that are known to

be associated with exposure?

Applicability – Similar Exposure?
  YES
  Azithromycin commonly to treat patients

with community acquired pneumonia

and chlamydia
Applicability Criteria - Harm
  Were the study patients similar to
the patients in my practice?
  Was follow-up sufficiently long?

  Is the exposure similar to what might

occur in my patient?

  Are there any benefits that are known to

be associated with exposure?
Applicability – Risk Magnitude?
  Relative – Hazard ratio (azithromycin/no
antibiotic) for CV death over 5 days HR
= 2.88 95% CI (1.79 – 4.63)

  Absolute – Hazard rate (death/million)

  No antibiotics – 29.8/1,000,000 = 0.00003
  Azithromycin – 85.2/1,000,000 = 0.000085
  ARD = - 0.000055 = - 0.0055%
Applicability Criteria - Harm
  Were the study patients similar to
the patients in my practice?
  Was follow-up sufficiently long?

  Is the exposure similar to what might

occur in my patient?
  What is the magnitude of the risk?
Applicability – Benefits?
  YES
  Treatment of infection

  Reduce complications of infection

 
Reduce spread of infection
Number Needed to Harm
  Risk of CV death over 5 days
  ARD = - 0.000055 = - 0.0055%

  NNH = 1/ARD = 1/0.000055 = 18,180

English Translation – NNH
NNH = 18,180

For every ___ (NNH) patients in the Exposed group 1

additional patient would have an ______ (outcome) over
time period X when compared to the Not exposed group

For every 18,180 patients in the Azithromycin group 1

additional patient would have a CV death over 5 days
when compared to the No antibiotic group
Clinical Bottom Line - Structure
  Background – context, why study was done?
  Study question – PICO format

 
Validity issues – major, minor, none
 
Primary results – clinically/statistically
 
significant
 
Applicability – your patient or population
Potential impact – change current practice?
Clinical Bottom Line - Study
  Case reports have implicated Azithromycin as
a cause of arrhythmias
  Do adult patients treated with Azithromycin

have an increased risk of CV death?

  Primary validity concerns are related to

observational studies such as recording

bias
  Magnitude of association CV death over 5 days
  Hazard ratio – 2.88 95% CI (1.79 - 4.63) Absolute
  risk increase - 0.0055%, NNH – 18,180
  Inadequate data on patient population
Summary – Harm Studies
  Observational studies typically used to assess
association of infrequent adverse events with exposures
  Observations studies have a higher potential for bias
  Relative and absolute measure of the magnitude
of association should be determined
  Practice saying the results in a sentence
  Confidence intervals - qualitative and
quantitative measure of statistical significance
  The NNH is a quantifiable measures of harm
QUESTIONS?

THE MAKING EBM SIMPLE SERIES