You are on page 1of 78

M.E.S.S.

HARM
THE MAKING EBM SIMPLE
SERIES
Goals
At the completion of this presentation
the learner will be able to:
  Evaluate the validity, results and

applicability of a harm study


  Develop a “clinical bottom line” for a

harm study
Principles

  This is not rocket science


  This is not neurosurgery
  It is not even statistics

  It is basic math and simple concepts


Harm Studies
  Interventional studies
  Randomized clinical trials

  Observation studies
  Cohort – prospective or
  retrospective Case – Control
Randomized Controlled Trial
P I C O

Intervention

n Random Efficacy Control

Allocation HARM

Exclusion
Criteria
Observational Studies
  Simpler to conduct than interventional studies
  Less time, effort, expense
  Less experimenter control
  Influence of confounding variables
  Greater bias potential = weaker strength of evidence
  Only choice if intervention not possible
  Unethical
  Not feasible (resources – patients, money)
Cohort Studies

  A group of subjects (cohort)


  Ideal for prognosis / harm studies

  Descriptive – incidence/prevalence of outcome


  Analytic – magnitude of the association

between exposures & outcome


Cohort - Orientation to Time

PAST PRESENT FUTURE

Prospective Exposure Harm

Outcome

Exposure Harm Retrospective


Prospective Cohort Study

PAST PRESENT FUTURE

Prospective Exposure Outcome

Chronic
Prospective Steroids Growth
Y/N
Retrospective Cohort Study

PAST PRESENT FUTURE

Exposure Outcome Retrospective

Febrile
Seizures Epilepsy Retrospective Y/N
Cohort Study - Strengths
  Limit risks to human subjects
  Not an intervention
  Only design when intervention not
feasible / ethical
  Can establish temporal relationship
  Causality vs Association
  Measurement of incidence/prevalence
  Calculate relative risk or hazard ratio
Cohort Study - Weaknesses
  Inefficient for rare outcomes
  Large sample size required
  Difficult to obtain controls if exposure is common
  Prospective – long follow-up – increase expense

  Potential biases

  Loss to follow-up (prospective)


  Detection bias – exposed group followed
more closely (prospective)
  Recall and record bias (retrospective)
Cohort Study - Comparison

Prospective Retrospective
  Exposure in the present   Exposure in the past
  Outcome in the future   Outcome in the present
  More expensive and   Less expensive
longer (follow-up) and faster
  Can match for   Confounders
confounders unmatched
  Detection bias   Recall bias
Case-Control Study
  Two study groups
  Cases – with outcome
  Controls – without outcome
  Retrospective
  Ideal for very rare outcomes

  Odds ratio – magnitude of association

between exposure and outcome


Case-Control Study

PAST PRESENT FUTURE

Outcome
Case

Exposure Retrospective
Y/N

Outcome
Control
Case-Control Study - Example

PAST PRESENT FUTURE

Intussusception
Rotavirus
Vaccine Retrospective
Y/N
No
Intussusception

Murphy NEJM 344(8) Feb 2001


Case-Control - Strengths
  Limit risks to human subjects
  Not an intervention
  Only design when intervention not
feasible / ethical
  May be only feasible method for very
rare outcomes
  Smaller sample required,

inexpensive, fast
Case-Control - Weaknesses

 

 
 
 
  Outcomes
  Does not yield a true prevalence or incidence
  Ratio Outcome:No Outcome determined
by investigator
  Odds ratio only measure of association

  Potential bias - retrospective


Sampling bias – unequal distribution of potential
confounders
Recall bias - inadequate/biased memory of exposure
Record bias - data not available
Survivor bias – early deaths lost
Patient Scenario – Pneumonia
60 year old man with 3 day history of cough, fever and
shortness of breath with exertion. No chest pain. No
pertinent past medical history

HR 100, BP 120/70, RR 16, 02 Sat 98% on room air.


Alert, well appearing without tachypnea or increased work
of breathing. Breath sounds decreased, rales at the left
lung base. Remainder of exam normal

Elect to treat his pneumonia with Azithromycin. As you


hand him his prescription he recognizes the name of the
medication and states that he in concerned because he
read on the internet that azithromycin may increase the
risk of abnormal heart rhythms and death.
Azithromycin and CV Death

PAST PRESENT FUTURE

Azithromycin
Cardiovascular
No ABx Retrospective
Other ABx

Ray,W NEJM 366(20) May 2012


Harm Study – PICO Outline
P I C O

Exposure
Inclusion Harm

Outcome
No
Exposure

Exclusion
Azithromycin and CV Death
P I C O

Azithromycin
Adults
Death
Medicaid
Database
No Antibiotic
Amoxicillin CV
Non-CV
High Levofloxacin
Risk
Death
Ray,W NEJM 366(20) May 2012
MESS SERIES DISCLAIMER
  The study discussed in this presentation is used only
to illustrate critical appraisal concepts
  The validity, results and applicability of the study
are presented only in part
  The charts and figures in the presentation are not
those found in the article
  This presentation should not be used to guide clinical
decision making
Critical Appraisal Process
  Clinical question – PICO format
  Search for available evidence

 
Assess
  Validity concerns – major, minor, none
  Primary results – clinically/statistically significant
  Applicability – to your patient or population
  Potential impact – change current practice?
Harm - PICO Question

P Does this patient/population with an ……….

I Exposure when compared to ………..

C No exposure have an increased risk of…….

O Harm? (negative outcome)


Azithromycin Study – PICO Question

P Does an adult patient receiving ………..

I Azithromycin when compared to …………

C No Antibiotics, Amoxicillin, Ciprofloxacin or


Levofloxacin have an increased risk of ……..

O Cardiovascular deaths?
Validity Criteria – Harm - Cohort
Aside from the exposure of interest did the
exposed and control groups start and finish
with the same risk for the outcome?

  Were the circumstances and methods


for detecting the outcome similar?
  Was follow-up sufficiently complete?
Validity – Equal Risk Factors?
  YES
  Used propensity matching

  Same purpose as randomization in a RCT


  Calculated from 153 covariates
  Included 22 possible factors including
a summary score for risk of CV death
  Risk factors very similar in each group
Demographics – Table 1
Cohort Group Size (N) CV Risk Score
No Antibiotic 1,3191,180 9.2
Amoxicillin 1,348,672 9.5
Ciprofloxacin 246,626 10.3
Levofloxacin 193,906 10.6
Azithromycin 347,795 9.3
Validity Criteria – Harm - Cohort
Aside from the exposure of interest did the
exposed and control groups start and finish
with the same risk for the outcome?
  Were patients similar for prognostic factors

that are known to be associated with the


outcome? (or were adjustments made using
statistical methods?)

  Was follow-up sufficiently complete?


Validity – Detection Methods?
  YES
  Medicaid database linked to

  Death certificates
  Statewide hospital discharge database
  Medicaid pharmacy files
Validity Criteria – Harm - Cohort
Aside from the exposure of interest did the
exposed and control groups start and finish
with the same risk for the outcome?
  Were patients similar for prognostic factors

that are known to be associated with the


outcome? (or were adjustments made using
statistical methods?)
  Were the circumstances and methods

for detecting the outcome similar?


Validity – Follow-up Complete?
  UNCLEAR
  Did not report if patients who started an

antibiotic course were not followed for


the full ten days
  Did not measure or report compliance

with the antibiotic course


Validity Concerns?
  None

  Major

  Which direction would the validity


concerns effect the results?
  Increased CV risk score in flouroquinolones –
Overestimate risk ciprofloxacin, levofloxacin
  Non-compliance – underestimate ABx risk
Results Criteria - Harm

  How precise is the estimate of the risk?


Outcome - Frequency
Prevalence
# with outcome at a point in time
# at risk for outcome at a point in time

Prevalence serious bacterial infection febrile infants = 10%

Incidence
# with outcome over a period of time
# at risk for outcome over a period of time

Incidence of death from ALL = 10% in the first year


Outcome - Strength of Association

  Relative risk – incidence (over time interval)


  Hazard ratio – incidence (at point in time)

  Odds ratio – neither – odds

  Standardized mortality ratio - prevalence


Risk vs Odds

RISK ODDS
Similar
# times occurred # times occurred
# times could occur # times did not occur
Ace of spades 1/52 1/51
Any ace 4/52 4/48
Any red card 26/52 26/26
Not Ace of spades 51/52 51/1

Different
Relative Risk vs Odds Ratio
RR =Risk of outcome
Intervention Risk of outcome
Control
Risk of outcome Exposed
Risk of outcome Not Exposed

OR = Odds of outcome Intervention


Odds of outcome Control
Odds of outcome Exposed
Odds of outcome Not Exposed
Interpreting RR & OR

RR Risk (exposed) / Risk (control)


> 1 Risk (exposed) > Risk (not exposed) =
1 Risk (exposed) = Risk (not exposed) < 1
Risk (exposed) < Risk (not exposed)

OR Odds (exposed) / Odds (not exposed)


> 1 Odds (exposed) > Odds (not exposed) =
1 Odds (exposed) = Odds (not exposed) < 1
Odds (exposed) < Odds (not exposed)
Calculating RR & OR

OUTCOME
YES NO

YES
Exposure
NO

150 250 400

Risk (E) = 50/200 = 0.25 RR = R (E) = 0.25 = 0.5


Risk (NE) = 100/200 = 0.5 R (NE) 0.5
0
Odds (E) = 50/150 = 0.33 OR = O (E) = 0.33 = 0.33
Odds (NE) = 100/100 = 1 O (NE) 1
When to Use RR & OR?
Relative Risk Odds Ratio
Cohort Study Case-Control Study
RCT Logistic Regression
Meta-analysis
Hazard Ratio
Incidence - expressed over a time period

HR = incidence exposed
incidence not exposed

= hazard rate exposed


hazard rate not exposed
Interpreting Hazard Ratios

HR* HR (exposed) / HR (not exposed)


> 1 HR (exposed) > HR (not exposed) =
1 HR (exposed) = HR (not exposed) < 1
HR (exposed) < HR (not exposed)

HR* = Hazard Ratio


HR = Hazard Rate
Results – Magnitude?
Death
Total
CV
Non-CV
*Hazard
English Translation – HR
Hazard ratio (azithromycin/no antibiotic)
for CV death over 5 days = 2.88

A patient in the exposed group was ___ (HR) times more


likely to have an ______ (outcome) over time period X
then a patient in the not exposed group

A patient in the Azithromycin group was 2.88 more times


more likely to have a CV death over 5 days then a patient
in the no antibiotic group
Who is at Greatest Risk?

Excess deaths*/million - 5 day course


CV Risk Score *Azithromycin – Amoxicillin
th 245 95% CI (63-576)
10 decile
th 45 95% CI (11-105)
6-9 decile
th 9 95% CI (2-21)
1-5 decile
Results Criteria - Harm
  How strong is the association
between exposure and outcome?
Results – Precision?
Death No ABx Amoxicillin
Total 1 0.86
(0.58-1.28)
CV 1 0.95
(0.55-1.63)
Non-CV 1 0.76
(0.42-1.37)
Hazard Ratio (95% Confidence Interval)
HR - Statistical Significance
Hazard ratio (azithromycin/no antibiotic) for CV
death over 5 days = 2.88 95% CI (1.79 – 4.63)

  Hazard ratio = numerator/denominator


  If the two groups are the same then the HR will be 1
  A confidence interval for a hazard ratio that includes
1 is not statistically significant

  The confidence interval for the 2.88 hazard ratio


above does not include 1 so the hazard ratio is
statistically significant
Why do I need both an
absolute and relative
measure of the results?
Results – Hazard Rates (5days)
Death No ABx Amoxicillin Azithromycin
Total 52.6
CV 31.5
Non-CV 21.0
Hazard rate = number deaths / million courses of antibiotics
Relative vs Absolute Magnitude
  Relative – Hazard ratio (azithromycin/no
antibiotic) for CV death over 5 days HR
= 2.88 95% CI (1.79 – 4.63)

  Absolute – Hazard rate (death/million)


  No antibiotics – 29.8/1,000,000 = 0.00003
  Azithromycin – 85.2/1,000,000 = 0.000085
Absolute Risk Difference
  Hazard rates (CV death/million courses)
  No antibiotics – 29.8/1,000,000 = 0.00003
  Azithromycin – 85.2/1,000,000 = 0.000085

  Absolute risk difference (increase)


= Hazard rate (not exposed – exposed)
= Hazard rate (no antibiotic - azithromycin)
= 0.00003 – 0.000085 = - 0.000055 = - 0.0055%
English Translation – ARD
ARD (Azithro – No ABx) = - 0.0055%

A patient in the exposure group was ___%


less(ARD(+)) / more(ARD(-)) likely to have a
______ (outcome) then a patient in
the non-exposure group

A patient in the Azithromycin group was


0.0055% more likely to have a CV Death then a
patient in the No antibiotics group
Evidence of Causality
Association ≠ Causality
  Biologic plausibility
  Strength of association - magnitude

  Dose-response relationship

  Temporal sequence - predictor precedes

  outcome Consistency - results reproduced


Kaplan-Meier Survival Curves

Survival
Percent Survival

Time

Not Exposed

Exposed

Years
Temporal Relationship?

1-5 days 6-10 days


100
CV Deaths/Million Courses
50

Azithromycin

No Antibiotic
0

0 5 10
Days Since Prescription Filled
Applicability Criteria – Harm

  Was follow-up sufficiently long?


  Is the exposure similar to what might

occur in my patient?
  What is the magnitude of the risk?

  Are there any benefits that are known to

be associated with exposure?


Applicability – Patients Similar?
  UNCLEAR
  Demographic data presented

included only medications used,


coexisting condition and recent illness
  75% women - potentially limiting

generalizing the results to men


  Dependent on the CV risk profile in

your population
Applicability Criteria - Harm
  Were the study patients similar to
the patients in my practice?

  Is the exposure similar to what might


occur in my patient?
  What is the magnitude of the risk?

  Are there any benefits that are known to

be associated with exposure?


Applicability – Follow-up Duration?

  YES
  Short term cardiovascular death related

to the potential pro-arrhythmic effects of


azithromycin
  Data reported for 10 days after

the initiation of antibiotics


Applicability Criteria - Harm
  Were the study patients similar to
the patients in my practice?
  Was follow-up sufficiently long?

Data reported for 10 days after the initiation of antibiotics

  What is the magnitude of the risk?


  Are there any benefits that are known to

be associated with exposure?


Applicability – Similar Exposure?
  YES
  Azithromycin commonly to treat patients

with community acquired pneumonia


and chlamydia
Applicability Criteria - Harm
  Were the study patients similar to
the patients in my practice?
  Was follow-up sufficiently long?

  Is the exposure similar to what might

occur in my patient?

  Are there any benefits that are known to


be associated with exposure?
Applicability – Risk Magnitude?
  Relative – Hazard ratio (azithromycin/no
antibiotic) for CV death over 5 days HR
= 2.88 95% CI (1.79 – 4.63)

  Absolute – Hazard rate (death/million)


  No antibiotics – 29.8/1,000,000 = 0.00003
  Azithromycin – 85.2/1,000,000 = 0.000085
  ARD = - 0.000055 = - 0.0055%
Applicability Criteria - Harm
  Were the study patients similar to
the patients in my practice?
  Was follow-up sufficiently long?

  Is the exposure similar to what might

occur in my patient?
  What is the magnitude of the risk?
Applicability – Benefits?
  YES
  Treatment of infection

  Reduce complications of infection

 
Reduce spread of infection
Number Needed to Harm
  Risk of CV death over 5 days
  ARD = - 0.000055 = - 0.0055%

  NNH = 1/ARD = 1/0.000055 = 18,180


English Translation – NNH
NNH = 18,180

For every ___ (NNH) patients in the Exposed group 1


additional patient would have an ______ (outcome) over
time period X when compared to the Not exposed group

For every 18,180 patients in the Azithromycin group 1


additional patient would have a CV death over 5 days
when compared to the No antibiotic group
Clinical Bottom Line - Structure
  Background – context, why study was done?
  Study question – PICO format

 
Validity issues – major, minor, none
 
Primary results – clinically/statistically
 
significant
 
Applicability – your patient or population
Potential impact – change current practice?
Clinical Bottom Line - Study
  Case reports have implicated Azithromycin as
a cause of arrhythmias
  Do adult patients treated with Azithromycin

have an increased risk of CV death?


  Primary validity concerns are related to

observational studies such as recording


bias
  Magnitude of association CV death over 5 days
  Hazard ratio – 2.88 95% CI (1.79 - 4.63) Absolute
  risk increase - 0.0055%, NNH – 18,180
  Inadequate data on patient population
Summary – Harm Studies
  Observational studies typically used to assess
association of infrequent adverse events with exposures
  Observations studies have a higher potential for bias
  Relative and absolute measure of the magnitude
of association should be determined
  Practice saying the results in a sentence
  Confidence intervals - qualitative and
quantitative measure of statistical significance
  The NNH is a quantifiable measures of harm
QUESTIONS?

THE MAKING EBM SIMPLE SERIES