You are on page 1of 8

A C TA Obstetricia et Gynecologica

AOGS M A I N R E S E A R C H A R T I C L E

Vacuum extraction: development and test of a


procedure-specific rating scale
MATHILDE MAAGAARD1 , JEANETT OESTERGAARD1 , MARIANNE JOHANSEN1 , LISE LOTTE ANDERSEN2 ,
CHARLOTTE RINGSTED3 , BENT OTTESEN1 & JETTE L. SRENSEN1
1
Juliane Marie Centre for Children, Women and Reproduction, Rigshospitalet, University Hospital of Copenhagen,
Copenhagen, 2 Obstetric Department, Odense University Hospital, Odense, and 3 Centre for Clinical Education,
Rigshospitalet, University Hospital of Copenhagen, Copenhagen, Denmark

Key words Abstract


assessment, clinical competence, obstetrics,
medical education, vacuum extraction Objectives. To develop and validate an Objective Structured Assessment of Technical
Skills (OSATS) scale for vacuum extraction. Design. Two-part study design: Primar-
Correspondence
ily, development of a procedure-specific checklist for vacuum extraction. Hereafter,
Mathilde Maagaard, Juliane Marie Centre for
Children, Women and Reproduction,
validation of the developed OSATS scale for vacuum extraction in a prospective
Rigshospitalet, University Hospital of observational study. Setting. Rigshospitalet, University Hospital of Copenhagen.
Copenhagen, Obstetrics Department, Population. For development, an obstetric expert from each labor ward in Denmark
Blegdamsvej 9, section 4074, 2100 (28 departments) was invited to participate. For validation, nine first-year residents
Copenhagen, Denmark. and 10 chief physicians with daily work in the obstetric field were tested. Methods.
E-mail: mmn@rh.regionh.dk or The Delphi method was used for development of the scale. In a simulated vacuum
mathildemaagaard@hotmail.com
extraction scenario, first-year residents and obstetric chief physicians were rated
using the developed OSATS scale for vacuum extraction to test construct validity
Conflict of interest
The authors have stated explicitly that there of the scale. Main outcome measures. Consensus for the content of the scale. To
are no conflicts of interest in connection with test the scale of Cronbachs alpha, interclass correlation and differential item func-
this article. tion was calculated in the prospective study. Results. 89% completed the first and
61% completed the second Delphi round. Hereafter, consensus was obtained. There
Please cite this article as: Maagaard M, was a significant difference between residents and experts performance for total
Oestergaard J, Johansen M, Andersen LL,
score and for the score of the separate parts of the scale. Cronbachs alpha for total
Ringsted C, Ottesen B, Srensen JL. Vacuum
score and for the separate parts of the scale was 0.910.95 and interclass correlation
extraction: development and test of a
procedure-specific rating scale. Acta Obstet 0.840.9. Conclusions. The OSATS scale for vacuum extraction is a reliable test for
Gynecol Scand 2012;91:14531459. differentiating between competence levels in a simulated setting.

Received: 10 April 2012 OSATS, Objective Structured Assessment of Technical Skills;


Abbreviations:
Accepted: 1 August 2012 PROMPT, PRactical Obstetric Multi-Professional Training.

DOI: 10.1111/j.1600-0412.2012.01526.x

Introduction
Throughout the past two decades there has been increasing
focus on measuring doctors clinical performance. Conse- Key Message
quently, a need for reliable and validated assessment instru-
ments has emerged (1,2). An OSATS scale for assessment of physicians perfor-
An uncomplicated delivery can quickly change to an emer- mance in vacuum extraction was developed and was
gency situation requiring highly skilled staff. Vacuum extrac- shown to be a reliable test for differentiating between
tion is the most frequently used method for assisted vaginal competence levels in a simulated setting.


C 2012 The Authors

Acta Obstetricia et Gynecologica Scandinavica 


C 2012 Nordic Federation of Societies of Obstetrics and Gynecology 91 (2012) 14531459 1453
An OSATS scale for vacuum extraction M. Maagaard et al.

Figure 1. The Objective Structured Assessment of Technical Skills (OSATS) scale for vacuum extraction.

delivery in industrialized countries (813% of all deliveries) procedure-specific part consisting of 16 items (items 116).
(35). In 1998, the US Food and Drug Administration issued All items of the OSATS for vacuum extraction scale (global
a public health notice: Need for CAUTION When Using Vac- and procedure-specific) were rated on a 5-point scale where
uum Assisted Delivery Devices (6) due to the higher mor- 1, 3 and 5 points were anchored with a behavioral description.
tality and morbidity observed among children delivered after The scale is presented in Figure 1.
vacuum extractions. One of the Food and Drug Adminis- The global-rating part of the OSATS scale for vacuum
tration recommendations was: those persons using vacuum extraction was built on the original OSATS scale by Reznick
devices must be versed in the use. Hence, there is a need for et al. from 1997 except for the two items: use of assistants and
valid assessment instruments to ensure performance stan- knowledge of specific procedure. These were excluded since
dards and develop effective training programs. they were represented in more detail in the procedure-specific
The aim of this study was to develop and explore the relia- part of the revised scale. The remaining items were modified
bility, content and construct validity of a procedure-specific for general performance assessment in vacuum extraction.
rating scale for vacuum extraction. Content validity refers to The procedure-specific part of the OSATS scale for vacuum
the extent to which a scale represents all the important facets extraction was based on a nationally recognized in-training
of the procedure it is going to measure. Construct validity assessment checklist for residents (8). To improve face and
refers to whether this assessment instrument can differenti- content validity this checklist was further developed with
ate between supposedly different levels of competency. consensus-seeking involvement of chief physician represen-
tatives from all obstetrical departments in Denmark using
Material and methods a modified Delphi technique. The Delphi technique is an
Inspired by the Objective Structured Assessment of Technical anonymous process where responses are collected and ana-
Skills (OSATS) for surgical residents scale (7), a procedure- lyzed until consensus is achieved (9). Items may be excluded
specific rating scale for vacuum extraction was developed. or added. Each item from the national in-training assessment
The OSATS scale for vacuum extraction consisted of: (1) a checklist was rated from 1 to 5 points concerning its relevance
global-rating part including five items (item AE) and (2) a and ability to assess a residents performance in vacuum


C 2012 The Authors

1454 Acta Obstetricia et Gynecologica Scandinavica 


C 2012 Nordic Federation of Societies of Obstetrics and Gynecology 91 (2012) 14531459
M. Maagaard et al. An OSATS scale for vacuum extraction

Figure 1. Continued.

extraction. Furthermore, suggestions for additional items First-year residents and chief physicians were tested in a
were encouraged. An item was excluded if more than one simulated vacuum extraction scenario using the PRactical
chief physician rated it below 3. A new item was included if Obstetric Multi-Professional Training (PROMPT) birthing
recommended by more than 40%. The anchoring of items, simulator (10) and rated by the OSATS scale for vacuum ex-
both in the global and the procedure-specific part of the traction for testing construct validity. The high competence
OSATS scale for vacuum extraction, was developed as a col- level group (experts) was recruited randomly from differ-
laborative effort between the main author and three obstetric ent departments and all worked daily in the obstetric field.
experts and co-authors (J.L.S., M.J. and L.A.). The low competence level group (residents) was recruited


C 2012 The Authors

Acta Obstetricia et Gynecologica Scandinavica 


C 2012 Nordic Federation of Societies of Obstetrics and Gynecology 91 (2012) 14531459 1455
An OSATS scale for vacuum extraction M. Maagaard et al.

Figure 1. Continued.

after attending their first off-site mandatory course during tions for assistance. The following equipment was available:
the first year of residency and they had to have done fewer a timer, delivery kit, two types of vacuum extractors [soft (sil-
than five supervised vacuum extractions. None of the partic- icon) and hard Malmstrom (metal)], analgesia and scissors
ipants had previous experience with the PROMPT birthing for episiotomy if needed.
simulator. Furthermore, the participants were unaware of The participants were told to proceed with the delivery, as
which simulated clinical scenario they were going to partic- they would in a real clinical setting, explaining their thoughts
ipate in. Participation was voluntary and no patients were and actions aloud. The participants were asked to stop the
involved; therefore according to Danish regulations, ethical delivery once the head was born.
approval was not required. Each simulated scenario was video-recorded. Two inde-
Each participant was presented with the following scenario: pendent raters using the OSATS scale for vacuum extraction
You are called to the delivery room to attend a 30-year-old assessed the recordings. The raters were selected based on
nulliparous, who has been actively pushing (second stage of their expertise within obstetrics and were employed at two
labor) for one hour and twenty minutes. She is exhausted different university hospital departments. The raters were
and in despair. Cardiotocography is classified as reassuring blinded to the level of clinical experience of the participant.
(baseline 140/minute, normal variability, no complicated de- To familiarize the raters with the OSATS scale for vacuum ex-
celerations and accelerations are present). The fetuss head is traction and ensure rater consistency, the raters jointly rated
on the pelvic floor, in an occipito-anterior position. During three test scenarios. Each of the raters individually assessed
contractions there is no progression. Contractions are reg- the three scenarios and afterwards they discussed their ratings
ular (every second minute) and good. The parturient is re- to reach consensus. In Figure 2 a flow chart presents the steps
questing your assistance. In the delivery room is an auxiliary used in validating the OSATS scale for vacuum extraction in
nurse who is experienced but needs your specific instruc- a simulated setting.


C 2012 The Authors

1456 Acta Obstetricia et Gynecologica Scandinavica 


C 2012 Nordic Federation of Societies of Obstetrics and Gynecology 91 (2012) 14531459
M. Maagaard et al. An OSATS scale for vacuum extraction

Figure 2. Validation of the Objective Structured Assessment of Technical


Skills (OSATS) scale for vacuum extraction.

Statistical analyses
Data were processed using SPSS 18.0 for Windows. Due to
the sample size and nature of the results, a Gaussian distri-
bution was not expected; therefore a non-parametric statis-
tical test (MannWhitney U) was used to investigate differ-
ences between the groups. A p-value < 0.05 was considered
Figure 3. Validation of the in-training assessment checklist for residents
significant. Scores are presented as medians and quartiles.
by the Delphi method.
Interclass-correlation and Cronbachs alpha were calculated
separately for the total score, the global-rating part, and the
procedure-specific part. Differential item function was cal-
culated by logistic regression.

Results
Figure 3 presents a flow chart of the development of the
procedure-specific part of the OSATS scale for vacuum ex-
traction. All invited residents and experts entered the study.
The expert group consisted of 10 experts, five men and five
women, from four different hospitals. Each expert had per-
formed more than 100 vacuum extractions. The resident
group consisted of nine first-year residents. The residents
were all females and were employed in six different depart-
ments; they had been responsible for a vacuum-assisted deliv-
ery fewer than two times (range from 04) and none had seen
the procedure being performed by a more experienced doctor Figure 4. Box-plot of the score of the global-rating part, the procedure-
more than 10 times. They had all previously performed the specific part and total score of the Objective Structured Assessment of
procedure on a mannequin. Technical Skills (OSATS) scale for vacuum extraction for residents and
Figure 4 shows how the total score and the score for each expert physicians in a simulated scenario. Global median score: 11 (res-
separate part of the OSATS scale for vacuum extraction can idents) and 18.5 (experts), p = 0.006. Specific median score 39.5 (resi-
significantly differentiate between residents and experts per- dents) and 46.25 (experts), p = 0.037. Total median score 46 (residents)
and 63.25 (experts), p = 0.01.
formance levels.


C 2012 The Authors

Acta Obstetricia et Gynecologica Scandinavica 


C 2012 Nordic Federation of Societies of Obstetrics and Gynecology 91 (2012) 14531459 1457
An OSATS scale for vacuum extraction M. Maagaard et al.

The interclass-correlation coefficient based on the total A disadvantage of the OSATS scale for vacuum extraction
score (all items), the global-rating part (items AE) and the may be the complexity of many items. One way of overcoming
procedure-specific part (items 116) was 0.90, 0.85 and 0.84, this problem would be to rate video-recorded scenarios which
respectively. Cronbachs alpha for the same parts was 0.95, gives the option of re-winding to improve the quality of
0.92 and 0.91, respectively. the rating. Video-recorded simulated scenarios also allow
Differential item function analysis using logistic regression assessment of performance and feedback to be delivered in
was not useful in two of the 21 items, which could be ex- a non-stressful setting. We validated the OSATS scale for
plained by the size of this material. These were the items con- vacuum extraction in a simulated environment where it is
cerning safety parameters (numbers 14 and 15 in Figure 1). possible to create a standardized scenario that does not exist
No differential item function was found in the remaining in real life. Therefore testing the OSATS scale for vacuum
items, suggesting that there is no reason to exclude any items extraction in clinical settings requires subsequent testing of
from the OSATS scale for vacuum extraction (11,12). the same resident in several clinical situations due to inter-
case variability (26).
Nonetheless, a valid performance assessment instrument
Discussion is the first step towards developing a simulation-based train-
In this study an OSATS scale for vacuum extraction was ing program in vacuum extraction. In a descriptive survey
developed and tested. The OSATS scale for vacuum extraction we found previously that most departments are performing
was found reliable for differentiating between competence simulation-based skills training in obstetric emergencies, but
levels in a simulated setting. The scale was based on an existing standardization of the training programs and assessment are
national in-training assessment checklist for residents, which still lacking to back this up (27).
was further developed through a consensus-seeking Delphi This OSATS scale for vacuum extraction can be used for
method process to ensure the content validity of this. assessment of physician competence in a simulated setting
This study is the first to present a validated performance before practicing on patients. Furthermore, it probably has
assessment instrument for vacuum extraction in a simulated potentials as a benchmark for structured elaborated feedback
environment. The results of this study are consistent with as part of a training program. Hopefully, through a validated
other studies: both the original study of the OSATS scale (7) training program we can facilitate better clinical practice and
and studies using the OSATS or a modified version of it in hence improve patient safety in the future. The OSATS scale
simulated or clinical settings (1318). The original OSATS for vacuum extraction is reliable for differentiating between
scale consisted of a global-rating scale with seven items rated competence levels in a simulated setting and has excellent
on a 5-point scale (7). Reznick et al. have since published sev- inter-rater reliability and internal consistency.
eral studies with use of an OSATS scale and today they base the
performance assessment on a combination of a global-rating
Funding
part for general skills and a checklist part for procedure-
specific skills (1922). The global-rating part is always rated No special funding.
on a 5-point scale, whereas the checklist part is mostly rated
on a dichotomous scale. However, the literature is versatile References
and many different ways of interpreting the OSATS scale can 1. Johannsson H, Ayida G, Sadler C. Faking it? Simulation in
be found (15,1925). We believe that the checklist part of the training of obstetricians and gynaecologists. Curr Opin
the OSATS scale should be rated on a 5-point scale, where Obstet Gynecol. 2005;17:55761.
1, 3 and 5 points are anchored with a behavioral description 2. Veltman LL. Getting to havarti: moving toward patient safety
rather than on a dichotomy scale, as the former seems to be in obstetrics. Obstet Gynecol. 2007;110:114650.
the best base for a clarified and elaborated feedback. For the 3. Murphy DJ, Liebling RE, Verity L, Swingler R, Patel R. Early
same reason we chose to call this part of the OSATS scale for maternal and neonatal morbidity associated with operative
vacuum extraction the procedure-specific part instead of the delivery in second stage of labour: a cohort study. Lancet.
checklist part. No matter how the scale is constructed, it is 2001;358:12037.
crucial to validate it in the context in which it is intended to 4. OMahony F, Settatree R, Platt C, Johanson R. Review of
be used. singleton fetal and neonatal deaths associated with cranial
We found the global-rating part to be superior in differenti- trauma and cephalic delivery during a national
ating the level of competency, whereas the procedure-specific intrapartum-related confidential enquiry. Br J Obstet
part is more complex in nature and will probably be of ben- Gynaecol. 2005;112:61926.
efit as a guide for feedback to the individual physician. Our 5. Towner D, Castro MA, Eby-Wilkens E, Gilbert WM. Effect of
results also showed that, depending on the situation, each mode of delivery in nulliparous women on neonatal
part of the scale can be used separately. intracranial injury. N Engl J Med. 1999;341:170914.


C 2012 The Authors

1458 Acta Obstetricia et Gynecologica Scandinavica 


C 2012 Nordic Federation of Societies of Obstetrics and Gynecology 91 (2012) 14531459
M. Maagaard et al. An OSATS scale for vacuum extraction

6. http://www.fda.gov/MedicalDevices/Safety/Alertsand 18. VanBlaricom AL, Goff BA, Chinn M, Icasiano MM, Nielsen
Notices/PublicHealthNotifications/ucm062295.htm. 1998. P, Mandel L. A new curriculum for hysteroscopy training as
Ref Type: Internet Communication. demonstrated by an objective structured assessment of
7. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, technical skills (OSATS). Am J Obstet Gynecol.
Hutchison C, et al. Objective structured assessment of 2005;193:185665.
technical skill (OSATS) for surgical residents. Br J Surg. 19. Anastakis DJ, Wanzel KR, Brown MH, McIlroy JH, Hamstra
1997;84:2738. SJ, Ali J, et al. Evaluating the effectiveness of a 2-year
8. http://www.dsog.dk/files/Checkliste%20gynobs%202007. curriculum in a surgical skills center. Am J Surg.
pdf. 2011. Ref Type: Internet Communication. 2003;185:37885.
9. Graham B, Regehr G, Wright JG. Delphi as a method to 20. Ault G, Reznick R, MacRae H, Leadbetter W, DaRosa D,
establish consensus for diagnostic criteria. J Clin Epidemiol. Joehl R, et al. Exporting a technical skills evaluation
2003;56:11506. technology to other sites. Am J Surg. 2001;182:
10. http://limbsandthings.com/global/products/prompt- 2546.
birthing-simulator-standard/. 2011. Ref Type: Internet 21. Dorman K, Satterthwaite L, Howard A, Woodrow S,
Communication. Derbew M, Reznick R, et al. Addressing the severe
11. Swaminathan H & Rogers HJ. Detecting differential item shortage of health care providers in Ethiopia: bench
functioning using logistic regression procedures. model teaching of technical skills. Med Educ. 2009;43:
J Educ Measurement. 1990;27:36170. 6217.
12. Zumbo BD. Handbook on the Theory and Methods of 22. Schijven MP, Reznick RK, ten Cate OT, Grantcharov TP,
Differential Item Functioning (DIF): Logistic Regression Regehr G, Satterthwaite L, et al. Transatlantic comparison of
Modeling as a Unitary Framework for Binary and Likert-type the competence of surgeons at the start of their professional
(Ordinal) Item Scores. Ottawa: Department of National career. Br J Surg. 2010;97:4439.
Defence, Directorate of Human Resources and Evaluation. 23. Collins AM, Ridgway PF, Hassan MS, Chou CW, Hill AD,
1999. Kneafsey B. Surgical instruction for general practitioners:
13. Datta V, Chang A, Mackay S, Darzi A. The relationship how, who and how often? J Plast Reconstr Aesthet Surg.
between motion analysis and surgical technical assessments. 2010;63:115662.
Am J Surg. 2002;184:703. 24. Setna Z, Jha V, Boursicot KA, Roberts TE. Evaluating the
14. Datta V, Bann S, Mandalia M, Darzi A. The surgical utility of workplace-based assessment tools for speciality
efficiency score: a feasible, reliable, and valid method of skills training. Best Pract Res Clin Obstet Gynaecol.
assessment. Am J Surg. 2006;192:3728. 2010;24:76782.
15. Goff B, Mandel L, Lentz G, Vanblaricom A, Oelschlager AM, 25. Siddiqui NY, Stepp KJ, Lasch SJ, Mangel JM, Wu JM.
Lee D, et al. Assessment of resident surgical skills: is testing Objective structured assessment of technical skills for repair
feasible? Am J Obstet Gynecol. 2005;192:13318. of fourth-degree perineal lacerations. Am J Obstet Gynecol.
16. Goff BA, Lentz GM, Lee D, Houmard B, Mandel LS. 2008;199:676.
Development of an objective structured assessment of 26. Konge L, Larsen KR, Clementsen P, Arendrup H, von BC,
technical skills for obstetric and gynecology residents. Obstet Ringsted C. Reliable and valid assessment of clinical
Gynecol. 2000;96:14650. bronchoscopy performance. Respiration. 2012;83:
17. Larsen CR, Grantcharov T, Schouenborg L, Ottosen C, 5360.
Soerensen JL, Ottesen B. Objective assessment of surgical 27. Maagaard M, Johansen M, Lottrup P, Sorensen JL. Clinical
competence in gynaecological laparoscopy: development and skills training in obstetrics-a descriptive survey of current
validation of a procedure-specific rating scale. Br J Obstet practice in Denmark. Acta Obstet Gynecol Scand.
Gynaecol. 2008;115:90816. 2012;91:1436.


C 2012 The Authors

Acta Obstetricia et Gynecologica Scandinavica 


C 2012 Nordic Federation of Societies of Obstetrics and Gynecology 91 (2012) 14531459 1459
Copyright of Acta Obstetricia et Gynecologica Scandinavica is the property of Wiley-Blackwell and its content
may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express
written permission. However, users may print, download, or email articles for individual use.