Sistrom and Informatics • Origina l Rese arch

Honeyman-
Buck
Information
Transfer:
Free Text vs
Structured
Format

Free Text Versus Structured

Format: Information Transfer
Efficiency of Radiology Reports
Chris L. Sistrom1 OBJECTIVE. We discuss the effect of radiology report format on the accuracy and speed
Janice Honeyman-Buck with which reviewers can extract case-specific information.
MATERIALS AND METHODS. A Web-based testing mechanism was used to present
Sistrom CL, Honeyman-Buck J radiology reports to each of 16 senior medical students and record their answers to 10 multiple
choice questions about specific medical content for each of 12 cases. Subjects were randomly
assigned to view the reports in either free text or structured format. In addition to number of
answers correct for each case, we recorded the time taken for each case and an efficiency score
(correctly answered questions per minute). These three outcomes were tested for differences
on report format using multifactorial analysis of variance. A postexperimental questionnaire
and a mediated focus group elicited subject preference as to radiology report format.
RESULTS. There were no significant differences in the three outcomes (score, time, and
efficiency) between the free text and structured format conditions. The power of the experiment
was sufficient to detect small differences in these outcomes by format. Subjects strongly and
consistently expressed a preference for the structured version.
CONCLUSION. We assert that free text and itemized (structured) forms of radiology re-
ports are equally efficient and accurate for transmitting case-specific interpretative content to
reviewers of the document.

Background and Significance The distinction between reporting into

Received November 1, 2004; accepted after revision
December 8, 2004.
C. L. Sistrom was funded by the General
Electric/Association of University Radiologists Research
Fellowship from July 2000 through June 2003.
1Both authors: Department of Radiology, University of
Florida Health Center, PO Box 100374, Gainesville, FL 32610.
Address correspondence to C. L. Sistrom
(sistrc@radiology.ufl.edu).
AJR 2005; 185:804–812
0361–803X/05/1853–804
© American Roentgen Ray Society
During his inaugural address to the 2004
ARRS Meeting, the incoming president,
Christopher R. B. Merritt, spoke extensively
about radiology reporting [1]. He emphasized
the need for using new technologies to
achieve standardization and structure to facil-
itate clinical care, research, and compliance.
In a 2005 review on the subject, Sistrom and
Langlotz [2] described a framework for im-
proved radiology reporting that articulated
three attributes as targets for improvement.
These were standard language, structured for-
mat, and consistent content [2]. In consider-
ing different options for report format, we be-
lieve that the choices made will affect the
process of care in two distinct ways. First,
during report creation, a predefined format
may fundamentally alter the way that the in-
terpreting physician thinks about the case as
he or she produces the document. We call this
reporting into structure. Second, during re-
port review, the choice of different report for-
mats may affect the way in which the reader
comprehends the information. We call this
reading structure.
structure and reading structure is by no means
purely academic. Current technology allows a
clinical document to be produced in one format
and displayed in an entirely different way. For
example, one vendor of a structured computer-
ized reporting product (eDictation) uses a so-
phisticated interface to allow radiologists to
choose relevant findings from a large and
highly structured menu of possibilities. At the
same time, the software produces phrases and
sentences that look just like a typical narrative
report and it is this document that is made
available for referring physicians to review.
The assumption is that clinicians will be more
comfortable with radiology reports that look
like what they are used to reading, that is, nar-
rative free text. Our research was designed to
obtain empiric data about this question. We
sought to examine the clinical utility of radiol-
ogy reports in terms of information transfer to
the reader. Specifically, what are the effects of
report format (independent of content) on the
efficiency with which medical personnel can
read them and obtain information needed for
patient care? Our working hypothesis was that

804 AJR:185, September 2005

 Information Transfer: Free Text vs Structured Format
Fig. 1—Screen capture images from the Web-based testing program used to perform the experiment.
A–C, Students first were presented with a clinical scenario page (A). Pressing the button labeled GO TO THE
REPORT caused the second page (B) to be displayed with the report text. Pressing the button labeled GO TO THE
QUESTIONS caused a third page (C) to appear. This contained the 10 questions pertaining to the report and had
buttons labeled GO BACK TO THE REPORT that caused redisplay of the report.
(Fig. 1 continues on next page)
consistently formatted (structured) reports
would be easier to read and comprehend result-
ing in greater efficiency for the task of answer-
ing content-specific questions.
Materials and Methods
Following approval for the research by our local
institutional review board (IRB), radiology reports
were selected from our departmental archive of rou-
tine clinical examinations performed between Janu-
ary 2001 and December 2002. Four reports were se-
lected from each of three types: songraphy of the
abdomen, CT of the abdomen, and head CT without
contrast. All examinations had been performed on
patients referred from the emergency department or
AJR:185, September 2005
A nior radiology residents and asked them to evaluate
B neys headings in the structured version. If a heading
outpatient clinics. We did not select the cases at ran-
dom but rather looked for reports that had some rel-
atively common abnormalities. We excluded reports
that described findings that might be considered as
unusual, emergent, or unexpected. All 12 reports
were blinded by removing patient identifying infor-
mation and any reference to proper names in the text.
For each report, we generated a series of 10 mul-
tiple-choice items, each having a question stem and
from 3–5 options. These were designed to have a
single option that was unambiguously correct
based only on the content of the report to which
they referred. The reports and candidate questions
were administered to three senior medical students
using a Web-based testing system (described be-
low) that allowed them to refer back to the report
text as needed. These students did not participate as
subjects in the subsequent experiment. We also
gave printed versions of the questions and associ-
ated reports to two faculty radiologists and two se-
them for clarity and consistency. Using feedback
given to us by all of these evaluators, we corrected
the wording of one of the question stems and three
of the options (all distracter items). We also elimi-
nated one option (also a distracter).
The original reports were in narrative format with
variable use of headings for indication, comparison,
examination details, and findings. All of the original
reports had a labeled impression section. These 12
reports along with the 10 questions pertinent to each
one formed the free text condition of the experiment.
A report structure shell was designed for each of the
three types of studies. These consisted of the compo-
nents typically found in narrative radiology reports
with additional headings in the findings section. For
abdominal CT and sonography, these basically were
anatomic. For head CT, we combined anatomic and
functional headings. The templates for all three ex-
amination types are listed in Appendix 1.
For each case, we parsed the free text report into
the appropriate structured template. This was done
so that all the original content was exactly and com-
pletely replicated in the structured version. We
were careful to leave basic sentence structure and
word choice intact. We duplicated or eliminated
words or phrases as needed to maintain proper syn-
tax in the structured version. For example, the free
text might contain “The pancreas, spleen, and kid-
neys are unremarkable.” The word “unremarkable”
would be placed after the pancreas, spleen, and kid-
in the structured template did not have any relevant
content from the free text version, it was left in the
report with no text after it. Appendix 2 is an exam-
ple of a free text abdominal CT report and Appen-
dix 3 contains the structured version. With 12
unique cases, each having two versions (free text
and structured format), there were 24 total cases.
Experimental Details
Our College of Medicine uses a locally developed
Web-based testing system for all examinations ad-
ministered to students. It has been in use for more
than 7 years and by the time our medical students
have reached their fourth year, they have taken at
least 20 tests using it. Since we used the same system
to administer our experimental tests, all of the sub-
jects were quite familiar with its function and ap-
pearance. This eliminated any need to train subjects
before their participation and reduced any potential
variability in performance based on differential
learning of the testing procedure itself.
805

Fig. 1 (continued)—Screen capture images from the Web-based testing program used to perform the experiment.
A–C, Students first were presented with a clinical scenario page (A). Pressing the button labeled GO TO THE
REPORT caused the second page (B) to be displayed with the report text. Pressing the button labeled GO TO THE
QUESTIONS caused a third page (C) to appear. This contained the 10 questions pertaining to the report and had
buttons labeled GO BACK TO THE REPORT that caused redisplay of the report.
The experimental testing software was designed
to present each case as a set of three Web pages
joined into a common frame set. The first page gave
a brief description of a clinical presentation derived
from the clinical history and the stated reason for
the examination in question. A button (labeled GO
TO THE REPORT) on the first page caused a sec-
ond page containing the report text to be displayed.
Buttons above and below the report text (labeled
GO TO QUESTIONS) served to advance to the
third page containing all 10 questions relating to the
report. The question page also had buttons (labeled
GO BACK TO THE REPORT) at the top and bot-
tom that caused redisplay of the report text while
keeping the question page active in the background,
thus preserving its state. Subjects could switch back
and forth between the report and questions as often
as needed and their previous answers would remain
intact. Figure 1 depicts the three frames making up
a single test case.
The Web pages all had JavaScript code (Sun Mi-
crosystems) embedded in them to record a time
stamp (at 1/10 sec precision) for every button click
and item selection as the subject navigated through
the case and answered questions. A submit button
on the question page served to record the answers
806
Sistrom and Honeyman-Buck
C Testing of all 16 subjects (5 women and 11 men)
and the time-stamped navigation data. Submission
was not accepted until all 10 questions had been an-
swered. It is important to note that the design of the
testing mechanism caused the entire set of pages
comprising a single case to be transferred from
Web server to the local computer when the subject
was ready to start. All navigation and recording of
time stamps was accomplished locally and required
no further traffic between the testing computer and
the server. Thus, there was no possibility that tim-
ing during a single case might be confounded by
variations in network speed. All testing was per-
formed on a single workstation located in a quiet
room. This computer had a single 18-inch flat panel
color monitor. Relevant software included Win-
dows 2000 professional and Microsoft Internet Ex-
plorer. It was connected to our hospital’s internal
network.
All of the subjects of the experiment were senior
medical students at our institution. They were re-
cruited by means of fliers posted in the medical
school teaching complex. The research (and the
flier) had prior approval by our local IRB. They
were required to have taken and passed their medi-
cine and surgery clinical clerkships before partici-
pating in the research. For incentive, a $100 account
was opened for each participant at the College of
Medicine bookstore. Each of the subjects was as-
signed all 12 cases to do during a single experimen-
tal session. Six of these cases were presented with
the free text version of the report and the remaining
six with the structured version of the report. The as-
signment of which of the 12 cases was presented to
each subject as free text versus structured was ran-
domized, with one restriction: We balanced the
number of times each case was presented in free text
versus structure across the entire experiment. The
order in which subjects took their 12 cases was ran-
domized, with one restriction: We balanced the as-
signment so that each case would be presented an
equal number of times in each quartile of the case
order (1–3, 4–6, 7–9, 10–12). Our initial power cal-
culations called for eight subjects. This allowed a
symmetric design in both the format and case order
factors. We performed a second repetition of the ex-
periment with an additional eight subjects, resulting
in a total of 192 sets of case/subject responses for
analysis. During the second repetition, the same ran-
domization scheme was used with the only differ-
ence being that the assignment of the free text or
structured format version of the report was reversed.
was completed in the 2002–2003 academic year.
There were no dropouts or technical failures during
the experiment and each of the subjects completed
their 12 cases in the sequence assigned during a sin-
gle session and none reported any problems with the
testing system.
Following completion of the testing, 15 of the 16
subjects met with the principal investigator in a
“debriefing” focus group. Before this meeting, sub-
jects had been told only that the purpose of the ex-
periment was to assess their ability to extract infor-
mation from reports. At the beginning of the
meeting, subjects were given a brief questionnaire
to fill out. This asked several general questions
about radiology report format and content. Next, a
brief paragraph defined free text and structure fol-
lowed by an example of each format (similar to
Tables 1 and 2). Next, a set of eight Likert-scaled
items asked for their preference concerning various
functional aspects of reading radiology reports (ac-
curacy, speed, certainty, items not mentioned, pos-
itive findings, negative findings, and general pref-
erence) based on their experience with the test
cases. The preference scores were anchored as fol-
lows: 1 = prefer free text, 5 = no preference,
10 = prefer structure. A mediated discussion was
then conducted to elicit opinions about radiology
report structure and content. Note that subjects
were not given any feedback—after they partici-
pated or during the debriefing meeting—concern-
ing individual or aggregate performance in answer-
ing questions for the cases.
AJR:185, September 2005
 Information Transfer: Free Text vs Structured Format

TABLE 1: Results of Analysis of Variance for the Number of Correctly Answered Questions Per Case
Source Effect Type Degrees of Freedom Mean Square Error F Value Pr > F
Format (free text vs structure) Fixed 1 1.069 0.86 0.3541
Case ID (case type) Random 8 5.254 4.25 0.0001
Subject Random 15 11.783 9.52 < 0.0001
Order Fixed 11 1.521 1.23 0.2734
Format × subject Random 14 1.580 1.28 0.2292
Error 1 — 139 1.237 — —
Case type Fixed 2 0.482 0.09 0.9153
Error 2 — 7.879 5.388 — —
Note—Pr = probability, — = not applicable.

TABLE 2: Results of Analysis of Variance for the Number of Seconds Taken to Complete Each Case
Source Effect Type Degrees of Freedom Mean Square Error F Value Pr > F
Format (free text vs structure) Fixed 1 3,056 0.67 0.4129
Case ID (case type) Random 8 24,503 5.41 < 0.0001
Subject Random 15 80,762 17.82 < 0.0001
Order Fixed 11 1,251 0.28 0.9893
Format × subject Random 14 2,866 0.63 0.8343
Error 1 — 139 4,531 — —
Case type Fixed 2 46,880 1.86 0.2174
Error 2 — 7.905 25,171 — —
Note—Pr = probability, — = not applicable.

Statistical Analysis
Each subject’s participation generated a set of 12
experimental results. These consisted of answers to
the 10 questions for a case and the time-stamped
navigation data. We scored the subject’s answers
against a key to obtain the number correct. The start
time was subtracted from the final submission time
to obtain number of seconds taken to do each case.
An efficiency score was then calculated for each
case by dividing the number of questions answered
correctly by the number of seconds taken to finish
the entire case. This result was multiplied by 60 to
give the number of correctly answered questions per
minute. The time-stamped navigation activity
records were processed to obtain two outcomes for
each case. The number of times the subject moved
back to view the report from the question page was
tabulated. This outcome could take any positive in-
teger and will be called report views. The number of
answer selections made during each case was
counted as well. This outcome was at least 10 and
was higher when subjects changed their minds about
one or more answers. Thus, there were five out-
comes analyzed: number of questions correct, time
in seconds taken to complete the case, efficiency
score, report views, and answer selections.
We used the Statistical Analysis System (SAS
Version 9 for Windows, SAS Institute) for all data
manipulation and statistical calculation. For all
tests of significance, we set p = 0.05 as the cutoff
and used two-tailed alternate hypotheses. The anal-
ysis was done on the basis of a balanced incomplete
block design. The factors included examination
type (3 levels), report format (2 levels), and indi-
vidual cases (4 levels per type). Thus, there were 24
factor level combinations (treatments), a block size
of 12 (cases per subject), and eight blocks (sub-
jects). The experiment was replicated twice with
two groups of eight subjects for a total of 16 sub-
jects and 192 experimental units. Summary statis-
tics for the five outcomes were generated, including
mean, median, mode, SD, frequency distribution
plot, and normal probability plot.
We performed analysis of variance with a gen-
eral linear model procedure (SAS PROC GLM) to
test for differences in each of the five outcomes
(percent of questions correct, time taken, effi-
ciency, report views, and answer selections)
jointly related to our independent variables. We
used the same linear model for each outcome.
Fixed effects included report format, case type,
and the order that the case was presented to the
subject. Random effects included case identity
nested within case type, subject identity, and re-
port format crossed with subject identity. The
model was specified as follows:
Outcome = β1 format + β2 case type +
β3 caseID(case type) + β4 subjectID +
β5 format × subjectID + β6 order + Error
Standard F statistics using the type 3 sums of
squares and appropriate error terms were used to
test the coefficients (β1 − β6) against the null hy-
pothesis of no effect (βi = 0). The Duncan proce-
dure was used to perform multiple comparisons of
mean number of report views by case type [3].
Since none of the fixed effects was significant in
any of the other models, no additional multiple
comparisons were performed. The two outcomes
relating to subjects’ test-taking habits (report views
and answer selections) were correlated within sub-
jects, formats, case types, and overall to obtain
Pearson correlation coefficients [4].
Because our results showed equivalence on the
main variable of interest (report format) we per-
formed post hoc power analysis. The sample size
was initially set with eight subjects each looking at
12 cases for a total of 96 experimental units. We
were able to double the planned sample size because
many students responded to the request for participa-
tion and running the experiments was quite easy due
to all subjects’ familiarity with the testing system.
Our analysis was based on paired testing between
free text and structured format with α = 0.05 and

AJR:185, September 2005 807

 Sistrom and Honeyman-Buck

β = 0.01 (90% power). We used the root mean
square error from the analysis of variance output as
the estimate of sigma for each outcome.
For the postexperimental survey, Likert-scaled
items from the debriefing questionnaires were sum-
marized by calculating the median value, the 10th
percentile, and the 90th percentile. The general
preference items were enumerated and percentages
calculated. The qualitative content of the debriefing
focus group was summarized from a transcription
of a tape recording made during the session.
Results
Experimental Results
Examination of the response data from the
testing system revealed that all 16 subjects
submitted valid responses to the 10 questions
for each of their 12 cases and that they per-
formed them in the assigned order. The timing
data were complete and consistent with the an-
swer responses with no gaps, extra entries, or
ambiguities. Thus, there were 192 complete re-
sponse sets available for analysis. These con-
sisted of timing data and answers to all 10
questions for each of 12 cases completed by
our 16 subjects. Each case was shown eight
times in its free text form and eight times with
the structured format. What follows are
univariate statistics on each outcome for the
entire data set (n = 192) along with the multi-
factorial analysis of variance results.
The number of correct responses (score)
ranged from two to 10 with a mean of 8.35
and SD of 1.52. The distribution was nega-
tively skewed with median and mode both be-
ing nine. None of the fixed effects (format
p = 0.35, order p = 0.27, and case type
p = 0.92) were significant and there was no
interaction between format and subject ef-
fects on the score. The majority of the vari-
ance (60%) was partitioned between subjects
and between cases within case type and R
squared for the model was 0.58. The analysis
of variance results are reproduced in Table 1.
The time taken to complete the cases
ranged from 30 to 707 sec with mean of 351
and SD of 108. The distribution was not
skewed with a median of 341 but there was
some kurtosis. The single observation at 30
sec was a distinct outlier with the next lowest
value being 102 sec (first percentile). None of
the fixed effects were significant (format
p = 0.41, order p = 0.99, and case type
p = 0.22) and there was no interaction be-
tween format and subject effects on the time.
The majority of the variance (62%) was par-
titioned between subjects and between cases
within case type and R squared for the model
was 0.71. The analysis of variance results are
reproduced in Table 2.
The efficiency (number of correctly an-
swered questions per minute) ranged from
0.52 to 4.7 with mean of 1.56 and SD of 0.56.
The distribution was nearly normal except
that the positive tail was longer. This was due
to the same outlier (30 sec to complete the
case) mentioned above. None of the fixed ef-
fects were significant (format p = 0.92, order
p = 0.60, and case type p = 0.48) and there
was no interaction between format and sub-
ject effects on the efficiency. Just under half
of the variance (46%) was partitioned be-
tween subjects and between cases within case
type and R squared for the model was 0.57.
The analysis of variance results are repro-
duced in Table 3.
The number of times that any answer was
selected for each case (answer selections)
ranged from 10 (obligate floor value) to 19
with a mean of 11.5, SD of 1.68, median of
11, and mode of 10. The distribution was, as
expected, not normal and looked more like a
Poisson type with a mean of 11.5. We elected
to proceed with analysis of variance despite
the violation of normality because this out-
come was considered to be secondary and the
results were relatively uninteresting. The only
significant effect was between subjects. Re-
port format and case type had the same mean
number across all levels.
The number of times subjects went back to
look at the report text (report views) while an-
swering questions ranged from two to 32 with
a mean of 12, median of 11, and SD of 6.8.
The distribution was near normal, allowing for
the discrete nature of the outcome. The
analysis of variance analysis showed no dif-
ference in the mean number by report type
(structured = 13.4, free text = 12.3, p = 0.93).
Again, there was considerable variance be-
tween subjects (p < 0.0001) with a minimum
of 4.7 times per case ranging up to 25 times
per case. Interestingly, there was a significant
difference between the type of case (p =
0.0016) even though there was no significant
difference between the individual cases
(p = 0.11). The mean number of times the re-
port was consulted for abdominal CT (13.1)
was essentially the same as for abdominal
songraphy (12.9). However, for head CT, sub-
jects went back to look at the report an average
of 11 times. We tested for interaction between
case type and subject effects and found none.
Therefore, the tendency to look back at head
CT reports less frequently than sonography
and abdomen CT was shared by all subjects.
Across the entire sample of 192 cases, the
correlation between report views and answer
selections was weakly positive with a Pearson
coefficient of 0.22 (p = 0.002). When we ex-
amined the relationship between report views
and answer selections for each subject, only
three of the 16 had significant correlations.
These were all positive (0.58, 0.64, 0.75) and
probably accounted for the aggregate correla-
tion. The correlations between report views
and answer selections stratified by case type
and report format were all weakly positive
(0.18 to 0.28). The one exception was the
head CT cases, where there was no correla-
tion between numbers of report views and an-
swer selections.

TABLE 3: Results of Analysis of Variance for Efficiency of Answering Questions (Score / Time) × (60)
Source Effect Type Degrees of Freedom Mean Square Error F Value Pr > F
Format (free text vs structure) Fixed 1 0.002 0.01 0.9172
Case ID (case type) Random 8 1.078 6.08 < 0.0001
Subject Random 15 1.126 6.36 < 0.0001
Order Fixed 11 0.148 0.84 0.6040
Format × subject Random 14 0.259 1.46 0.1339
Error 1 — 139 0.177 — —
Case type Fixed 2 0.881 0.80 0.4843
Error 2 — 7.915 1.108 — —
Note—Pr = probability, — = not applicable.

808 AJR:185, September 2005

 Information Transfer: Free Text vs Structured Format
The head CT cases seemed to elicit a some-
what different pattern of report viewing unre-
lated to the number of times answers were se-
lected. Considering that the head CT
structured format was functionally organized
rather than strictly by anatomy, we wanted to
be sure there was no interaction between the
type of case and our main independent vari-
able of interest, the report format. When we
added this interaction term to the analysis of
variance models for score, time, and effi-
ciency, we were reassured to find that it was
not significant for any of the outcomes. Thus,
the finding that report format had no effect on
the outcomes is conclusive and holds across
case type.
For the post hoc power analysis, the root
mean square errors (sigma) were as follows:
score = 1.32, time (seconds) = 65, and effi-
ciency = 0.43. As described above, we set
α = 0.05 and power to 90%. For score, we
could have detected a difference of about 0.5
in the mean of correctly answered questions.
The observed mean scores were 8.28 for free
text and 8.43 for structured format, with a dif-
ference of 0.15. For time to complete each
case, we could have detected a difference of
about 30 sec. The observed mean times were
355 sec for free text and 347 sec for structured
format with a difference of 8.4 sec. Finally,
for efficiency of answering questions, we
could have detected a difference of about 0.2
questions per minute. The observed effi-
ciency for free text was 1.57 questions per
minute and for structured format, it was 1.55
questions per minute for a difference of 0.02
questions per minute. For all three of the main
outcomes of interest, the observed effect of
report format was far less (approaching an or-
der of magnitude) than the difference our ex-
periment was powered to detect.
Qualitative Results
The debriefing meeting was attended by 15
of 16 subjects. One of the male students was
serving in a clinical clerkship at another insti-
tution. One of the general questions asked for
preference about the report organization. One
option to this question was “like a laboratory
report.” By this we meant standardized head-
ings in the body with results organized under
these headings. The choices and number of
responses were as follows. Like a laboratory
report (11/15 = 73%), like a newspaper story
(1/15 = 7%), and in the current (unstructured)
format (3/15 = 20%). We also asked how
they would prefer to have uncertainty ex-
pressed. The choices and responses were in
AJR:185, September 2005
words (8/15 = 53%), as a semiquantitative
scale (3/15 = 20%), and in quantitative terms
(2/15 = 13%). The final general question
asked about how subjects would respond to
an explicitly worded recommendation in a ra-
diology report. The responses were as fol-
lows. The subject would be compelled to fol-
low the recommendation (2/15 = 13%), they
might be compelled to follow the recommen-
dation (10/15 = 67%), and they would not
feel so compelled (3/15 = 20%).
The responses to the Likert-scaled ques-
tions about preference between free text (1)
and structured format (10) are summarized in
Table 4 with median, mode, and range for
each one. Clearly, the subjects strongly
tended to prefer the structured format for the
seven separately articulated domains as well
as overall. During the mediated discussion,
this perception was reinforced. At least five
subjects clearly expressed the opinion that
they would like to see all radiology reports
formatted in a manner similar to our struc-
tured condition (as in Appendix 3). One par-
ticipant mentioned that the headings should
be consistent across all instances of a report
type. He said “it might be confusing if some
people include biliary system under liver and
some people included it under gallbladder,
for example.” Others felt that the order of
headings should be altered dynamically so
that the abnormal findings would be at the
top of the report. One potential drawback to
the structured format was expressed as fol-
lows: “The overall gestalt of the examination
might be lost in structure whereas with free
text a sense of severity and more acuity can
be conveyed.” A corollary opinion that was
expressed quite forcefully and frequently
was that reports should still have a clearly la-
beled impression section. The concept of or-
ganizing a report like a newspaper story was
linked to the need for an impression. Like a
TABLE 4: Summary of Responses to Format Preference Questions
Question—Preference With Respect To:
Your accuracy (number correct)
Your speed in finding answers
Your efficiency in finding answers
Your certainty of knowing the answer
Knowing what was not mentioned in the report
Negative findings
Positive findings
General preference
Note—Questions were all Likert scaled with 1 = free text, 5 = no preference, and 10 = structured format.
news story, subjects wanted to see reports
have an equivalent of the lead paragraph in
which findings are synthesized and con-
densed to allow rapid review and to highlight
the diagnostic impression.
Discussion
The working hypothesis of the experiment
was that our subjects would take significantly
less time to read a structured report and an-
swer questions about its content than they
would with the free text version. We were not
certain that subjects would attain higher or
even equal scores with the structured version.
To allow for this, we introduced a measure of
efficiency (correctly answered questions per
minute). The pattern of outcomes that we ex-
pected was less time for structure, equal to
slightly lower scores (accuracy) with struc-
ture, and increased efficiency with structure.
To our surprise, there were no significant dif-
ferences on any of the three measures. The
planned power to detect differences between
free text and structure was achieved and in-
deed exceeded in the experiment. Further-
more, all hypotheses were tested using two-
tailed methods thus allowing for either alter-
native to the null of equivalence.
We assert that there is no effect of report
format on speed, accuracy, or efficiency for
our subject population reading the types of re-
ports we presented to them. By extension, we
suggest that the same phenomenon may per-
tain to practicing physicians. If this is true, de-
signers of reporting systems may not need to
work so hard to produce old-style narrative
documents out of structured elements. At the
same time, it would seem that free text reports
are not as difficult to read for content as some
believe. The choice of report format and
structure may be made on the basis of refer-
ring physician preference and considering the
effect of reading into structure by radiolo-
Median Mode Range
9 10 6–10
9 10 3–10
9 10 6–10
9 10 5–10
10 10 5–10
9 10 2–10
9 10 7–10
10 10 3–10
809

gists. However, we advise those seeking to
fundamentally change the way in which radi-
ology reports are created and displayed to
proceed with caution in consideration of the
following. In a recent article, Ash et al. [5]
discussed unintended consequences of over-
emphasizing structured information entry in
health care informatics. They cite evidence
from studies in cognitive psychology and so-
ciology that in a shared context, concise, un-
constrained, free text communication is the
most effective for coordinating work around a
complex task (5, 6). Overly structured data
can lead to loss of cognitive focus by clini-
cians, both during input and review. This can
cause clinicians to experience a loss of over-
view about the case at hand when they have to
attend to data contained in many different
fields, sometimes on different screens within
an interface (7, 8). Furthermore, the act of
writing or dictating in narrative form may be
integral to the cognitive processing of the
case [9]. Our finding of dissonance between
subjects’ preferences concerning report for-
mat and their actual performance reading
them for comprehension confirms that the
cognitive issues are complex. In our opinion,
tried and true methods of authoring and dis-
playing radiology reports should not be aban-
doned without considering the consequences.
Our follow-up session and the question-
naires completed by the subjects shed light
on reader preference for report format. They
all strongly and consistently preferred the
structured version to the free text. This pref-
erence for structured format was consistent
across all seven domains that we asked about
with modal values on the 10-point Likert
scale all being 10 (prefer structure). Also,
the corollary question about general report
organization resulted in 73% preferring a
“laboratory report” format over the alterna-
tives. The opinions of our subjects are en-
tirely consistent with other workers’ find-
ings with respect to physician preferences
about radiology reports. There is a large
body of published research detailing the
opinion of referring physicians regarding the
content and format of radiology reports
[10–15]. The terminology differs somewhat
but attributes consistently endorsed by con-
sumers (readers) of radiology reports in-
clude complete, itemized, and structured.
Another element that is commonly preferred
by referring clinicians is that the report
should contain a complete listing of perti-
nent negative findings. In aggregate, these
opinions seem to militate for a report orga-
810
Sistrom and Honeyman-Buck
nization and format like the “laboratory re-
port” option preferred by our subjects.
Selection of senior medical students to serve
as subjects proved to be quite successful. Inter-
est and enthusiasm were such that we were
able to double the sample size with little effort.
Even after the study had closed, numerous stu-
dents asked to participate. We found the exper-
imental paradigm to be very acceptable to sub-
jects and quite easy to administer. These
factors should allow us to easily extend and ex-
pand the experiments—using additional co-
horts of senior medical students—to address
limitations described below.
Perhaps the most important limitation in our
study has to do with generalizing our results
from senior medical students answering con-
tent-specific questions to practicing physicians
using radiology reports for clinical decision
making. We narrowed the focus of the research
to evaluate readability of the documents con-
taining radiology interpretations with respect
to the format alone. Medical students’ subse-
quent experiences during training and practice
certainly do lead to differences in many skills
and habits. However, we argue that the simple
ability to read a passage of text and compre-
hend its content is already well established by
the senior year of medical school.
A difficult design consideration was wheth-
er to test subjects with both the free text and
structured version of each case. This would
have provided even greater power to detect
the effect of format on outcomes by virtue of
having a directly paired comparison. We
think that our choice of a balanced block de-
sign, incomplete in the format factor, was
valid for two reasons. First, the planned and
achieved power of the chosen design allowed
us to detect differences between free text and
structure that were far less than what we con-
sidered to be practically relevant. Second,
having subjects see cases twice would have
introduced methodologically difficult prob-
lems with memory effects.
Another issue is that our subjects had no
time constraints or other pressures placed on
them during testing. We plan on adding fea-
tures to the experimental paradigm that will
stress a subject’s short-term memory of the
material. The structured versions of the reports
we used had phrasing and syntax identical to
that found in the original narrative versions. In
practice, the language and construction of in-
terpretative statements would likely be rather
different in structured reports. Readers may be
more (or less) able to rapidly comprehend
medical content presented in structured format
using “telegraphic” constructions such as
“LIVER: Negative.”
To address limitations described above and
extend the scope of our inferences, we plan on
at least three extensions to the experiment us-
ing the same cases, questions, and randomiza-
tion scheme. First, we will remove the button
on the question page that allows going back to
review the report. Subjects will know this and
that they must answer all 10 questions after
one reading. There will be no time limit for ei-
ther reading the report or answering the ques-
tions. Second, we will place a time constraint
on how long the report is visible before
switching to the question page. Again, sub-
jects will know that they cannot go back to re-
view the report while answering questions.
Third, we will enable either structured or free
text formats to be viewed at the discretion of
the subject while they go through a case. The
tracking code will record which version(s)
they look at and for how long. This will allow
us to determine if subjects develop and actu-
ally act on a preference for one format or the
other as they move through the cases.
Further research will involve psychometric
evaluation of the questions themselves. Once
the three additional experiments detailed
above have been completed, we will have a
large number (64) of answers to each of 120
different questions about radiology report con-
tent. This should allow us to use standard tech-
niques to assess item difficulty, reliability, var-
ious correlations, and discriminatory power.
These results will be interesting in their own
right by revealing what kinds of questions are
challenging for readers to answer. This might
guide radiologists in explicitly including
phraseology in their reports to address these
difficulties. Types of questions that exhibit
high levels of variance in the answers given or
are poorly correlated with subject’s overall
scores will also be of interest. Given this
knowledge about the types of questions that
are most reliable and discriminatory, we can
redesign the cases and questions to optimize
power to detect subtle differences in reader
performance. Such information about question
content may guide other researchers in their
own experiments about readability of medical
documents.
To our knowledge, this work is the first
experimental evaluation of radiology reports
whose primary outcomes are quantitative
measures of information transfer to readers
of the documents. Based on the results de-
scribed above, we assert that there is no dif-
ference in information transfer efficiency
AJR:185, September 2005
 Information Transfer: Free Text vs Structured Format

between free text (narrative style) report for-
mat and structured (itemized) reports having
the same content. Despite the fact that they
performed no better with the structured ver-
sions, our subjects clearly preferred it to the
free text format.
References
1. Merritt CRB. New president says emphasize signal,
delete noise from radiology reports. ARRS Memo:
Newsletter of the American Roentgen Ray Society
2004; 15(3):1–8
2. Sistrom CL, Langlotz CP. A framework for im-
proved radiology reporting. J Am Coll Radiol 2005;
2:159–167
3. Duncan DB. t tests and intervals for compari-
sons suggested by the data. Biometrics 1975;
31:339–359
4. Pearson K. Mathematical contributions to the the-
ory of evolution. III. regression, heredity and pan-
mixia. Phil Trans Royal Soc Ser A 1896; 187:253
5. Ash JS, Berg M, Coiera E. Some unintended con-
sequences of information technology in health care:
the nature of patient care information system-re-
lated errors. J Am Med Inform Assoc 2004;
11:104–112
6. Garrod S. How groups co-ordinate their concepts
and terminology: implications for medical infor-
matics. Methods Inf Med 1998; 37:471–476
7. Patel VL, Kaufman DR. Medical informatics and
the science of cognition. J Am Med Inform Assoc
1998; 5:493–502
8. Patel VL, Kushniruk AW. Understanding, navigat-
ing and communicating knowledge: issues and
challenges. Methods Inf Med 1998; 37:460–470
9. Berg M. Practices of reading and writing: the con-
stitutive role of the patient record in medical work.
Sociol Health Illness 1996; 18:499–524
10. Clinger NJ, Hunter TB, Hillman BJ. Radiology re-
porting: attitudes of referring physicians. Radiology
1988; 169:825–826
11. Gunderman RB, Ambrosius WT, Cohen M. Radi-
ology reporting in an academic children’s hospital:
what referring physicians think. Pediatr Radiol
2000; 30:307–314
12. Johnson AJ, Ying J, Swan JS, Willicam LS, Apple-
gate KE, Littenberg B. Improving the quality of ra-
diology reporting: a physician survey to define the
target. J Am Coll Radiol 2004; 1:497–505
13. Lafortune M, Breton G, Baudouin JL. The radio-
logical report: what is useful for the referring phy-
sician? Can Assoc Radiol J 1988; 39:140–143
14. McLoughlin RF, So CB, Gray RR, Brandt R. Ra-
diology reports: how much descriptive detail is
enough? AJR 1995; 165:803–806
15. Naik SS, Hanbidge A, Wilson SR. Radiology re-
ports: examining radiologist and clinician prefer-
ences regarding style and content. AJR 2001;
176:591–598

APPENDIX 1: Headings Used for Structured Format Reports

UNENHANCED CT OF THE HEAD ABDOMINAL ULTRASOUND ABDOMINAL COMPUTED

Comparison: Comparison: TOMOGRAPHY
Procedure: Procedure: Comparison:
Image Quality: Image Quality: Procedure:
Bones-Sinuses: Liver: Liver:
Extracranial Soft Tissues: Gallbladder: Gallbladder/Biliary:
Ventricles: Biliary: Pancreas:
Brain-Developmental: Pancreas: Spleen:
Brain-Age Related: Spleen: Kidneys/Ureters:
Brain-Postive Findings: Kidneys: Adrenals:
Brain-Negatives: Vascular: Aorta/Vessels:
Extra-Axial Spaces: Other Findings: Bowel/Appendix:
Vascular Structures: Impression: Fluid/Free Air:
Other Findings-Comments: Lymph Nodes:
Impression: Pelvic Organs:
Other Findings:
Impression:

Source: Department of Radiology, University of Florida, Gainesville, FL 32602.

AJR:185, September 2005 811

 Sistrom and Honeyman-Buck

APPENDIX 2: Free Format Report of an Abdominal CT Scan as Obtained from the Medical Record

A CT scan of the abdomen and pelvis was
performed on 3/3/02 without priors. Routine
contiguous images were obtained from the
upper abdomen to the proximal femurs with-
out the use of intravenous contrast, following
a renal stone protocol.
The visualized lung bases are clear and
show no evidence of pulmonary parenchymal
nodules or masses.
There are multiple bilateral renal stones.
The largest is present within the left renal
pelvis and measures 16 × 9 mm. There is a 6
mm stone at the right ureteropelvic junction
with mild-to-moderate proximal hydroneph-
rosis. The distal ureter is decompressed.
There is perinephric fat stranding adjacent to
the right kidney. No additional ureteral or
bladder stones are identified. The left kidney
is without hydronephrosis or significant
perinephric fat stranding. However, there is
an exophytic 1.5 cm hypodensity off the in-
ferior pole of the left kidney that likely rep-
resents a renal cortical cyst.
Unenhanced examination of the liver is
unremarkable without biliary dilatation. Un-
enhanced examination of the spleen,
adrenals, and pancreas is unremarkable. The
bowel is not opacified but otherwise is
unremarkable. There is a normal-appearing
appendix.
There is no mesenteric or retroperitoneal
lymphadenopathy. No pathologic osseous le-
sions are seen.
Impression:
1. Multiple bilateral renal calculi as de-
scribed above. There is a 6 mm at least par-
tially obstructing calculus at the right uretero-
pelvic junction that is causing mild-to-
moderate proximal hydronephrosis. Addi-
tionally, there is perinephric fat stranding ad-
jacent to the right kidney.
2. Cortically based hypodensity within the
inferior pole of the left kidney that likely rep-
resents a renal cortical cyst.

APPENDIX 3: Report from Appendix 2 Parsing into the Structured Template

COMPARISON: Without priors.

PROCEDURE: Routine contiguous images were obtained from the upper abdomen to the proximal femurs without the use of
intravenous contrast, following a renal stone protocol.
LIVER: Unremarkable.
GALLBLADDER/BILIARY: Without biliary dilatation.
PANCREAS: Unremarkable.
SPLEEN: Unremarkable.
KIDNEYS/URETERS: There are multiple bilateral renal stones. The largest is present within the left renal pelvis and measures 16 × 9
mm. There is a 6 mm stone at the right ureteropelvic junction with mild-to-moderate proximal hydronephrosis.
The distal ureter is decompressed. There is perinephric fat stranding adjacent to the right kidney. No additional
ureteral or bladder stones are identified. The left kidney is without hydronephrosis or significant perinephric fat
stranding. However, there is an exophytic 1.5 cm hypodensity off the inferior pole of the left kidney that likely
represents a renal cortical cyst.
ADRENALS: Unremarkable.
AORTA/VESSELS:
BOWEL/APPENDIX: The bowel is not opacified but otherwise is unremarkable. There is a normal-appearing appendix.
FLUID/FREE AIR:
LYMPH NODES: There is no mesenteric or retroperitoneal lymphadenopathy.
PELVIC ORGANS:
OTHER FINDINGS: The visualized lung bases are clear and show no evidence of pulmonary parenchymal nodules or masses. No
pathologic osseous lesions are seen.
COMMENTS:
IMPRESSION:
1. Multiple bilateral renal calculi as described above. There is a 6 mm at least partially obstructing calculus at the
right ureteropelvic junction that is causing mild-to-moderate proximal hydronephrosis. Additionally, there is
perinephric fat stranding adjacent to the right kidney.
2. Cortically based hypodensity within the inferior pole of the left kidney that likely represents a renal cortical cyst.

812 AJR:185, September 2005