Instructional Science 27: 269–284, 1999.

269
c 1999 Kluwer Academic Publishers. Printed in the Netherlands.

Learning by exploration: Thinking aloud while

exploring an information system

HERRE VAN OOSTENDORP and SJAAK DE MUL

Department of Psychonomics, Utrecht University, Heidelberglaan 2, 3584 CS Utrecht,
The Netherlands; Phone: +31 30 2533390; Fax: +31 30 2534511;
E-mail: H.vanOostendorp@fsw.ruu.nl

Abstract. The aim of the current research is to examine the ability of people to learn a
computer system by exploration and to asses the efficacy of a user interface with properties
that are supposed to support exploration. The study described in this paper used the think-
aloud method to obtain detailed information about the goals of the user and their realization
during the initial learning phase. The focus here will be on discussing the role of thinking
aloud and reflection during performing complex cognitive activities. In one condition of the
experiment described here an interface was used with exploration-supportive properties. In
the other condition these properties were removed from the interface. Subjects (university
students) solved a number of (e-mail) tasks with these interfaces, without training, and had to
think aloud during the first half. After solving the tasks a knowledge test about the interface was
presented. The results on the three kinds of measures (think aloud measures, task performance
and performance on the knowledge test) broadly were in favor of exploration-supportive
interface. In the discussion attention is paid to the positive influence of thinking aloud, probably
occurring because subjects were encouraged to use the available information on screen in the
exploration-supportive condition. The consequences and potential disadvantages of display-
based, exploratory learning in relation to planning are also discussed.
Key words: display-based problem-solving, exploratory learning, think-aloud method

Introduction

With the use of computer systems by a growing, heterogeneous group of

people, the topic of learning how to use these systems becomes more and
more important. Users often go through a formal learning stage before actually
using a system to perform a task. However, in other situations this investment
will simply be too high. One way to tackle this problem is to make manuals
less extensive and more accessible, and to focus on the actual tasks users
have to perform. This ‘minimalistic’ approach is advocated by Carroll (1990).
Nevertheless, consulting a manual may still be too high an investment for
some applications, or may be infeasible. When a manual cannot be used, the
user has to find out the missing information about the functionality by mere
exploration.
270
Still, even when the situation does not necessarily require exploratory
learning, it may be preferable to learn by exploration. Advantages of this type
of learning over other forms of training are found when the exploration is
guided by requiring the users to solve given tasks instead of setting their own
tasks (Chamey, Reder and Kusbit, 1990; Kerr and Payne, 1994). However,
possible drawbacks of (unguided) exploration have also been reported, for
instance the lack of metaknowledge that is needed to decide what is to be
learned (Briggs, 1990), or difficulties with determining what sequence of
steps led to the desired result the user stumbled upon (Payne and Howes,
1992).
In the present study we focus on the situation where users immediately
try to solve tasks that are realistic to the users, which involves the guided
type of exploration that can be beneficial to the user (Carroll, Mack, Lewis,
Grischkowsky and Robertson, 1985). The research reported here addresses
the question of how people can learn to use computerized information systems
without training or manuals, exploiting the knowledge they already have and
their ability to learn by doing and exploration. At the same time we want the
system to be designed in such a way that it supports the user in the exploration
process as much as it can.
Thus, the relevant question is: what kind of support should a system offer
to be explorable? Visualization is supposed to be important. For that reason
we used a graphical user interface based on the principles of “direct manipu-
lation” (Shneiderman, 1983; Van Oostendorp and Walbeehm, 1995), such as
the “dragging” and “dropping” of objects on screen. The objects on screen
sometimes directly suggest an appropriate action (Gaver, 1991) or may func-
tion as an externalization of the rules that determine the current problem space
(Newell and Simon, 1972). Zhang and Norman (1994) recently showed that
externalizing the rules of a puzzle, instead of having them internalized in our
mind, can have a dramatic (positive) impact on problem difficulty, even if the
underlying formal structures are the same.
But there are more ways to support explorative behavior. We may conceive
working with a complex display-based system as a special kind of problem
solving process, that is, as navigating in a problem space, in which every
step that is taken continually results in a new display which reflects the new
problem state (Larkin, 1989). Zhang and Norman focused on the construction
of the problem space itself, while we will focus on navigating in this problem
space.
Within this framework several ways to support navigating in the problem
space – and thereby improving exploratory behavior – come to mind. These
ways can be found in the work of Lewis and Polson (1990) and Draper and
Barton (1993a, b). They can be expressed in the form of guidelines:
 271
, keep the number of possible operations small at any given moment;
, make these operations distinguishable;
, make clear what the consequences of an operation will be;
, make the effects of operations visible once they have been executed;
, show the last operation performed by the user;
, make operations easily undoable to make it safe to experiment.
These guidelines may support users in their decision what alternative opera-
tions (or actions) to choose when traveling through the problem space.
As a result of this framework our main research question was: Do these
exploration-supportive features indeed allow the user to explore the system
and result in better performance? To study learning by exploration, we built
an interface for an electronic mail application. It must be emphasized that
we did not intend to make a full functional and realistic application. It must
be seen as a research tool to study exploratory learning. This type of email-
application was chosen, because the primary task of sending, reading, copying
and throwing away letters, et cetera, is known to everyone and the main
difficulty would be in the use of the interface, and not the task itself. An object-
oriented electronic mail system was developed which largely adheres to the
“direct manipulation” principle (Shneiderman, 1983). Four types of objects
can be “dragged” on screen (see Figure 1): letters (e.g., the opened letter
‘Tas’), addresses (e.g., ‘Jan’), folders for letters (the suitcase ‘Verstuurd’) and
folders for addresses (e.g., ‘Koken’ (2), containing two addresses).
In addition, there are four corresponding templates from which a new
object can be dragged (see Figure 1 on the right). Most of the functions of the
application can be performed by “dropping” an object onto another object. For
instance, a letter can be sent to a person by dragging the icon that represents
the letter to the icon that represents the addressee, or it can be sent to a group
of persons by dragging it to the icon that represents that group. Objects are
copied by dragging them to the copier icon, et cetera. The tasks which the
subjects received consisted of applying sequences of 6 to 12 basic operations
(see Table 1 for two examples, and their corresponding correct sequence of
basic operations).
In the experimental, exploration-supportive interface most of the
exploration-supportive features that were mentioned above (cf. Lewis and
Polson, 1990; Draper and Barton, 1993a, b) were implemented (see Fig-
ure 1): when the user points to an object, a text at the bottom of the screen
briefly describes what kind of object it is; once an object is grabbed by press-
ing the mouse button, all objects on which the object cannot be dropped are
dimmed to limit visually the number of steps that can be taken; when the
user drags the object over another object, the text at the bottom of the screen
briefly describes what would happen if the user drops the object there, and
272

Figure 1. Screen shot of the interface.

Table 1. Examples of tasks and corresponding sequences of basic operations

 Make a letter folder titled ‘January’. Put the letter ‘Conclusion’, which you can find
in the letter folder ‘Read’, in this new letter folder. Throw away the folder ‘Read’.
Correct sequence of operations:
Create object (letter folder ‘January’); Open object (letter folder ‘January’); Edit
object (letter folder ‘January’); Open object (letter folder ‘Read’); Get object
(letter ‘Conclusion’) from folder (‘Read’); Put object (letter ‘Conclusion’) in
folder (letter folder ‘January’); Close object (letter folder ‘Read’); and Trash
object (letter folder ‘Read’).

 Write a letter to your ‘Bank’ about a mistake at your disadvantage. Copy that letter,
send one copy to your ‘Bank’, and store one copy in your folder ‘Finances’.
Correct sequence of operations:
Create object (letter); Open object (letter); Edit object (text and title of letter
‘Mistake’); Close object (letter ‘Mistake’); Copy object (letter ‘Mistake’); Send
letter (‘Mistake’ to person (‘Bank’); and Put object (letter ‘Mistake’) in folder
(‘Finances’).

most actions can be undone. The control version of the interface lacks these
exploration-supportive features.
 273

Figure 2. Task performance (mean percentual score per block).

In a previous study (for details see De Mul, Van Oostendorp and White,
1994) we examined the effect of the exploration-supportive interface by
comparing it to an interface that lacked these properties. In addition, we
studied the effect of a manual by adding a third condition in which subjects,
working with this ‘bare’ interface without exploration-support, could consult
a one-page user guide. The results of this experiment, however, showed no
significant differences in task performance between the three conditions. See
Figure 2 for a summarization of the main results.
In the previous experiment the subjects solved three blocks of 9 tasks. Sub-
jects who worked with the interface that lacked the exploration-supportive
properties performed no worse in terms of accuracy than the subjects who
used the other version of the interface. The one-page manual did not seem to
have an effect on performance: in the condition without the manual the per-
formance was approximately equal (for details see De Mul, Van Oostendorp
and White, 1994). Neither were there any significant differences in speed of
tasks correctly solved between the three conditions (not shown here).
Although in the previous experiment the performance on the tasks in the
different conditions was found to be approximately equal, we do not know
whether the process of exploration that led to this performance was equal as
274
well. In the study that will be presented here, a thinking aloud method was used
to obtain more detailed information on this process during the initial learning
phase. The protocols should give an indication of the ‘explorability’ of various
parts of the interface. For practical reasons, we left out one condition – the
user-guide condition. Thus, in the current study we compared the condition
with the exploration-supportive interface to the condition with the control
interface, that lacked these properties. Subjects had to think aloud during the
first half of the tasks. During the second half of the tasks they worked silently.
Afterwards they received a knowledge test consisting of items measuring
declarative and procedural knowledge, respectively (Anderson, 1983).
In the rest of this article we briefly describe the general characteristics of
the method and results of this experiment (for details see De Mul and Van
Oostendorp, 1996). We want to focus here on the method of thinking aloud
during performing complex activities, and we will discuss critically the role of
thinking aloud and reflection during display-based problem solving. In the last
part of the Discussion Section we will address the consequences and potential
disadvantages of display-based, exploratory learning in relation to planning.
Here we will argue that problem solving that is highly display-oriented and
display-supported may have its restrictions.

Method

Subjects

Sixteen subjects (university students) were randomly assigned to the two

conditions. They had little computer experience and no experience with a
graphical user interface.

Materials

The subjects solved 12 tasks (see Table I for some examples) and had to
think aloud during the first 6 tasks. A knowledge test about the interface
was also presented. This knowledge test contained two kinds of items. The
first category of questions (12 items) asked for specific details regarding
the appearance of the interface. For instance, the subjects were shown two
icons that differed in color only. One of these icons actually appeared in the
interface, and the subjects had to indicate which one they thought they had
seen before. The second category (24 items) concerned the actions required
to solve a particular task, and the order of these actions. These questions
were of the form: ‘You want to do X. Action Al comes before action A2.
True or False? These categories are supposed to measure ‘declarative’ and
‘procedural’ knowledge, respectively (Anderson, 1983).
 275
Equipment

The experimental, exploration-supportive interface and the control interface

as described above, were used.

Procedure

The subjects did not receive any training, nor did they have a manual at their
disposal. They were asked to solve a series of 12 tasks and to think aloud
while solving the first 6 tasks. The computerized knowledge test about the
interface was presented after completion of the tasks.
In the analysis of the think-aloud protocols, we adopted a method suggested
by Draper and Barton (1993). In our email-application thirteen elementary
actions or ‘categories’, such as ‘Copy object’ or ‘Open object’, were dis-
tinguished. In most cases these actions consisted of a simple drag-and-drop
operation. The protocols of all six tasks for each subject were analyzed by
first dividing them into units. Each unit, often, but not always, a sentence, in
the think-aloud protocols was judged in terms of these (13) categories, and it
was determined whether the action was successful (+) or not (–). See Table 2
for an example-fragment of this analysis. Notice that more than one action
can be assigned to one sentence, and also that one action can be assigned to
more than one sentence.
To obtain an impression of the reliability of judging the think-aloud units
Cohen’s kappa was determined for assigning the units to the 13 different types
of actions or functions. The intracoder reliability (‘stability’, with a 3 month
interval) was 0.996, while the intercoder reliability also appeared to be very
high, 0.992.
Next, the protocol units and their categorizations were split up into thirteen
separate groups according to the type of action, e.g., the category ‘Open
object’. Within each of these 13 sets the events for all six tasks were sorted
chronologically, with the event with the lowest number at the top, i.e., the
event occurring first. This resulted in one table for each of the thirteen action
categories.
From these categorizations of actions, and judged in terms of success or
failure, three kinds of figures have been calculated. First, for each type of
action the failures occurring before the first successful operation have been
counted; this gives an idea of the learnability of the system functions. It
provides an indication of how easy a function, such as ‘Open object’, can
be found or how successful it is in suggesting its meaning when tried out
by the user. The lower this figure, the easier this type of action can be
understood or found. Secondly, for each type of action the number of failures
after the first success has been computed. This provides an indication of
276
Table 2. Example of a think aloud protocol fragment corresponding to the second example-
task, division in units (indicated by / < number > /), and the categorization in terms of
success(+) or failure (–) (Translated from Dutch)

 Write a letter to your ‘Bank’ Subject is simply

about a mistake at your disadvantage. reading the task
Copy that letter, send one copy to your ‘Bank’,
and store one copy in your folder ‘Finances’. /0/
Thus making a letter with the title ‘mistake’, Summarizing task
sending it to the ‘bank’ and store one in ‘Finances’ /1/.
I have first to write a letter
<moves cursor to letter template>. /2/ Create letter +
I grab this letter <put empty page on eye-icon> /3/ Open letter +
: : : Title ‘mistake’, and that there has been made a

mistake by the bank, does it have to be a long letter?

: : : , /4/ Edit letter +
finished, ok. /5/ Close letter +
Copying, I don’t know how that’s working,
may be this way, why doesn’t move
<clicking several times on the copier>,: : : /6/ Copy letter –
maybe clicking twice, no, /7/ Copy letter –
maybe with the pointer, no, it doesn’t work: : : /8/ Copy letter –
I better remove this letter here,
this one called ‘mistake’, copying I don’t know. /9/ Copy letter –
I know how to store. /10/ Put letter in folder +
I really don’t know how to copy. /11/ Copy letter –
Maybe I have first to have the letter and
bring letter to it <letter to the copy-icon>,
yes, it works, /12/ Copy letter copy-icon +
and next a letter to the ‘bank’, yes,
sent to the ‘Bank’. Ok. /13/ Send letter to person +
And a letter to ‘Finance’ folder, Put letter in folder +
let’s see if the letter is in it
<folder on eye-icon>. Yes. /14/ Open folder on eye-icon +
Ready.

how consistent the knowledge about a function is. The lower this figure, the
higher the consistency. Finally, by dividing the number of successes by the
sum of successes and failures, a more general figure is obtained that reflects
the general success of a function. The minimum and maximum values are 0
and 1.
 277
Table 3. Means of the thinking aloud measures

Learnability Consistency General success

Action description Control Expl. Control Expl. Control Expl.
support support support

1 Create Object 1.3 0.4 1.1 0.3 0.6 0.9

2 Open Object 4.0 1.4 9.0 2.9 0.6 0.8
3 Edit Object 1.0 1.0 2.3 0.6 0.7 0.8
4 Close Object 0.0 0.1 2.8 0.0 0.9 1.0
5 Get Object from Folder 0.1 0.1 1.9 0.0 0.6 1.0
6 Put Object in Folder 1.5 0.0 0.2 0.1 0.6 0.9
7 Send Letter to Person 1.7 0.1 0.9 0.0 0.6 0.9
8 Send Letter to Group 0.4 0.1 0.4 0.0 0.8 0.9
9 Trash Object 0.3 0.1 0.0 0.1 1.0 1.0
10 Get Trash 0.0 0.0 0.0 1.0
11 Get from In-tray 0.0 0.7 2.0 0.0 0.5 0.9
12 Copy Object 0.0 0.3 1.1 0.3 0.9 0.9
13 Move Object 0.3 0.5 1.0 0.8 0.6 0.6

The pairs of figures that differ significantly (p < 0.05, Mann–Whitney test) are shown in bold.

Results

Analysis of the think-aloud protocols

In Table 3 the results of the thinking aloud measures are summarized. As

can be seen, all eleven significant differences are in favor of the interface
with the exploration-supportive features. Please note that for the first two
variables lower values mean a better exploration. A lower learnability score
means that the corresponding action can be understood or found earlier, and
a lower consistency score means that less failures are occurring after a first
successful visit. Further note that in some categories there are only a few
or even no observations, since not every category was equally needed for
solving the tasks. This explains the blanks (no observations) and the round
numbers (based on only few observations).
The learnability of a‘putting an object in a folder’ was significantly (p <
0.05) higher in the condition with the exploration-supportive interface, and for
a number of other commands there were trends in the expected direction (0.05
< p < 0.10). The consistency and general success were also significantly (p <
0.05) higher for a number of functions in the exploration-supportive condition.
Four commands showed a higher consistency, and the general success was
higher for six commands. For seven out of thirteen actions a significant (p
< 0.05) advantage was found for the exploration-supportive interface on any
278
of these measures. Most of the other commands showed trends (0.05 < p <
0.10) in the expected direction.

Task performance

We also analyzed how well the tasks were solved (Figure 3a). In the first
half of the tasks (while thinking aloud) there was no significant difference
between conditions. However, in the second half the exploration-supportive
condition solved significantly (p < 0.05) more tasks correctly. The subjects in
the exploration-supportive condition also took less time (p < 0.05) working
on these tasks (not shown here).

Knowledge about the interface

After completion of the tasks, subjects received the knowledge test. In the
exploration-supportive condition subjects scored significantly (p < 0.05)
higher on the procedural questions; the scores on the declarative questions
did not differ significantly (see Figure 3b).

Discussion and conclusions

The think-aloud method allowed us to examine the process of exploration in

the two conditions more closely. The data suggest that the use of exploration-
supporting facilities resulted in a more effective exploratory behavior. A
number of the system’s functions were found more quickly or remembered
for a longer time, and also the general success in using the functions was
higher in the exploration-supportive interface condition. While the previous
experiment (De Mul, Van Oostendorp and White, 1994) failed to show differ-
ences between the two versions of the interface concerning task performance
and knowledge of the interface, the current experiment did also reveal dif-
ferences with respect to task performance, and procedural knowledge. The
exploration-supportive condition showed a better score on the second block of
tasks (during which the subjects worked silently) and also on the procedural
questions of the knowledge test.
Since the only difference between this experiment and the previous one
was that we required subjects to think aloud during the first six tasks, the
differences between the two conditions that appeared in the current exper-
iment could be attributed to a side-effect of thinking aloud. It is of course
possible that the lack of difference between conditions in the previous exper-
iment is caused by a lack of power. This is, however, not very plausible. The
first experiment was carefully designed, and a lot of conditions were highly
favorable here (number of subjects, number of tasks, and so on).
 279

Figure 3. (a) Mean percentual scores for task performance (1st half thinking aloud; 2nd half
not thinking aloud), and (b) the knowledge test.

Not only was there a difference between conditions during the first half
of the tasks in the current study, but the experimental condition also outper-
formed the control condition in the second half (and on the knowledge test),
when the subjects were no longer thinking aloud. Apparently, there appears
to occur positive (learning) transfer from the first half to the second one.
Thinking aloud seems to force subjects to watch the screen more closely in
order to describe what they are doing, increasing the chance that they notice
and use the facilities that support the exploration of the system, or – by the
280
addition of text labels to the objects (one of the exploration-supportive facil-
ities) – to become better aware of the metaphor that is used. The procedural
knowledge about the interface was also higher in the exploration-supportive
condition. Verbalization in the exploration-supportive condition might have
led to a better explicit, procedural knowledge, because the interface used
in this condition mainly helped the users with determining their next step
by referring to task-like units in the support texts that were provided at the
bottom of the screen. The better procedural knowledge could in turn have
resulted in a better task performance.
It seems that thinking-aloud positively influenced the metacognition of our
subjects, in the sense of monitoring and controlling one’s own problem solv-
ing behavior. It is, however, worthwhile to note that this factor alone is not
a sufficient condition: also the subjects in the control condition verbalized
their thoughts and still we find a significant difference in task performance
and procedural knowledge. So, it seems that the combination of increased
attention paid to the environment (the interface) and use of the information
provided in the environment by the higher degree of externalization of infor-
mation (available in the exploration-supportive interface) is responsible for
the facilitation.
This result confirms notions that thinking aloud may indeed encourage users
to focus on one’s own behavior, leading to more reflections on that behavior
and to a higher investment in cognitive monitoring and control and, finally, to
an improvement of performance (Reither, 1977; Berry, 1983; Van Someren
and Elshout, 1985; for a review see Ericsson and Simon, 1984). The study by
Van Someren and Elshout (1985) showed, for instance, that learning by doing
was fostered by reflecting on one’s own problem solving process during task
performance. The tasks consisted of solving end situations in chess games.
Half of the subjects were instructed to verbalize after each move how they
came to their move. This group of subjects performed better than the non-
verbalization group on transfer problems. The acquisition of knowledge and
skills relevant to solving problems can apparently be enhanced by reflection.
In the context of human-computer interaction similar results are obtained
by Trudel and Payne (1995). They showed that the acquisition of knowledge
derived by the interaction with the system can be improved by encouraging
users to think about actions and their consequences. Reflection of the subjects
was manipulated by imposing a key-stroke limit on subjects, that is, a limit
on the amount of physical interaction with the device (a digital watch). The
number of allowed key-strokes was here limited. They found that despite
spending less total time on interaction with the device, subjects who had been
imposed a key-stroke limit learned better.
 281
Summarizing, it seems rather clear from these studies on the role of reflec-
tion that exploratory learning a complex system can be positively affected
by the degree to which subjects reflect on their behavior, and use available
supportive information. This means that thinking aloud can influence ‘nor-
mal’ processing. In this case it happened to be a positive influence, probably
because subjects were encouraged to use the available information on screen
in the exploration-supportive condition. The side-effect of thinking aloud also
means that, in general, ‘normal’ processing of complex cognitive problems
can be altered by thinking aloud, in particular if paying attention to visual
information in the environment plays an important role to problem solving
(Ericsson and Simon, 1984).
The main question of this study was whether the exploration-support indeed
supports exploration and leads to better task performance.
The effect on exploration itself is rather difficult to evaluate, because explo-
ration can be seen from two perspectives. If we consider exploration simply
as the ability to gather useful information in an unfamiliar situation, then
exploration is not necessarily linked to task performance, at least not directly.
Efficiency does not need to be the primary goal of someone who just started to
learn an interface. Information may be gathered that is not directly relevant for
the task at hand. On the other hand, if we see exploration as the ability to cope
successfully with a situation that is partly unknown to us, then exploration
is more or less linked to performance. The better the task performance, the
better the exploration must have been. The thinking aloud data showed that
a number of the system’s functions were understood or found more quickly
and remembered for a longer time in the exploration-supportive condition.
These data, reflecting primarily the process of exploration, support the notion
that better task performance is indeed caused by a more efficient explorative
behavior.
The question whether there is an effect of exploration-support on task
performance (and knowledge acquisition) is easier to answer. As mentioned
in the introduction, we conceive the function of exploration-support as being
formally equivalent to providing an external representation on the level of
navigating. The representation is continually updated and offers constraints
on how to proceed during problem solving. Other research has shown that
externalization can considerably enhance performance (Larkin, 1989; Zhang
and Norman, 1994). Also in the current experiment we find a positive effect
of externalization on the level of navigation (by the exploration-support) on
performance and knowledge acquisition, though it appeared to be dependent
on thinking aloud, which somehow forced subjects to process that what is
shown on screen more deeply. The fact that we did not find a significant
difference between the conditions with external and internal representations
282
in the previous experiment could be due to the small difference between the
‘external’ and ‘internal’ conditions, or by the fact that subjects did pay too
little attention to the external representation in the exploration-supportive
condition.
The findings discussed here may have significant practical consequences,
which are not restricted to the specific email application we studied: It might
apply to complex computerized information systems in general. Users can
learn these systems by themselves when they receive exploration-support, at
least if that support is sufficiently attracting the attention of users. This effect
might be achieved by making the support-features more prominent in order to
attract the attention of the user, for instance, by using more distinctive colors.
At this point we want to make two qualifications concerning exploratory
learning. First, in the introduction we approached exploratory learning and
exploration-support as an alternative to learning from a manual. Exploration
and exploration-support may, however, also be seen as complementary to,
rather than an alternative for, a manual of a system. The manual may, for
instance, provide support for the high-level planning of tasks, while the more
detailed operations are learned by exploration.
The second qualification concerning exploratory learning, especially in
the context of display-based problem solving, might be that the resulting
knowledge is relatively volatile. Draper and Barton (1993a, b) indeed found
that users tend to forget what they discovered, even within the same session.
Payne and Howes (1992) identify “exploration traps” into which learners
frequently fall: for instance, they may accomplish a goal but forget how they
did so, or they discover nonideal methods and use them thereafter.
There could be a potential negative influence and constraints of too strong
a reliance on visual displays with regard to planning of behavior and transfer
of skills. It is worthwhile to introduce the distinction here between situation
or display-based problem solving and plan-based problem solving (Vera and
Simon, 1993). The first is characterized by the execution of procedures trig-
gered by information in the external situation, in our context often perceptual
information on the display (or screen), while the second one is characterised
by using an elaborated internal representation to determine a future sequence
of actions.
O’Hara and Payne (in preparation) report a series of experiments which
manipulated the user interface to computerized versions of classic problems
like the Tower of Hanoi. When the cost of making a move in the problem
space is high, because the computer command is clumsy, subjects solve these
problems in significantly fewer moves, and acquire problem solving skills
more rapidly. The clumsiness of commands was manipulated here by varying
the number of keystrokes per command.
 283
O’Hara and Payne explain their findings by assuming that learners evaluate
the costs and benefits of planning, and plan more when they have more
to gain, in terms of saved efforts. These studies show that, within limits,
learners can decide how much to reflect on the problems they are solving,
and that the more they reflect, the more they will learn. In other words,
they demonstrated that increasing the cost of interaction at a user interface
can improve performance (less errors) and learning (less trials to achieve a
certain criterion). They also showed that high-cost training interfaces may
lead to better subsequent performance (transfer) with a low-cost interface.
They assume that when the mental cost associated with an operator was
relatively high, problem solving strategy shifted towards the plan-based end
of the continuum, resulting in less error-prone performance. Conversely, when
the cost associated with the operator was relatively low, subjects shifted to
the situation-based end of the continuum, to a strategy which was essentially
reactive and display-based. The implication of this work is that it may be
desirable to increase interface costs when the number of errors has to be
reduced, longer lasting, efficient plans have to be acquired, and transfer is
important. Under these circumstances it could be preferable to encourage
subjects to pay more attention to, and to think harder about the interaction
with the interface.

References

Anderson, J.R. (1983). The Architecture of Cognition. Cambridge, MA: Harvard University
Press.
Berry, D.C. (1983). Metacognitive experience and transfer of logical reasoning. Quarterly
Journal of Experimental Psychology 35A: 39–49.
Briggs, P. (1990). Do they know what they’re doing? An evaluation of word-processor user-
s’ implicit and explicit task-relevant knowledge, and its role in self-directed learning.
International Journal of Man-Machine Studies 32: 385–398.
Carroll, J.M. (1990). The Núrnberg Funnel: Designing Minimalist Instruction for Practical
Computer Skill. Cambridge, MA: The MIT Press.
Carroll, J., Mack, R., Lewis, C., Grischkowsky, N. and Robertson, S. (1985). Exploring a word
processor. Human-Computer Interaction 1: 283–307.
Charney, D., Reder, L. and Kusbit, G.W. (1990). Goal setting and procedure selection in acquir-
ing computer skills: A comparison of tutorials, problem solving and learner exploration.
Cognition and Instruction 7: 323–342.
De Mul, S., Van Oostendorp, H. and White, T. (1994). Learning user interfaces by exploration.
In R. Opperrnann, S. Bagnara and D. Benyon, eds, ECCE-7 Conference Proceedings.
September 5–8, Bonn.
De Mul, S. and Van Oostendorp, H. (1996). Learning user interfaces by exploration. Acta
Psychologica 91: 325–344.
Draper, S.W. and Barton, B. (1993a). Learning by exploration, and affordance bugs. In S.
Ashlund, K. Mullet, A. Henderson, E. Hollnagel and T. White, eds, Adjunct Proceedings
of INTERCHI’93, pp. 75–76. Amsterdam, Netherlands: ACM.
284
Draper, S.W. and Barton, B. (1993b). Detecting bugs with learning by exploration. Department
of Psychology. University of Glasgow, unpublished report.
Ericsson, K.A. and Simon, H.A. (1984). Protocol Analysis: Verbal Reports as Data. Cam-
bridge: MIT Press.
Gaver, W.W. (1991). Technology Affordances. In Proceedings of the CHI’91 Conference.
ACM Press.
Kerr, M.P. and Payne, S.J. (1994). Learning to use a spreadsheet by doing and watching.
Interacting with Computers 6: 3–22.
Larkin, J.H. (1989). Display-based problem solving. In D. Klahr and K. Kotovsky, eds.,
Complex Information Processing. Hillsdale, NJ: Erlbaum.
Lewis, C.H. and Polson, P.G. (1990). Theory-based design for easily learned interfaces. Special
Issue: Foundations of human-computer interaction. Human Computer Interaction 5: 191–
220.
Newell, A. and Simon, H.A. (1972). Human Problem Solving. Englewood Cliffs, NJ: Prentice-
Hall.
O’Hara, K.P. and Payne, S.J. (in preparation). Cost of operations affects planfullness of problem
solving. Department of Psychology, University of Glasgow.
Payne, S.J. and Howes, A. (1992). A task-action trace for exploratory learners. Behaviour &
Information Technology 11: 63–70.
Reither, F. (1977). Der Einfluss der Selbstreflexion auf die Strategie und Qualitát des Prob-
lemlösens. In H.K. Garten, ed., Diagnose von Lernprozessen. Braunschweig: Westermann.
Shneiderman, B. (1983). Direct manipulation: A step beyond programming languages. IEEE
Computer 16: 57–69.
Trudel, C.I. and Payne, S.J. (1995). Reflection and goal management in exploratory learning.
International Journal of Human-Computer Studies 42: 307–339.
Van Oostendorp, H. and Walbeehm, B.J. (1995). Towards modelling exploratory learning in
the context of direct manipulation interfaces. Interacting with Computers 7: 3–25.
Van Someren, M.W. and Elshout, J.J. (1985). Het effekt van zelfreflektie op leren probleemo-
plossen (The effect of self-reflection on learning to solve problems). In J.G.L.C. Lodewijks
and P.R.J. Simons, eds, Zelfstandig Leren (Independent Learning). Proceedings Educa-
tional Research Conference, 1985. Lisse, The Netherlands: Swets & Zeitlinger.
Vera, A.H. and Simon, H.A. (1993). Situated action: a symbolic interpretation. Cognitive
Science 17: 7–48.
Zhang, J. and Norman, D.A. (1994). Representations in distributed cognitive tasks. Cognitive
Science 18: 87–122.