How Much Effort is Necessary for Blind Users to Read
Web-based Mathematical Formulae? A comparison using
task models with different screen readers
Luiz Felipe da Paixão Silva, Otavio de Faria Oliveira,
Evelise Roman Corbalan Góis Freire, Rosana Maria Mendes, André Pimenta Freire
Universidade Federal de Lavras
Lavras, MG, Brazil CEP 37200-000
luizfelipe_p@hotmail.com, otavio.faoliveira@gmail.com, evelise.freire@dex.ufla.br,
rosanamendes@dex.ufla.br, apfreire@dcc.ufla.br
ABSTRACT
Providing access to mathematical content on websites for
blind users who use screen readers has been an important
challenge for accessibility in Human-Computer Interaction.
Further to limitations in reading content in languages such
as Brazilian Portuguese, there are many issues related to the
effort demanded from blind users to navigate within mathe-
matical formulae when solving problems to reach for specific
information. In this paper, we performed estimations of the
effort required by blind users to navigate within mathematical
formulae using task models with GOMS (Goals, Operators,
Methods, and Selection rules) and KLM (Keystroke-Level
Model). We analyzed task models for different mathematical
formulae using available navigation resources in the screen rea-
ders JAWS (Job Access With Speech), ChromeVox and NVDA
(NonVisual Desktop Access), considering the optimal paths
that could be used by experienced blind users. The results
showed that JAWS currently provides more detailed naviga-
tion resources that allow for faster and more precise navigation
than ChromeVox and NVDA. However, we observed that there
are important issues regarding the complexity of the interacti-
ons required to use those navigation resources. This could be
very challenging for blind users when performing mathemati-
cal problem solving tasks, which are cognitively-demanding
themselves. The investigation of new enhanced interaction
approaches for within-formulae navigation in screen readers
is very important to make mathematical learning and problem
solving on the Web easier for blind users.
ACM Classification Keywords
H.5.2. Information Interfaces and Presentation (e.g. HCI):
User Interfaces
Permission to make digital or hard copies of all or part of this work forpersonal or
classroom use is granted without fee provided that copies arenot made or distributed
for profit or commercial advantage and that copiesbear this notice and the full citation
on the first page. To copy otherwise,or republish, to post on servers or to redistribute
to lists, requires priorspecific permission and/or a fee.
IHC 2017, October 23–27, 2017, Joinville, Brazil
© 2017 Copyright 2017 SBC. ISBN 978-1-4503-6377-8/17/10. . . 15.00
DOI: https://doi.org/10.1145/3160504.3160549
Author Keywords
GOMS, keystroke-level model, screen reader, accessiblity,
Web-based mathematical content
INTRODUCTION
Enabling access to education for people with visual disabilities,
such as blind users, is paramount for their development and
engagement in society. The use of assistive technologies plays
a key role in ensuring those people can have autonomy, be
independent and have the same opportunities as non-disabled
people. However, there are many areas in which the lack of
accessibility to educational material can present difficulties to
people with visual disabilities.
Those difficulties have shown to be even more pronounced
in the case of mathematical problems, particularly those in-
volving algebraic notation. The use of symbols in algebraic
structures is very challenging to students in general. However,
when not equipped with appropriate tools, students with visual
disabilities can be in disadvantage in relation to their peers
in Mathematics classes. In cases in which students have alre-
ady developed a good understanding of the problem solving
strategies, the success in solving algebraic problems involves
being able to interpret the symbols and to correctly “navigate”
within the formulae. Although previous studies showed that
blind students did not have differences in terms of their cog-
nitive abilities to solve algebraic problems [9] [1], they can
be seriously disadvantaged if they are not able to read and
navigate within mathematical content in an accessible and
efficient manner.
In a broader perspective, Warren et al. [20] organize the al-
gebraic thinking fundamentally in two pillars: the ability to
generalize variables and to understand algebraic structures,
in terms of their functions, equations, etc. This organization
prioritizes the execution of problem-solving processes, with
an emphasis on the algebraic language and not necessarily
on the conceptualization of algebraic thinking [10]. As stu-
dents become aware of those structures, they can become able
to execute the different stages to find solutions to algebraic
problems, classified by Carry [5] in organization, collection
and isolation. At the stage of organization, students can have
a general view of algebraic symbols, which enables them to
organize which symbols can be operated algebraically. Des-
pite the limitations of this process-oriented structure in not
characterizing the acquisition of algebraic thinking, it provi-
des a good understanding of the cognitive processes involved
in tasks related to reading and understanding mathematical
formulae, such as those analyzed in this paper.
This solution structure for algebraic problems can be used for
sighted students by means of written language, and by blind
students by means of Braille or spoken text. However, in the
case of blind students, the reading of content in Braille can li-
mit their experience with mathematical content, because there
is limited possibilities to “navigate” between lines, besides the
need to memorize terms and partial results [9]. Thus, the use
of adequate computing resources, such as those with speech
synthesis, can contribute significantly to helping blind users
have access to algebraic content. However, if the navigation
within the formulae is not provided with appropriate means,
those can also limit the access to parts of content in a suitable
manner.
One of the main types of assistive technologies used by blind
people is the screen reader software. This software provides
features to synthesize voice and to collect content from the
screen to be read out for those users. However, in the context
of Mathematics, screen readers still have many limitations, as
they can sometimes convey to the listener wrong information
or ambiguous mathematical content. Further to this, providing
effective means to navigate within formulae in Websites is still
a challenge in mainstream general-use screen readers.
Despite the existence of more specialized stand-alone mathe-
matical reading software for blind users, such as Lambda
[8] and rules for mathematical reading in English such as
MathSpeak® [11], the incorporation of native features to read
mathematical content is very recent in screen readers available
in Brazil, such as commercial software JAWS1 , and free soft-
ware systems NVDA2 and ChromeVox3 . The use of screen
readers is very important for reading Web content, particularly
for cases in which students have to read material on virtual
learning environments, for example, in which stand-alone
software may not help as much as their main screen reader.
The first JAWS version to incorporate mathematical content
reading on Websites was version 16, released in 2014. Chro-
meVox with mathematical reading was also released in 2014,
whilst NVDA released a version integrating the external plugin
MathPlayer [15, 16] in 2015. Further to this, there is still very
limited language support in those screen readers. Although
those screen readers are able to read general-purpose text in
Brazilian Portuguese, to this date, none of them provides sup-
port for reading mathematical content in Portuguese.
Many of the existing free screen readers on the market do
not offer efficient intra-formulae keyboard-based navigation
for blind users. JAWS is one of the only screens which offer
intra-formulae reading features. However, the current existing
1 JAWS - Available at http://www.freedomscientific.com/
2 NVDA - NonVisual Desktop Access - Available at
http://www.nvaccess.com
3 ChromeVox - Available at http://www.chromevox.com
navigation features do not take into account specific semantics
of mathematical content, such as numerator and denominator
in a fraction or the parts of an integral.
Considering this context, there is a lack of investigation
into the effort required for blind users to use intra-formulae
keyboard-based navigation, to solve problems, or simply to
identify parts of formulae available on Web sites.
Since the navigation of mathematical formulae is a relatively
complex task, in this paper, we performed estimations of the
effort required by blind users to navigate within mathematical
formulae using task models with GOMS (Goals, Operators,
Methods, and Selection rules) and KLM (Keystroke-Level
Model). We analyzed task models for different mathematical
problems using available navigation resources in the screen re-
aders JAWS, ChromeVox and NVDA, considering the optimal
paths that could be used by experienced blind users.
This article is divided into six sessions. After the introduction,
we present the theoretical background section, with related
studies, as well as the definition of assistive technologies.
Following, the methods section contains information regarding
the tools and techniques used. In the next section, we present
the results obtained and discussion. Finally, the conclusion and
future works section shows the main implications for design
obtained from the analysis and future work derived from the
results obtained.
THEORETICAL BACKGROUND
Assistive Technology for Mathematical Content
Throughout the world there are millions of people with some
type of disability. Many of those need support to help with
learning to overcome difficulties in certain areas of knowledge.
Mathematics is a very important area that currently presents a
great challenge for the learning of people with some type of
disability, especially those with visual disabilities. Therefore,
it would be the duty of all educational institutions to take care
of planning and teaching techniques, especially for students
with visual disabilities. However, there are accessibility chal-
lenges related to the availability of resources in technologies
themselves.
In Brazil, the Committee for Technical Aids (Comitê de Ajudas
Técnicas) [6] defined Assistive Technology as follows:
Assistive Technology is an area of knowledge, interdisci-
plinary, which encompasses products, resources, metho-
dologies, strategies, practices and services that aim to
promote functionality related to the activity and partici-
pation of people disability, disability or reduced mobility,
with a view to their autonomy, independence, quality of
life and social inclusion
Assistive technologies have enabled disabled people, including
people with visual disabilities, to use smartphones, laptops,
desktop computers, by means of assistive technologies such
as screen readers, whose purpose is to read content available
on the screen using a text-to-speech (TTS) mechanism.
Screen readers are widely used by people with visual disa-
bilities, and can also be used to read mathematical content.
However, they still have many limitations to read such content,
particularly in the case of languages such as Brazilian Portu-
guese. People with visual disabilities also use other assistive
technologies, such as as calculators, mathematical notation,
equation-resolution tools etc.
For reading formulae on the Web, different screen readers and
tools support the reading of mathematical content. One of
the earliest solutions for reading mathematical content on the
Web was MathPlayer [15, 16], made available as a plugin to
Web browsers. ChromeVox also made mathematical reading
support available in a more recent version [17], as well as other
screen readers such as JAWS, Window Eyes, Supernova and
others, aside from stand-alond systems such as Lambda [8].
Task modelling and KLM
Task modeling is the step-by-step description of tasks that
must be fulfilled by a user in order to achieve some objective.
Task models can be used to create interfaces as well as to
analyze and evaluate the interactivity of applications [2].
Over the years, several approaches to task models have been
developed. One of the most widely used methods is GOMS,
defined by its authors as “a set of Goals, a set of Operators, a
set of Methods for achieving the goals, and a set of Selections
rules for choosing among competing methods for goals” [3],
and how to use operators, methods and selection rules.
For this study, we used a model called Keystroke-Level Model
or simply KLM [3] in conjunction with GOMS. KLM uses
only the pressed keys, mouse movements, and mouse keys as a
way of evaluating and analyzing a task. Unlike the traditional
GOMS model from which it was derived, KML is only a sim-
plified version, which has no goals, methods or even selection
rules.
Related Work
In the proposal developed by Stoeger et al. [18], they discussed
requirements to make navigation and reading of mathematical
content possible to blind students. They pointed out many
issues with the inaccessibility of digital content and how screen
readers and other technologies should be improved to help
them overcome those difficulties.
There are few reports in the literature regarding the use of
task models to evaluate the use of assistive technologies by
blind users on the Web. Following we describe some research
studies which either used task models for web navigation or
for the use of technologies by disabled users.
Schrepp and Fischer [14] approached the lack of accessibility
in pages that contain many interactive element. The authors
used a GOMS model to estimate the time that a user takes
to perform a task in applications whose keyboard and mouse
are often used. The authors concluded that by using average
values estimated by GOMS, it is possible to evaluate different
interfaces.
The goal of the paper developed by Edwards [7] was to verify
if it was possible to adapt the visual interfaces of a WIMP
(Windows, icons, menus and pointers) style application to an
auditory form that could be used by blind people. For that,
he decided to adapt a piece of WIMP software to create a
word processor called soundtrack. With an analysis of the
number of keystrokes and time on task, the author concluded
that the current state of development of assistive technologies
for blind people at the time had many challenges related to the
complexity of interaction.
Trewin et al. [19] addresses in their paper the need for a re-
liable prediction by means of keyboard-level modelling for
an efficient analysis of the accessibility of screen readers for
disabled users. For this, a tool was developed to allow the
modelling of screen reading approaches, besides exploring
action sequences and the interaction of less experienced users.
According to Schrepp [12], an efficient keyboard access to
Web sites is highly important for many groups of disabled
users. Several Web sites were evaluated to demonstrate the
main challenges encountered by visually-disabled users, who
just use keyboard navigation to access the content of Web sites.
For this, GOMS modelling was used to obtain quantitative
values of keyboard navigation performance. The authors iden-
tified the main design errors made by Web sites that impair
navigation via keyboard.
In another article, Schrepp [13] demonstrated how practical
the application of the GOMS model can be as a tool for ve-
rifying the usability of Web units for disabled users. It is
important to note that two main advantages of GOMS were
listed: first, it does not require user participation and thus it
can an inexpensive method of interface evaluation; second,
it does not require a running system or even a high-fidelity
prototype. However, it also presents certain disadvantages
inherent of applying modelling techniques, such as being res-
tricted to predicting problems, failing to answer how easy it is
to understand the content of a site by a blind user.
METHOD
Study design
In order to compare the estimation of effort when navigating
within Web-based formulae by blind users, we chose a case
with quadratic equations, and selected a problem-solving stra-
tegy to guide the task models according to well-established
rules and principles to solve algebra problems, as described
by Carry et al. [5].
The task models considered a best-case scenario in which
blind users would have full command of their screen readers
and would have well-developed abilities to solve quadratic
equations. This way, the models portrayed an estimation of
the minimum effort that would be required using the resources
available in the three screen readers analyzed. Although we
recognize that this does not represent what would be done
by all blind users, this model with a best-case scenario use
represents the complexity of the tasks in terms of the way the
resources available were implemented.
The models were created considering the commands for key-
board navigation within mathematical formulae available in
JAWS, ChromeVox and NVDA, which are the screen readers
that provide mathematical reading among the most widely
used screen readers by blind users in Brazil.
 Figure 1: Character navigation using ChromeVox
Sample of evaluated screen readers
For this study, three screen readers were used:
• ChromeVox version 53.0.2784.5: this screen reader is wi-
dely available via the Chrome browser, and is available
to multiple platforms. Further to this, it is free and open
source, which allows for future modifications to achieve
the purpose of the overarching project under which this
research was developed. ChromeVox allows users to edit
their command keys for any type of action, and by default
some commands are defined with the addition of other keys.
For the evaluation of the screen reader only some basic
commands were used, considering the default settings:
– TAB key - goes to the next focusable item;
– ChromeVox command - standard command key to start
commands, being the combination of the Shift + Alt
keys by default;
– ChromeVox command + N > M - Go to the next mathe-
matical expression found on the page;
– ChromeVox command + equal - increases the granu-
larity, i.e. causes the focus on the expression to be
changed by levels such as the characters, for example.
– ChromeVox key + down arrow - navigates forward or
to the next element
The main features provided by the software in question aim
to allow the possibility of finding mathematical content in
Web pages with specific keys and the possibility of increa-
sing the granularity, navigating through characters within
the mathematical formula. Figure 1 shows an example of
reading in a higher level of granularity. One limitation of
the screen reader is that it only works within the Chrome
browser envinronment, which means it cannot be used as a
general-purpose screen reader for all applications and the
operating system.
• The Non-visual Desktop Access (NVDA) 2016.4 was
used because it is a free screen reader, compatible with
the Windows operating system, and it also has support for
commonly used applications, including browsers, emails,
spreadsheets, text editors, and others. NVDA also has a
series of commands, and allows for powerful resources to
navigate in Web elements, such as navigating accross he-
adings, paragraphs, links, and others. However, for the
particular case of this analysis, only basic commands to
access mathematical elements were used. Mathematical
reading in NVDA is done by access to an external plugin,
MathPlayer, and NVDA has no control over the navigation
within formulae, which is done by the third-party software.
Figure 2: Tree and the levels where the reader reads the partial
expressions
In order to re-read a part of a formula, it is necessary to re-
turn to its beginning and read it all over again. Following are
some basic commands from NVDA used in this research:
– NVDA key- Insert of the numerical keypad, Exten-
ded Insert or the Caps Lock key can be used as your
modifier keys.
– NVDA + tab - focuses on the current item.
– Down arrow - moves to next element on page
– Up arrow - returns to previous element on page
Although NVDA is a powerful and widely used software, in
the case of mathematical content it does not provide support
or means for internal navigation.
• The Job Access With Speech (JAWS) version 17.0 was
chosen because it is currently the most popular screen reader
worldwide, and is compatible with the Windows operating
system, despite being a commercial software with high cost
to blind users. The screen reader presents relevant features
compared to its competitors regarding reading of mathema-
tical content on Websites. Like the other readers analyzed,
JAWS has a number of commands to explore different parts
of Web pages. However, in this paper, we focused on the
commands used to reach a formula within a page and on
the commands used to navigate within formulae, shown as
follows::
– Down arrow - focuses to the next page element;
– Enter key - open the mathematical formula viewer
when finding an expression;
– Right arrow - read the next character of the word where
the system focus is, also used to navigate within the
mathematical formulas viewer.
Among all readers, JAWS provide the most advanced sup-
port for mathematical content. The software allows users
to navigate within mathematical formulae, so that it trans-
forms a mathematical expression into a tree, as illustrated in
Figure 2. It allows users to navigate in levels, considering
the root to the tree as the entire formula, until its leaves, thus
making it possible to identify parts of expressions within a
mathematical formula.
Note that when JAWS’ Math Viewer is open, it already starts
reading the tree from level 1, where it is already possible
to navigate in pieces by pressing the left and right keys, as
displayed in Figure 3.

Figure 3: JAWS Math Formula Display Window
Task modelling
In order to analyze and evaluate the complexity and effort
demanded fom people with visual disabilities to identify parts
or even solve mathematical expressions, we used the two
predictive models GOMS and KLM.
GOMS was used to model users’ goals and actions that would
need to be taken to perform common tasks related to mathe-
matical quadratic equations. The proposed tasks were:
1. to verify if a quadratic equation has two identical real roots;
2. to assess whether the concavity of the parabola is upwards
or downwards;
3. to verify if the equation is complete.
For the first task, users should find the mathematical formula
in a Web page, and after that they should find the value of a,
b and c of the equation and then calculate the value of delta.
The terms a, b and c refer, respectively, to the multiplier term
of x2 , to the multiplier term of x and the independent term
(constant).
In the second task, users would have to find the formula on
the page and find the value of a in the expression.
In the third task, besides finding the mathematical formula on
the page, users should check if the expression had the terms b
and c.
We considered that after each keyboard operation, blind users
would have to perform a mental operation to assess whether
the content they had listened to was relevant to their task.
For the planning of the tasks, we defined the use of a quadra-
tic equation (x2 +4x+4=0) in written MathML (Mathematical
Markup Language) [4] 4 . The MathML code is presented in
Listing 1.
It is important to highlight that, in order to use predictive mo-
dels such as GOMS, the models will be limited to conditions
in which users already know what the tasks are, what they
have to do and also that they would have previous knowledge
about Mathematics, in this particular case.
Finally, the KLM model was used to estimate the time required
by users to complete the selected tasks. In this work, only the
4 taken from MathML examples page - for each task using the reference values presented by Card
http://www.freedomscientific.com/content/html/jawshq/MathML-
Samples.html
operators K (keystroke), M (mental operation) and W (waiting
for system response) were used. Considering that the models
were targeted at blind users, only keyboard operations would
be used as input methods to perform the defined tasks.
Listing 1: MathML code for quadratic equation x2 +4x+4=0
<math xmlns =’http :// www.w3.org /1998/ Math/MathML ’ display =’
block ’>
<mrow >
<mrow >
<msup >
<mi >x </mi >
<mn >2 </mn >
</msup >
<mo >+ </mo >
<mrow >
<mn >4 </mn >
<mo >& InvisibleTimes ;</mo >
<mi >x </mi >
</mrow >
<mo >+ </mo >
<mn >4 </mn >
</mrow >
<mo >= </mo >
<mn >0 </mn >
</mrow >
</math >
Data analysis
The tasks to be analyzed encompassed the steps to navigate
a Web page, find a quadratic equation and obtain information
from the equation by means of the terms analyzed. Given
the tasks and their actions, we used KLM to evaluate each
task according to the number of operations that needed to be
carried out.
Unfortunately, we could not find previous research with refe-
rence values for times taken by blind users to perform tasks
using screen readers. Were are fully aware that the time taken
by blind users can be considerably longer than for sighted
users. However, in order to have some means of comparing
the values numerically, we used the reference values for KLM
operations defined by Card et al. [2].
The execution time of a task was the sum of the time of all
the operators involved to complete the task. From the GOMS
model, we obtained the physical and mental actions that users
would need to accomplish the proposed objective. From that
estimation we calculated the sum of all the operators used,
being K (keystroke), M (mental operation) and W (wait for
system response). The W operator was used mainly for cases
in which users would wait until the screen reader finished
reading a word. We used the value of 1 second per word
as an average estimation. This is particularly important in
the consideration of such tasks in comparison to the interac-
tion by sighted users, as listening to spoken content takes a
considerable time for blind users.
Three models of a GOMS with time analyses using KLM
were used for each screen reader. By obtaining the total time
et al. [2], the time on each task was compared between the
screen readers used.
As previously mentioned, the values do not reflect the actual
time that blind users would take on tasks, but were important
to enable comparisons in terms of the proportion of time dif-
ference for the navigation tasks between the evaluated screen
readers.
RESULTS AND DISCUSSION
Task models
The tasks were modelled using the CMN-GOMS (Card, Moran
and Newell GOMS), with a structure that represents a pseudo-
code of the realized tasks. This was the original version of the
GOMS model proposed by Card, Moran and Newell [3]. Nine
task models were created, being one for each screen reader,
and each task devised in order to obtain information from a
quadratic equation.
In Listing 2, we show the GOMS model for Task 1 (to verify
if a quadratic equation has two identical real roots) using the
screen reader ChromeVox. The total number of operations on
Task 1 on ChromeVox was 41 K, 14 M and 32 W, being 41
keystrokes, 14 mental operations and waiting for reading 32
words.
Listing 3 shows the GOMS model for Task 1 for the JAWS
screen reader. The total number of operations on Task 1 on
JAWS was 12 keystrokes, 12 mental operations and waiting
for reading 32 words.
Listing 4 shows the GOMS model for Task 1 for the NVDA
screen reader. The total number of operations on Task 1 on
JAWS 12 keystrokes, 32 mental operations and waiting for
reading 50 words, considering all the repetitions necessary to
re-read the content due to the lack of in-formula navigation.
In Listing 5, we show the GOMS model for Task 2 (to as-
sess whether the concavity of the parabola is upwards or
downwards) using ChromeVox. The total number of ope-
rations on Task 2 on ChromeVox was 20 keystrokes, 7 mental
operations and waiting for reading 19 words.
Listing 6 shows the GOMS model for Task 2 for the JAWS
screen reader. The total number of operations on Task 2 on
JAWS was 12 keystrokes, 12 mental operations and waiting
for reading 26 words.
Listing 7 shows the GOMS model for Task 2 for the NVDA
screen reader. The total number of operations on Task 2 on
NVDA was 5 keystrokes, 13 mental operations and waiting
for reading 28 words.
In Listing 8, we show the GOMS model for Task 3 (to verify if
the equation is complete) using the screen reader ChromeVox.
The total number of operations on Task 3 on ChromeVox was
41 keystrokes, 14 mental operations and waiting for reading
32 words.
Listing 9 shows the GOMS model for Task 3 for JAWS. The
total number of operations on Task 3 on JAWS was 7 keystro-
kes, 11 mental operations and waiting for reading 32 words.
Listing 10 shows the GOMS model for Task 3 for the NVDA
screen reader. The total number of operations on Task 3 on
Tasks K M W Time
1 41 14 32 60.28s
2 20 7 19 33s
3 41 14 32 60.28s
Table 1: Estimated time to perform tasks on ChromeVox
Tasks K M W Time
1 12 12 32 49.76s
2 6 5 26 33.68s
3 12 11 32 48.56s
Table 2: Estimated time to perform tasks on JAWS
Tasks K M W Time
1 9 32 50 90.92s
2 5 13 28 45s
3 7 22 39 67.36s
Table 3: Estimated time to perform tasks on NVDA
NVDA was 7 keystrokes, 22 mental operations and waiting
for reading 39 words.
Time estimation comparisons between screen readers
We used the values for time estimates suggested by Card [2] to
obtain a quantitative number that could be used to compare the
performance of screen readers analyzed in this article. Thus,
the time for operator K (Keystroke) was assigned the value of
0.28 seconds for its execution. For the M (Mental) operator,
the value of 1.2 seconds was used, and for the operator W
(Wait) it was decided that the value of 1 second was used for
each word, since this time is very relative to user preferences
with their screen readers.
The time spent on each task was calculated given the following
formula Texecute = 0.28K + 1.2M + 1W , the total time of
each task on each screen reader can be found in Table 1, Table
2 and Table 3 in column Time. With this estimation, and ad-
ding up the times of each task for each reader we noticed that
ChromeVox, shown in Table 1 had a total of 153.56 seconds to
complete all three tasks, and JAWS, shown in Table 2 had 132
seconds and, finally, NVDA, shown in Table 3 with 203.28
seconds.
Discussion
As seen in the application of GOMS and KLM on the screen
readers evaluated, quantitative values were provided, in order
to perform a critical analysis of the proportion of effort and
time blind users would have to devote in the execution of tasks
of reading mathematical content in problem-solving tasks.
For this, we applied time values, widely used and advised
in the academic scope [2], for each operator used in the mo-
dels. This yielded metrics that can be used to compare the
performance of each screen reader evaluated. However, it is
worth noting that these values do not reflect the times spent
by people with visual disabilities, and that this value would
probably be much higher.
 Listing 2: GOMS model for Task 1 using ChromeVox
GOAL: VERIFY THAT THE EQUATION HAS TWO IDENTICAL REAL ROOTS
. .GOAL: FIND THE FORMULA OF THE EQUATION OF THE SECOND DEGREE IN THE WEB PAGE
. . . .GOAL: FIND THE EQUATION USING SPECIFIC COMMAND FOR FORMULAS ..
. . . . . .GOAL: SEARCH FOR THE NEXT FORMULA ON THE PAGE
. . . . . . . .PRESS KEY 1K TAB
. . . . . . . .PRESS SHIFT + ALT + N> M 4K + 1M + W (10)
. .GOAL: FIND VALUE OF A IN EQUATION
. . . .GOAL: FIND THE VALUE OF A USING GRANULARITY
. . . . . .PRESS SHIFT + ALT + EQUAL .. PRESS UNTIL CHARACTER LEVEL 12K + 4M + W (4)
. . . . . .PRESS SHIFT + ALT + DOWN ARROW UNTIL YOU FIND THE VALUE OF A] 3K + 2M + W (5)
. .GOAL: FIND THE VALUE OF B IN THE EQUATION
. . . .GOAL: FIND THE VALUE OF B USING GRANULARITY
. . . . . .PRESS SHIFT + ALT + DOWN ARROW UNTIL YOU FIND THE VALUE OF B] 12K + 4M + W (6)
. .GOAL: FIND THE VALUE OF C IN THE EQUATION
. . . .GOAL: FIND THE VALUE OF C USING GRANULARITY
. . . . . .PRESS SHIFT + ALT + DOWN ARROW UNTIL YOU FIND THE VALUE OF C] 9K + 3M + W (7)
. .GOAL: CALCULATE DELTA VALUE USING THE VALUES OF A, B AND C FOUND . 1M

TASK 1: 41K + 14M + W (32)

Listing 3: GOMS model for Task 1 using JAWS

GOAL: VERIFY THAT THE EQUATION HAS TWO IDENTICAL REAL ROOTS
. . . GOAL: FIND THE FORMULA OF THE EQUATION OF THE SECOND DEGREE IN THE WEB PAGE
. . . . . PRESS DOWN ARROW 1K + W (4) + 1M
. . . . . PRESS DOWN ARROW 1K + W (6) + 1M
. . . . . PRESS DOWN ARROW 1K + W (2) + 1M
. . . . . PRESS DOWN ARROW 1K + W (12) + 1M
. . . . . PRESS ENTER KEY 1K
. . . GOAL: FIND VALUE OF A IN EQUATION
. . . . . GOAL: FIND THE VALUE OF A USING GRANULARITY
. . . . . . . PRESS DOWN ARROW 1K + 1M + W (2)
. . . GOAL: FIND THE VALUE OF B IN THE EQUATION
. . . . . GOAL: FIND THE VALUE OF B USING GRANULARITY
. . . . . . . PRESS RIGHT ARROW 1K + 1M + W (1)
. . . . . . . PRESS RIGHT ARROW
. . . GOAL: FIND THE VALUE OF C IN THE EQUATION
. . . . . GOAL: FIND THE VALUE OF C USING GRANULARITY
. . . . . . . PRESS RIGHT ARROW 1K + 1M + W (1)
. . . . . . . PRESS RIGHT ARROW 1K + 1M + W (1)
. . . . . . . PRESS RIGHT ARROW 1K + 1M + W (1)
. . . GOAL: CALCULATE DELTA VALUE USING THE VALUES OF A, B AND C FOUND . 1M

TASK 1: 12K + 12M + W (32)

Listing 4: GOMS model for Task 1 using NVDA

GOAL: VERIFY THAT THE EQUATION HAS TWO IDENTICAL REAL ROOTS
. . . GOAL: FIND THE FORMULA OF THE EQUATION OF THE SECOND DEGREE IN THE WEB PAGE
. . . . . . GOAL: SEARCH FOR THE NEXT FORMULA ON THE PAGE
. . . . . . . PRESS DOWN ARROW 1K + W (6) + 2M
. . . . . . . PRESS DOWN ARROW 1K + W (2) + 1M
. . . . . . . PRESS DOWN ARROW 1K + W (9) + 1M
. . . GOAL: FIND VALUE OF A IN EQUATION
. . . . . . . PRESS UP ARROW KEY 1K + W (2)
. . . . . . . PRESS DOWN ARROW KEY 1K + 9M + W (9)
. . . GOAL: FIND THE VALUE OF B IN THE EQUATION
. . . . . . . PRESS UP ARROW KEY 1K + W (2)
. . . . . . . PRESS DOWN ARROW KEY 1K + 9M + W (9)
. . . GOAL: FIND THE VALUE OF C IN THE EQUATION
. . . . . . . PRESS UP ARROW KEY 1K + W (2)
. . . . . . . PRESS DOWN ARROW KEY 1K + 9M + W (9)
. . . GOAL: CALCULATE DELTA VALUE USING THE VALUES OF A, B AND C FOUND . 1M

TASK 1: 9K + 32M + W (50)

 Listing 5: GOMS model for Task 2 using ChromeVox
GOAL: DEFINE THE CONCAVITY OF THE PARABOLA WHETHER IT IS UP OR DOWN
. .GOAL: FIND THE FORMULA OF THE EQUATION OF THE SECOND DEGREE IN THE WEB PAGE
. . . .GOAL: FIND THE EQUATION USING SPECIFIC COMMAND FOR FORMULAS
. . . . . .PRESS KEY 1K TAB
. . . . . .PRESS SHIFT + ALT + N> M 4K + 1M + W (10)
. .GOAL: FIND VALUE OF A IN EQUATION
. . . .GOAL: FIND THE VALUE OF A USING GRANULARITY
. . . . . .PRESS SHIFT + ALT + EQUAL .. PRESS UNTIL CHARACTER LEVEL 12K + 4M + W (4)
. . . . . .PRESS SHIFT + ALT + DOWN ARROW UNTIL YOU FIND THE VALUE OF A] 3K + 2M + W (5)

TASK 2: 20K + 7M + W (19)

Listing 6: GOMS model for Task 2 using JAWS

GOAL: DEFINE THE CONCAVITY OF THE PARABOLA WHETHER IT IS UP OR DOWN
. . GOAL: FIND THE FORMULA OF THE EQUATION OF THE SECOND DEGREE IN THE WEB PAGE
. . . .PRESS DOWN ARROW 1K + W (4) + 1M
. . . .PRESS DOWN ARROW 1K + W (6) + 1M
. . . .PRESS DOWN ARROW 1K + W (2) + 1M
. . . .PRESS DOWN ARROW 1K + W (12) + 1M
. . . .PRESS ENTER KEY 1K
. . GOAL: FIND VALUE OF A IN EQUATION
. . . . GOAL: FIND THE VALUE OF A USING GRANULARITY
. . . . . . PRESS DOWN ARROW 1K + 1M + W (2)

TASK 2: 6K + 5M + W (26)

Listing 7: GOMS for Task 2 using NVDA

GOAL: DEFINE THE CONCAVITY OF THE PARABOLA WHETHER IT IS UP OR DOWN
. . . GOAL: FIND THE FORMULA OF THE EQUATION OF THE SECOND DEGREE IN THE WEB PAGE
. . . . . . GOAL: SEARCH FOR THE NEXT FORMULA ON THE PAGE
. . . . . . . PRESS DOWN ARROW 1K + W (6) + 2M
. . . . . . . PRESS DOWN ARROW 1K + W (2) + 1M
. . . . . . . PRESS DOWN ARROW 1K + W (9) + 1M
. . . GOAL: FIND VALUE OF A IN EQUATION
. . . . . . . PRESS UP ARROW KEY 1K + W (2)
. . . . . . . PRESS DOWN ARROW KEY 1K + 9M + W (9)

TASK 2: 5K + 13M + W (28)

Listing 8: GOMS model for Task 3 using ChromeVox

GOAL: DEFINE IF THE EQUATION IS COMPLETE
. .GOAL: FIND THE FORMULA OF THE EQUATION OF THE SECOND DEGREE IN THE WEB PAGE
. . . .GOAL: FIND THE EQUATION USING SPECIFIC COMMAND FOR FORMULAS
. . . . . .PRESS KEY 1K TAB
. . . . . .PRESS SHIFT + ALT + N> M 4K + 1M + W (10)
. .GOAL: FIND THE VALUE OF B IN THE EQUATION
. . . .GOAL: FIND THE VALUE OF B USING GRANULARITY
. . . . . .PRESS SHIFT + ALT + EQUAL .. PRESS UNTIL CHARACTER LEVEL 12K + 4M + W (4)
. . . . . .PRESS SHIFT + ALT + DOWN ARROW REPEAT UNTIL YOU FIND THE VALUE OF B] 15K + 6M + W (11)
. .GOAL: FIND THE VALUE OF C IN THE EQUATION
. . . .GOAL: FIND THE VALUE OF C USING GRANULARITY
. . . . . .PRESS SHIFT + ALT + DOWN ARROW UNTIL YOU FIND THE VALUE OF C] 9K + 3M + W (7)

TASK 3: 41K + 14M + W (32)

It can be observed that the JAWS reader obtained a better time navigation. In addition, its navigation can be modified by the
(132 seconds) in performing tasks than the other screen readers. user as more internal in the formula or more superficial. This
This is due, in addition to the greater capacity of navigation by makes users able to have access to individual elements of the
the formulae, to the smaller amount of keys necessary for its formula, which assists in the identification of some specific
 Listing 9: GOMS model for Task 3 using JAWS
GOAL: DEFINE IF THE EQUATION IS COMPLETE
. . GOAL: FIND THE FORMULA OF THE EQUATION OF THE SECOND DEGREE IN THE WEB PAGE
. . . . PRESS DOWN ARROW 1K + W (4) + 1M
. . . . PRESS DOWN ARROW 1K + W (6) + 1M
. . . . PRESS DOWN ARROW 1K + W (2) + 1M
. . . . PRESS DOWN ARROW 1K + W (12) + 1M
. . . . PRESS ENTER KEY 1K
. . GOAL: FIND THE VALUE OF B IN THE EQUATION
. . . . GOAL: FIND THE VALUE OF B USING GRANULARITY
. . . . . . PRESS DOWN ARROW 1K + 1M + W (2)
. . . . . . PRESS RIGHT ARROW 1K + 1M + W (1)
. . . . . . PRESS RIGHT ARROW 2K + 2M + W (2)
. . GOAL: FIND THE VALUE OF C IN THE EQUATION
. . . . GOAL: FIND THE VALUE OF C USING GRANULARITY
. . . . . . PRESS RIGHT ARROW 1K + 1M + W (1)
. . . . . . PRESS RIGHT ARROW 1K + 1M + W (1)
. . . . . . PRESS RIGHT ARROW 1K + 1M + W (1)

TASK 3: 12K + 11M + W (32)

Listing 10: GOMS for Task 3 using NVDA

GOAL: DEFINE IF THE EQUATION IS COMPLETE
. . . GOAL: FIND THE FORMULA OF THE EQUATION OF THE SECOND DEGREE IN THE WEB PAGE
. . . . GOAL: SEARCH FOR THE NEXT FORMULA ON THE PAGE
. . . . . . PRESS DOWN ARROW 1K + W (6) + 2M
. . . . . . PRESS DOWN ARROW 1K + W (2) + 1M
. . . . . . PRESS DOWN ARROW 1K + W (9) + 1M
. . . GOAL: FIND THE VALUE OF B IN THE EQUATION
. . . . . . PRESS UP ARROW KEY 1K + W (2)
. . . . . . PRESS DOWN ARROW KEY 1K + 9M + W (9)
. . . GOAL: FIND THE VALUE OF C IN THE EQUATION
. . . . . . PRESS UP ARROW KEY 1K + W (2)
. . . . . . PRESS DOWN ARROW KEY 1K + 9M + W (9)

TASK 3: 7K + 22M + W (39)

element and thus avoids the loss of time involved in having to
review the entire formula to search for a desired element.
As for the screen reader ChromeVox, it had an average time
of 153.56 seconds. Among the analyzed screen readers, its
time was close to that of the execution time of the tasks of
JAWS. This is due to the same internal formulae navigation
capability much like JAWS. It provides possibilities to navigate
mathematical formulae element by element, and thus having a
greater freedom of search of some specific element. However,
on average, three navigation keys are used, which increases
the amount of time to navigate. However, this time is still
relatively small, considering that the user does not need to
listen to the entire formula for to find a specific element. On a
negative note, the capabilities to change between levels is not
still as developed as in JAWS.
Compared to the other two readers analyzed, NVDA had the
worst time (203.28 seconds) in relation to the execution of
the tasks elaborated. This is due to the lack of resources for
internal navigation of mathematical formulae. Although a
small number of keys are required for navigation, if users
need to search for a specific element in the formula, they will
need to listen to the entire formula in order to find the desired
element. It may occur that the formula is very long and thus
users may become lost in understanding the formula, having
to listen to it several times.
As can be seen, there is a need for studies for specific forms of
navigation based on strategies of resolution and in a way that
gives users freedom of navigation. The results show that it is
very important to allow users to choose the levels of navigation
in the mathematical formula, thus allowing the identification of
an individual element or a sub-expression of the mathematical
formula. Even though more advanced screen readers such as
JAWS already implement some of these features, it is still very
difficult for blind users to understand what the different levels
of granularity are. Considering the possibilities of semantic
mark-up provided by MathML, it would be interesting for
blind users to have ways of navigating to specific elements,
such as parts of fractions, indices, and others, in a similar way
as they can navigate through paragraphs, items, figures and
form elements on the Web.
CONCLUSIONS AND FUTURE WORK
With the results obtained by this study, we noticed that there
are important research gaps to improve the navigation capabi-
lities of assistive technologies for mathematical content.
Even though the availability of reading of mathematical con-
tent in MathML in mainstream screen readers for Web brow-
sers is relatively recent, much work is needed to enhance the
navigation within formulae, particularly in the case of free
and open-source screen readers. Those difficulties can be very
serious as students may have to face further difficulties with
complex navigation within formulae as they have to use those
resources to learn and solve problems.
In order to advance on research using predictive models such
as GOMS and KLM by blind users, further studies should
be carried out to explore values for times for the operations
conducted.
As future work, we intend to involve blind users to help im-
plement prototypes of enhanced navigation strategies using
semantic elements present in MathML to help blind users re-
ach elements they need in a less abstract manner than only
increasing and decreasing the levels of granularity. Following
the implementation of these prototypes, evaluations with blind
users will be carried out, in the context of the overarching
research project in which this paper is inserted.
ACKNOWLEDGMENTS
The authors would like to thank CNPq (process 442431/2016-
3) for supporting the NavMatBR project and FAPEMIG for
the financial support.
REFERENCES
1. Emily C Bouck and Pei-Lin Weng. 2014. Reading math:
A comparison of reading and listening to algebraic
problems. Journal of Special Education Technology 29, 4
(2014), 1–13.
2. Stuart K. Card, Thomas P. Moran, and Allen Newell.
1980. The Keystroke-level Model for User Performance
Time with Interactive Systems. Commun. ACM 23, 7
(July 1980), 396–410.
3. Stuart K Card, Thomas P Moran, and Allen Newell. 1983.
The psychology of human-computer interaction. L.
Erlbaum Associates.
4. David Carlisle, Patrick Ion, and Robert Miner. 2014.
Mathematical Markup Language (MathML) Version 3.0
2nd Edition. (2014). Available online at
https://www.w3.org/TR/MathML/, last accessed
15/06/2017.
5. L. Ray Carry, Clayton Lewis, and John E. Bernard. 1979.
Psychology of Equation Solving: An Information
Processing Study. Final Technical Report. (1979), 130p.
ERIC. University of Texas at Austin.
6. Comitê de Ajudas Técnicas. 2009. Assistive Technology
(in Portuguese). Brasília: CORDE (2009).
7. Alistair D. N. Edwards. 1989. Modelling Blind Users’
Interactions with an Auditory Computer Interface.
International journal of man-Machine studies 30, 5
(1989), 575–589.
8. Alistair DN Edwards, Heather McCartney, and Flavio
Fogarolo. 2006. Lambda:: a multimodal approach to
making mathematics accessible to blind students. In
Proceedings of the 8th international ACM SIGACCESS
conference on Computers and accessibility. ACM, 48–54.
9. Silvia Fajardo Flores and Dominique Archambault. 2012.
Understanding algebraic manipulation: Analysis of the
actions of sighted and non-sighted students. In The
International Workshop on Digitization and E-Inclusion
in Mathematics and Science, Vol. 2012.
10. Dario Fiorentini, Maria Angela Miorin, and Antônio
Miguel. 1993. Contributions to rethinking about... the
basic algebric teaching (in Portuguese). Pro-posições 4, 1
(1993), 78–91.
11. Mick D Isaacson, Dave Schleppenbach, and Lyle Lloyd.
2014. Increasing STEM accessibility in students with
print disabilities through MathSpeak. Journal of Science
Education for Students with Disabilities 14, 1 (2014), 3.
12. Martin Schrepp. 2006. On the efficiency of keyboard
navigation in Web sites. Universal Access in the
Information Society 5, 2 (2006), 180–188.
13. Martin Schrepp. 2010. GOMS analysis as a tool to
investigate the usability of web units for disabled users.
Universal Access in the Information Society 9, 1 (2010),
77–86.
14. P Schrepp, M. & Fischer. 2007. GOMS models to
evaluate the efficiency of keyboard navigation in web
units. Eminds - International Journal of Human
Computer Interaction 1, 2 (2007), 33–46.
15. Neil Soiffer. 2005. MathPlayer: web-based math
accessibility. In Proceedings of the 7th international
ACM SIGACCESS conference on Computers and
accessibility. ACM, 204–205.
16. Neil Soiffer. 2009. A flexible design for accessible
spoken math. Universal Access in Human-Computer
Interaction. Applications and Services (2009), 130–139.
17. Volker Sorge, Charles Chen, TV Raman, and David
Tseng. 2014. Towards making mathematics a first class
citizen in general screen readers. In Proceedings of the
11th Web for All Conference. ACM, 40.
18. Bernhard Stoeger, Mario Batusic, Klaus Miesenberger,
and Philipp Haindl. 2006. Supporting blind students in
navigation and manipulation of mathematical expressions:
Basic requirements and strategies. Computers Helping
People with Special Needs (2006), 1235–1242.
19. Shari Trewin, Bonnie John, John T Richards, Calvin
Swart, Jonathan P Brezin, Rachel K E Bellamy, and
John C Thomas. 2010. Towards a tool for keystroke level
modeling of skilled screen reading. In ASSETS’10 -
Proceedings of the 12th International ACM SIGACCESS
Conference on Computers and Accessibility. 27–34.
20. Elizabeth Warren, Maria Trigueros, and Sonia Ursini.
2016. Research on the Learning and Teaching of Algebra.
In The Second Handbook of Research on the Psychology
of Mathematics Education. Springer, 73–108.