NAVAIR Final Report

Recommendation Report
A compilation of recommendations for the NAVAIR Career

Guidebook mobile application, based on engineering
students’ user interface interactions.
Prepared for:
Adrienne Somerville, NAVAIR (adrienne.somerville@navy.mil)
Charles Irons, NAVAIR (charles.irons.ctr@navy.mil)
Dorothy Cavanagh, NAVAIR (dorothy.cavanagh@navy.mil)
Prepared by:
Xue Ding: xued@g.clemson.edu
Valerie Smith: vmsmith@g.clemson.edu
Ciara Marshall: ciara.m.marshall@gmail.com
Keith Sechrist: ksechri@g.clemson.edu
Supervised by:
Dr. Tharon Howard: tharon@clemson.edu
Confidential: NAVAIR
Introduction & Principle Goal
NAVAIR has indicated that the current NAVAIR Career Guidebook is about to be revised to
improve its performance and usability; however, in order to complete the revision of the phone
application successfully, its software designers and product managers need to understand the
goals and task performance needs of their users. Consequently, the primary purpose of this
study is to conduct a usability study of the NAVAIR Career Guidebook (CGB) mobile
application. Our group conducted the study with engineering students who may be interested in
applying for positions with NAVAIR.
Video evidence is referred to at the bottom of each recommendation within the

Recommendation section. The videos provide documentation for each of our recommendations
with use of actual participant statements about their experience with the NAVAIR mobile
application. The names of the videos will be the same names of the videos sent to you through
Clemson University’s File Drop service. All videos will either be .WMV or .MP4 formats.
Executive Summary
As per instructions of NAVAIR personnel Dorothy Cavanagh and Charles Irons during our
February 16th meeting, we focused our efforts in improving the usability of the NAVAIR Career
Guidebook application for users external to the organization wishing to learn more about what
positions were available for them to pursue. More specifically, our target demographic was
mechanical and computer engineering undergraduate students, who we learned are highly
sought after as new hires by NAVAIR upon completion of their degrees.
For our study, we conducted a think-aloud protocol in which participants performed specific
tasks while verbalizing their actions, thoughts, and impressions. The participants’ performance
and comments were coded, analyzed, and evaluated to create easily-interpreted videos that
highlight the participants’ interactions with the application. In this final report, we give our
actionable recommendations for changes that could help improve the usability of the application
for public audiences outside NAVAIR.
Our more detailed findings are discussed in later sections of this report. In general, however,
our findings indicate that participants spent a disproportionate amount of time navigating the
pathways to specific competencies and exploring the content of specific competency pages,
suggesting that they were experiencing difficulty. This observation was also confirmed verbally
by several participants as evidenced in the videos discussed in the recommendations section. It
was also shown that the bookmark, note-taking, and contact functions were relatively easy to
use, due to the short task duration times of these activities. The installation process was also
easy. The post-test questionnaire revealed that the amount of time participants spent on a
specific task correlated with their perceived difficulty of that task.
When participant comments were sorted into the categories of “positive”, “negative”, and
“recommendation”, it was found that 50% of all remarks were negative, 30% were positive, and
20% consisted of recommendations to improve the application. Under a variety of scenarios,
most participants stated that they would not use the application to research potential career
opportunities, which was the purpose of releasing the application publicly.
Based on our findings and participant feedback, we recommend the following to improve the
application’s use for the target demographic described. These are the main areas where we
made recommendations:
 Navigation
 Contact
 Content
 Bookmark
 Notes
 Aesthetic
For a detailed discussion, please see the Recommendation section below. The appendices
house all written materials generated for this study.
Methodology
Participants
Six participants were recruited for this study, with P1 participating in a pilot session and the
remaining five participants’ data being used for the final analysis. Comments made by all six
participants were used in generating recommendations; however, all feedback is considered
valuable.
None of the participants had experience with the CGB mobile application and only P1 recalled
knowing about NAVAIR as an organization before the study. Participants ranged from 21 to 25
years of age and were all male. All were pursuing studies in mechanical engineering with the
exception of P4, who was a computer science student. P1, P2, and P5 wished to pursue
careers in mechanical engineering, P3 wished to pursue manufacturing, P4 wished to pursue
computer engineering, and P6 wished to pursue architecture. P5 and P6 were the only
participants to use phones with Android operating systems, with the remainder of participants
having iPhones. The majority of participants counted more than twenty applications they had
downloaded for use on their personal mobile, with social media and sports-related being the
most common across all participants. The most recent smartphone technology adopter was P5
in 2015, and the earliest was P3 in 2009. Reported daily use of applications ranged in duration
from 30 minutes to over 2 hours. Below is a table summarizing demographic data and mobile
application experience across all participants.
Avg. Age 23.3
Gender 6 of 6 participants were male
Field of Study 5 mechanical engineering; 1 computer engineering
Career Plans 3 mechanical engineers; 1 software engineer; 1 manufacturing

engineer; 1 architect
Previous NAVAIR 0 of 6 participants

CGB Experience
Previous 1 of 6 participants
Knowledge of
NAVAIR
Operating 2 pf 6 participants used Android; 4 of 6 used iOS
Systems
Avg. Years of 5.8 years
Mobile App
Experience
Number of Mobile 1 of 6 participants use 5-10 apps; 1 of 6 participants use 10-15
Apps Used on apps; 4 of 6 participants use 20+ apps
Regular Basis
Most Common Music, sports, news, banking, social media
App Types Used
Avg. Daily App 2 of 6 participants used apps 30-60 minutes; 2 of 6 participants used
Use apps 1-2 hours; 2 of 6 participants used apps 2+ hours
Each participant performed the tasks (described below) individually on their personal cell
phones, on specific days and times arranged to their convenience. All study sessions were
conducted in Clemson University’s Usability Testing Facility. Morae Recorder and Morae
Manager software were used to record sessions and analyze results. All study sessions took
place between February 21 and March 28, 2017. Below are the primary four phases included in
this study:
Phase 1: Participant Recruitment & Pre-Test Questionnaire
According to Jakob Nielsen’s research on subject pool size, five participants can come up with
90% of the usability errors. Therefore, we decided to recruit 6 participants—the first of which
was used as a pilot session. As per instructions of NAVAIR personnel Dorothy Cavanagh and
Charles Irons during our February 16th meeting, we focused our recruitment efforts on
mechanical and computer engineering students likely to pursue careers with NAVAIR upon
graduation.
Administrators issued a call for participants during classes in which they knew the target
demographic would be present, providing contact information so that interested candidates
could reach out via email. A $5.00 incentive was advertised to encourage participation, as well
as an additional $5.00 for each successful referral they provided. After interested parties
contacted administrators, they were sent a pre-test questionnaire that collected a number of
data points to ensure they matched the engineering student demographic Charles and Dorothy
described. Once it was found they were a suitable match, a date and time convenient for them
to meet was set.
Phase 2: Participant Orientation
Upon arriving to their scheduled session, participants were asked to review and sign a consent
agreement (see Appendix A) in accordance with federal regulations regarding human test
subjects. Participants were made aware of accommodations available to them and asked to
complete the pre-test questionnaire if they had not done so previously. After this, participants
received a short, scripted briefing which they were instructed to follow along as it was read
aloud to them. This introduction explained the objective of examining the user-friendliness of the
NAVAIR CGB mobile application and assured participants they were not being scored on their
technical skills or knowledge.
Phase 3: Performance Test
As requested by NAVAIR personnel, participants were presented with a scenario in which they
had attended a career fair and had been advised by a NAVAIR representative to download the
CGB application to find positions that fit their career interests. The major tasks of this scenario
included:
1) Installing and opening the application.

2) Finding jobs available in their discipline.
a) Finding the requirements for these positions.
3) Bookmarking a position that piqued their interest.
4) Taking notes about a specific position.
5) Contacting NAVAIR through the application
Participants were prompted throughout the study to verbalize their actions, thoughts,
impressions, and problems encountered. For each task, participants were given roughly two
minutes to complete the item; at which point, administrators intervened to provide indirect
guidance. In some cases, the participants were asked to elaborate on problems so the exact
source of frustration or difficulty could be identified.
Phase 4: Post-Test Questions
After the performance test, each participant was asked to answer a series of post-test questions
regarding his/her subjective observations and opinions of the application. Participants were
asked to elaborate on their answers and explain why they chose each rating. They were also
asked to score the likelihood of using the application in the future under specific circumstances
and to provide insight as to how the application could be improved.
Video and audio of phases 2-4 were collected using Morae Recorder software and two Logi-
Tech 1000 cameras after participants had signed the corresponding consent form. One of the
cameras was angled to capture the expression and body language of the participant, while the
other was suspended on the desk above participants’ phones to capture their activity within the
mobile application.
Findings
Figure 1 Task Times
Figure 1 is a graph of the time it took the participants to complete each task in minutes. For
ease of coding, each task was represented by a letter. Please note that part of the task times is
partially influenced by the method of testing - Think-Aloud Protocol. When participants have to
speak as they function, it took them longer to complete tasks.
Within the goal of the participant finding a job they are interested in, we coded for specific tasks
that were completed while achieving this aim:
● C: Contact- user reaches contact us page and creates an email to send to
NAVAIR.
● B: Bookmark- user goes to a page, successfully bookmarks a page, and returns
to the Bookmarks tab to verify that their bookmark is there.
● N: Navigate- user goes to different competencies and their subpages searching
for job titles that suit their needs.
● E: Explore- user goes to a specific page and reads information about the job
position, starting with definition and going all the way through the recommended
training section.
● T: Note- user takes a note within a job position, and goes to the note section to
verify that the note exists.
If a user did not complete one of these tasks in their initial search, the post-test questionnaire
asked the users questions that engaged them to complete the tasks in the section “Rate the
difficulty of each task.”
As you can see from the graph, a large amount of the time was spent navigating to find jobs and
once within the jobs, exploring the information that was provided. The amount of time spent on
navigating to find a job that suits their needs is a problem because it shows that there was a
difficulty finding a job that met their specific desires. To reduce the amount of time spent on
navigation, we have a few suggestions that we will discuss in our recommendations. The
longest amount of time was spent exploring the content, which can be seen as good and bad
results. In a positive light, the content was valuable enough for users to spend time looking
through it to find the information they wanted. In the negative light, the participants had to spend
this length of time to get the general ideas of jobs and to determine if the jobs were something
of interest to them.
The task times also show that outside of navigate and explore, the other functions of the
application were relatively simple for the users to complete. The contact us page was directly
accessible from the homepage, so users did not have a problem finding the information, and
they did not have trouble creating an email. For most users, bookmarking the page was a
simple task, and taking a note was similarly as simple for users.
Comment Types by User:
There are 8 hours of videos collected for the performance test, and all the comments were
coded according to the types of comments the participant made. In the graph Distribution of
Marker Types by Participants, we can see percentage of different types of comments made by
participants 2-6. For this graph, we highlighted three main types of comments that were made:
positive comments (green), negative comments (yellow), and recommendations (red).
Figure 2 Distribution of Marker Types by Participant
The average number of comments that we received from users in our interviews was roughly
21.17 comments. As you can see from the data the average number of positive comments is
roughly 30%, the number of negative comments 50% and the number of recommendations
given by users was 20%. Throughout this document, we will use various comments they made
to point out the good, the bad, and other aspects of the application that can be changed to
increase the number of positive comments and to decrease the number of negatives.
Post-Test Questionnaire: Questions Rate the difficulty each task: 1-9 and 11-12
In this portion of the Post-Test Questionnaire (Appendix D) we asked questions about the
perceived difficulty of completing different functions within the application. As mentioned in task
times, if users did not complete one of these tasks, they were instructed to return to the
application and complete the task. They were also encouraged to do this if they were unsure of
how to rate the difficulty of a certain task.
Figure 3 Post-Test Average Difficulty
There were three main tasks that users found to be the most difficult:
3. Reach a specific page that you are interested in-2.67
8. Answering questions about the engineering career options- 2.67
9. Difficulty navigating, or finding where to go for information-2.5
The difficulty each participant perceived for these task directly relate to the amount of time that
users spent on the tasks of Navigate and Explore, though only one of the participants rated 3, 8,
and 9 as extremely difficult. In the questionnaire, we also found that participants in the study did
not have any problems downloading and installing the application. Overall, this was the easiest
task for the users to complete. Similar to the task times, the contact, bookmark, and note
functions were rated at minimal difficulty in both of terms of completing the action and verifying
that the action was completed. However, it should be noted that returning to the home page
from a specific page was rated as a quite difficult or higher by 4 out of the 6 participants. They
perceived this function as tedious and believe that it could be simplified by keeping the
navigation bar constant on all pages.
The next graph will show us the perceived use of the application for different scenarios found in
Post-Test Questions 11A-C and 12A-C. The questions 11A-C were meant to receive the
participants’ perceptions of the usefulness of the application as applied outside the task they
were given in terms of a much broader audience. The questions 12A-C were used to gain
insight into how the participants valued the application for their use. We assigned values to each
of the answers and graphed the results.
Users did not perceive themselves using the application any more than sometimes in all of the
situations. This perception of the application that users will only use it sometimes shows that
users did not believe that the application was very valuable. The highest score given to the
application in terms of use was only frequently. None of the user perceived that this application
would always be used. Given our task scenario was to assess how this application could be
used to recruit prospective employees at a job fair, the ratings in 11A, 4 out of 6 users believed
that the application would only be useful sometimes. In terms of the users saying they would
use the application to look up job opportunities (12A), half said they would frequently use it.
However, in terms of using the application as a training guide for new or current employees
(11B and 11C) and if the participants would use the application themselves to complete required
training or seek recommended training (12B and 12C), the majority of users stated they would
only use the application sometimes. Based on the recommendations of the participants and our
own recommendations, we believe that changes to make the application easier to navigate and
explore would greatly improve the perceived usefulness of this application.
Recommendations
Features to Retain
Amount of Information: The large amount of information provided in each competency area is
helpful to users in learning as much as they can about positions they’re interested in.
Several participants remarked on the wealth of information provided by the application and felt it helped
them learn all they could about positions they were interested in. Despite what other changes are
recommended, it is vital to the application's purpose to retain as much information from the Career
Guidebook as possible.
P1_explorepositives_lotsofinfo
P3_positive_content
Ease of Contact: Email option makes it very easy for users to send feedback.
Featuring the “Send Feedback” button on the application’s Homepage is a wise design choice. The
Homepage will have the highest traffic of all sections, so it would be very unlikely for users to miss it.
Users commented that they appreciated that the link was setup to create an email once they clicked on it.
This removes the steps of copying and pasting an email address into the appropriate fields, which can
prove difficult on a mobile device.
p3_post-test_contact-positive
p1_positive_contact
Contact Us
Sending Feedback: Provide a way to send feedback on every page by building a consistent
menu with the option to contact NAVAIR representations at all times. Provide a phone number
to make this easier.
Finding the “Send Feedback” button was simple. Sending feedback was simple. However, navigating to
send feedback from certain pages was not. A positive here is that Participant 1 found it very easy to
access NAVAIR representatives if he had questions. Participant 5 wanted more accessible ways of
contacting NAVAIR with the use of phone numbers. Participant 6 liked that the “Send Feedback” was
easy to find at the beginning of the application. However, he thought it wasn’t convenient for users since it
is just located at the beginning of the application, making it hard to access at different and deeper levels
of competencies. For example, once the participant found a job he liked, he couldn’t contact someone
with questions about that particular job. Other participants had problems with contacting NAVAIR
representatives because they wanted more than just an email built into the application itself.
P1_contact-positive
p5_Recommend-add phone number
p6_Intro-contact us
p6_Recommend-contact every page
Navigation
Search Function: Include a search field that would allow application users to search competency
content for a specific keyword.
Several participants reported feeling overwhelmed by the sheer amount of content housed within the
competencies and rated navigation as at least somewhat difficult. A frequent recommendation
participants made was the inclusion of a search field, preferably in the navigation bar, or at least
somewhere easily accessible such as the homepage. A search bar would help users find content they are
interested in much more quickly and make the content feel more accessible.
P1_navigation_searchbar
P4_navigation_search
P2_navigation_searchbar
Competency Numbering: Make numbering consistent and whole throughout the competencies,
excluded altogether, or some clear notation should be provided as to why certain sections are
not available in the application.
The current numbering of competencies skips entire sections. It was explained by Charles that this is
because the numbers are drawn from the Career Guide Book, and not all of the competencies in the CGB
were made available in the application. However, there is currently no explanation provided in the
application as to why the numbering system skips numbers and three of our six participants commented
that they were confused by these gaps. To remedy this, we suggest one of three possible options: 1)
Including all competency sections of the career guide book so that the numbering system appears whole,
2) Excluding the numbering system from the application altogether to prevent confusion, or 3) Providing
some sort of notation as to why certain competency areas are missing, such as listing the available
competencies, but greying them out and notating that they are “Currently Unavailable” in the application.
P5_navigation_numbers
P2_navigation_number
Navigation Bar: The navigation bar should follow the user consistently as they browse for easier
navigation through the competencies.
Navigation bars are a vital functionality when browsing content; they allow users to easily go forward,
backward, find the home page, and locate what they’re interested in. With the large amount of
competencies users can browse through in this application, a navigation bar that is consistent throughout
each pathway is a necessity for users to be able to orient themselves and develop a strong mental model
of where they are within the application. P1 and P6 mentioned the inconsistency of the navigation bar
detracted from functionality and aesthetics.
P6_navigation_navbar
P1_navigation_bar-not-consistent
General Navigation/Competency Organization: Competencies could be reorganized in a more

intuitive fashion that uses shorter pathways.
Several participants reported that they had difficulty orienting themselves through the application’s
navigation pathways to reach specific competencies. They also stated that they felt that the way
navigation pathways were organized was not intuitive. If at all possible, it is suggested that competencies
be reorganized with clearer titles and in an order that would shorten pathways, minimizing the time users
spend looking for the entries they’re interested in.
P4_navigation_difficult3
P4_navigation_difficult2
P4_navigation_difficult
P1_navigation_no-explanation
P3_post-test_navigation2
P2_navigation_difficult
Application Layout: Require fewer clicks to get to the information desired. Use accordion menus
for competencies and subsections on the competencies page.
Many of the users stated that there was a lot of information to go through and that it was hard to find the
information you were looking for because everything was on separate pages. This is an ease of use issue
for the user because it requires more effort on their part to click through various pages, and they have to
remember where all the information is. Also, lots of pages make it more difficult for the user to view other
information quickly for comparison. Therefore, our group suggests accordion menus to lessen the number
of pages. With accordion menus, each competency can have a drop-down list of the subsections and
each of those subsections could have a drop-down list within it of job titles. In doing so, the user will not
have to click between pages to view information, but rather they will be able to compare information on
one page.
Exploration and Content
Job Titles: Provide official job titles and the responsibilities of the jobs, accompanied with the
descriptions and skill sets already present. Place these in the same place the descriptions and
skill sets are already located.
The mobile application does a great job of illustrating the various types of jobs available at NAVAIR.
However, the app did not list the actual job titles with responsibilities and prior knowledge. Many of our
participants found this to be problematic.
P1_explore-confusingcontent
P1_explore-confusingcontent2
P4_content-confusingwords
P4_content-job titles not specific
P5_Content-vague
Audience: Make a separate program of the NAVAIR CBG available to the public eye for
potential employee recruitment.
The mobile application was designed to help serve people already hired and employed at NAVAIR.
However, our specific intended audience is for undergraduates in the engineering field of study who
attend a job fair. The scenario we were given is for the job fair to recruit potential students interested.
However, this is altogether unlikely to happen considering anyone outside of NAVAIR wouldn't
understand how to use this app.
p4_Content-confusing_
p5_PostTest-content not for job fair
General Information: Make the written content a little clearer and cohesive for new users with
the use of language aimed toward a broader audience.
Participants thought the content was a little too wordy to understand thoroughly. For example, the
Advanced Detail Knowledge under 4.5 - 11. SSS Database Support states, “Ability to demonstrate
comprehensive experience of the software tool through the completion of multiple deficiency and
enhancement requests and the generation and delivery of accurate performance estimates.” This may be
difficult for someone in college with no military experience to understand. Some participants also agreed
that the app was full of information, but needed more visuals to keep them interested. It is easier on the
eyes and allows users to continue reading for longer periods of time. It also keeps participants and users
from getting easily overwhelmed.
P5_Content-toomuch
P5_Content-appealing
P2_Introduction_badaesthetic
Hierarchy of Information: Create a tutorial video to show how the hierarchy of information within
the mobile application works. Consider inserting a little about NAVAIR as a company itself.
Separate videos should be created for iOS and Android users because the application is
different depending on the operating system. Place this somewhere obvious like on the Home
Page.
There is a lot of information placed into the NAVAIR CGB that can be very overwhelming to new users.
As mentioned previously, users who are not a part of NAVAIR frequently become lost within the
application. A short tutorial video that guides users through the use of the app would be very useful.
p5_Content-levelsconfusing
General Information: Positive- There was plenty of information under competencies.
All of our participants agree that there was a lot of information listed under each competency. There is a
wealth of information given throughout the application that provides users with a more grounded and
thorough understanding of the job positions. This is preferable to not having enough information.
P1_explorepositives-lotsofinfo
P1_explorepositives-lotsofinfo2
Acronyms: Define military-related acronyms, place them in a glossary, or build in a pop-up that
defines the military acronym.
The scenario was for the app to be shown to college undergraduate engineering students interested in
NAVAIR. However, these students do not understand military acronyms. Instead of changing this
completely around, provide a pop-up that defines all the military acronyms so that users can easily
understand what is trying to be said within the mobile application.
P5_Content-Acronyms
P5_Content-Acronyms don’t know if
P2_Acronyms
Bookmarks
Positive feedback: Keep the bookmark function.
All participants were able to navigate easily from the Bookmark page to a certain page and then go back
to the Bookmark page by clicking on the Back button. The participants feel there is no difficulty in using
the Bookmark function.
P1_bookmark-notdifficult
p4_Positive-bookmark
Confirmation of bookmark: Clarify the ribbon symbol’s meaning; need to add a confirmation
notice once a bookmark is created.
First, a user specifically pointed out that he did not know that the symbol at the top left corner
represents the “Bookmark” function in the first place and only had to try multiple times to learn that
function. Therefore, the users need instructions (perhaps an introductory video or image). Second, the
users also mentioned they need a notification sign to confirm once they’ve bookmarked/un-bookmarked a
certain page.
p5_Recommend-bookmark&notes
p6_bookmark confirmation
Accessibility: Add accesses other than the homepage for users to locate their bookmark more
quickly.
In the performance tests, our users realized that they can only access their bookmarks through the
Homepage. Participant 5 specifically mentioned that he wanted to know where to find the bookmarks &
notes while on a certain competency page instead of having to clicking on the “Back” button several times
to return to the Homepage and then get to the Bookmark page. Also, one user, Participant 6, mentioned
that he would like to be able to use bookmarks to compare different job positions so that he wanted the
bookmark function to be available throughout the application. Therefore, the users need access to the
Bookmark page when they are on a certain competency page. It is recommended that a link to the
bookmarks is added to the Navigation bar or embedded in a dropdown menu.
p6_PostTest-job positions hard to compare
p6_Recommend-bookmark function throughout
Notes
Symbol is not intuitive: Improve or change the “+” symbol to make users understand its function
is to create notes.
As was mentioned in the Bookmark function, our users found the meaning of confusing and not
very clearly related to the Notes function. In fact, both the Bookmark and Notes symbols at the top right
corner, , seemed unintuitive and the participants were unhappy because they felt they
had to learn those functions by themselves.
*Videos for this recommendation can be found under “No confirmation of notes” section.
No confirmation of notes: need confirmation notification when notes are successfully created
Similar issue to the need of confirmation for bookmarking a page, the users need a confirmation after
clicking on the Save button to know that they’ve successfully created a note. However, currently, clicking
on the Save button would lead the user directly back to the competency page without a notification.
Participant 1 complained that “I don’t know where the note went.” and he had to go all the way back to the
homepage to find the notes (at least 3 clicks on the Back button).
P1_bookmark_accessibility
p4_note confirmation
Editing is difficult: Improve the editing page of a customized note to be more user-friendly. A
dropdown menu is recommended.
Although all our participants reported that they found the Note-taking function helpful and not very difficult
to learn, some mentioned they’ve encountered severe problems while trying to create a customized note
directly from “My Notes” page by clicking on the button.
First, Participant 2 recommended a dropdown menu in the “Enter Competency” and “Enter Focus Area”
so that he didn’t have to remember the exact name of the competency or focus area.
Second, our participants also complained that they found it impossible to delete a note (or a bookmark)
within the Bookmark or My Notes pages. In fact, they had to click into the individual competency page in
order to un-bookmark or delete the notes.
Third, Participant 4 complained that when taking a customized note, there is no instruction about the
actual procedures. In fact, he ran into some trouble (i.e. he didn’t put in anything in “focus area”) and was
then notified that “There was an issue saving your notes” without being clarified about what caused the
error and how to fix it. So, he spent a while to figure it out by himself. Therefore, he recommended the
error message provide feedback as to how to remedy the issue.
p2 not recommendation drop down
p4_Notes-custom note issue
p5_Positive-Notes
Aesthetics
Image Resolution: Use higher resolution images for icons and department logos. Enlarge
photos on competencies page.
When looking at the application, many of the users complained
that the images were difficult to see and even more difficult to
read. As you can see in this photo, which is about the size of a
phone screen, the images are not large nor clear enough for the
user to read. Often their comments were that the images were
“blurry” or “impossible to read.” This is important for the home
screen of this application, where the images on the carousel are
links to the varying department competencies. Users
complained that they could not read what each department was,
so they would not click on these to navigate. Also, when on the
Competencies page, the icons were too small, so users were
still unable to read what they said. If users are unable to read or
properly view the icons, then they do not have a purpose for them on either page.
Within each competency, users also complained that the bookmark function and the note function were
low-resolution images as well. This is an aesthetic issue that users said made the application look like “It
needs a little work.” If you are going to launch an application, little design issues like this are displeasing
to users and they make a product look undesirable, but it is also an easy fix. Adding higher-resolution
images will look more professional, and users will be able to function with the application more easily.
p4_Aesthetic_blurry icons
p5_Aesthetic_Blurry Images
Content Layout: Break up text-heavy sections with visuals and colored headings.
Users will not read large chunks of text. After the first line or two, users scroll and look for headlines and
other markers of new topics. As you can see in the videos, none of the users read through the entire
section of text. They made various suggestions to engage the reader with more visual text. P2 was not a
fan of the white background and black text. As you can see in the video, he looked at the Contact Us
page for 15 seconds before making a complaint about the text. On the Android application, the text is
broken up to be more visually appealing. As you can see in the video below, P5_Content-Appealing,
there are colored bars separating each section of content. Though it is a long scroll for the user, the
information is broken up into readable sections and the color coding shows a separation of the
information into varying sections. These are the kinds of additions that make a page visually appealing.
However, we would like to note that in the second video, “P2 wordy content”, that these visuals do not
appear for iPhone users. In our study, over half of the users used an iPhone, so it is important that the
application be visually appeal for both Android and iPhone users.
P5_Content-Appealing
P2 wordy content
Images: Add images of people in each job position and environment to make the jobs more
visually interesting.
To make job positions more appealing and engaging to an audience, one idea that we came up with was
to add different visuals for each job position to make them stand out, and as a way to kind of show what
that person would do. We agree that this would be a great addition to the application. Whenever you are
discussing something like a job, it is important that people can visualize what a person in that position can
do or does. This association of jobs with people makes the job seem more real, and it will add a visual
variety to the application.
Functions: Add more functionality to the application than bookmark, note, and contact us. The
basic functions and design of the page give off the impression that the application is not
complete.
When asking the users about their overall impression of the application, many of the users had the
impression that the application was outdated in its color scheme, its functionality, and its overall layout.
P1 thought that a company that does avionics should have a better application. He said “It looks like they
did not spend very much time, or it appears like they don’t have much experience with technology.” As a
company that is very high-tech and has tons of resources, to give off this impression to a possible
employee is a bad first impression. When users view applications like these, they form impressions about
the companies based on their applications. If you are technologically advanced company, your application
should illustrate an ease of use and have modern functions. P1 also stated that this was not really a
mobile app and described it as a “mobile book” with the connotation that this was not what he wanted. P4
suggested that the application homepage could be made simpler because it is “way too much information
to throw at a user.” He also stated that the application looks like designers were “rushed” and he said that
the information looked like it was just put in the application for the user. What he is referring to is an ease
of use factor that is important to design. People do not like have to dig for information within an
application, and when the application is difficult to use, users form the impression that the company does
not recognize their needs. P6 described the application as looking “New, and I could tell it hasn’t been
worked on too much” because it has only the basic functions. P6 suggested more functionality by adding
an ability to apply to each page, more interactive checklist of competencies, and a search engine to
improve this.
P1_Post-Test_Impression
P1_Post-Test_bad app
P4_PostTest_opinion of app
P6_Aesthetic_BareBones
User-Oriented Information: Present the information with the user in mind. The user is coming to
this application to look for jobs, so show them jobs, not competencies to get jobs.
We were given the task scenario that the application would be used as a recruitment tool by potential
employees at a job fair, but we found that users did not think this application provided information about
jobs for them. Students come in with the mindset that they need to find a job, so when they look at an
application that says “Career Guide Book” they expect to find specific jobs that they can obtain, but this
application shows the competencies to get jobs without giving much detail about available jobs and
positions. Therefore, there is a schism where the intent of application does not meet the users’
expectations, so the users say they will not use the application. P1 said that they would never use the
application to look up job opportunities because the content did not list jobs, but rather it only had
competencies. P4 said they would never use the application to look up job opportunities because he
thinks the computer version would be easier to use. P4 stated that they did not think this application was
useful for recruiting new employees. P6 stated that the content seemed like it was “informally related to a
job,” and “that it seems like it was a name just thrown onto a list.” For P6, the content did not match jobs,
so they found the application undesirable.
P1_Post-Test_Job Titles
P4_Post-Test_Never App
P4_Post-Test-not useful
p6_Recommendation-Organization
Interactive Function: Add an interactive function to allow users to check off training as they
progress towards a career.
P1 stated that they would sometimes use the application because once they found the job they desired,
they would be able to see the training required for that position. However, they also said that they were
unsure if they would constantly look back to it as progressed. P4 stated that the application was not
preferable for looking up required training, but they would use it if this was their only option. In contrast,
P6, came up with the idea that a checklist for the training section would keep them more engaged with
the application, and they thought it would be able to keep track of their progression towards their ideal
career. We agree that a way for users to keep track of their training would keep users more engaged with
the application and would be a way to encourage continual use of the application.
P1_Post-Test-Job
p4_Post-Test_Never App.
P6_content_interactivity
Conclusion
We appreciate the opportunity to conduct usability testing on the NAVAIR CGB mobile
application. Our Engineering Student Group helps NAVAIR’s software designers and product
managers understand the goals and task performance needs of their users within the CGB
mobile application. Please contact us with any further questions by sending an email to Valerie
Smith (vmsmith@g.clemson.edu), Doris Ding (xued@g.clemson.edu), Ciara Marshall
(ciara.m.marshall@gmail.com), and/or Keith Sechrist (ksechri@g.clemson.edu). We sincerely
appreciate the opportunity to have been involved in a project of this scale.
Appendix
Appendix A: Consent Agreement (for distribution to participants)

Students of ENGL 8340: Usability Testing Methods, under the supervision of Dr. Tharon
Howard (tharon@clemson.edu), request your participation in a usability study of NAVAIR’s
Career Guide Book mobile application. The purpose of this study is to assess the user-
friendliness of the application for engineering students wishing to learn more about available
career options with NAVAIR.
All sessions for the study will take place in room 410 of Daniel Hall on Clemson University’s
main campus. Administrators will arrange dates and times convenient for all interested
participants. You can expect for your session to last roughly two hours, during which time you
will be seated as a desk. You can take breaks as needed.
Information about your experience and interaction with the mobile application will be collected
with pre- and posttest questionnaires, administrator observations, and audio and video
recordings. This data will be shared with NAVAIR personnel as well as other ENGL 8340
students. Under this agreement, administrators own all data collected. By signing this form, you
give your consent for the use and reproduction of your voice, verbal statements, and video
footage for the purpose of evaluating and presenting the results of testing this application. Your
name and other personal information will not be disclosed to any parties, with the exception of
the administrators.
There are no significant risks involved, your comments about the application will not affect your
eligibility for future employment for NAVAIR should you wish to pursue a position with them.
Neither Clemson University nor the researchers can be held liable for any injury you may
receive as a result of your participation in this study.
You may withdraw from this study at any time. Please inform the administrator at once if you
wish to do so. Please feel free to ask any clarifying questions at any time.
If you agree with the terms above, please indicate your acceptance by signing below.
Date: ____________ Signature: ___________________________________________
Initials: __________ Name (printed): _______________________________________
Date: ____________ Witness Signature: ____________________________________
Date: ____________ Test Administrator: ____________________________________
Once participants have signed the consent agreement, collect it immediately and begin both
audio and video recording before administering the pretest questionnaire. Record any answers
that the subject provides and note all observations of the subject’s responses and reactions.
Appendix B: Pre-Test Questions for NAVAIR CGB Application Usability Study
Personal background information

1. Age:__________
2. Circle one: Male Female
3. Department & Area of Specialization:______________________________________
4. What kind of positions are you interested in applying for after graduation?
________________________________________
Technology Experience
5. Do you have any experience using NAVAIR’s Career Guide Book mobile application?
Yes No
6. Are you familiar with NAVAIR? (if they aren’t, then you should tell them about the
organization)
7. What kind of cell phone operating system do you use?

Apple iOS Android Other:_________
8. How long have you been using a cell phone that uses applications? __________
9. How many applications do you have on your phone that are not preinstalled?
0-5 5-10 10-15 15-20 20+
10. What are the main applications that you use on your cell phone?
_______________________________________________________
11. Approximately, how much time do you spend on your cell phone every day?
0-30 minutes 30 minutes-1hour 1-2 hours 2+hours other: _____
Appendix C: Test Script
First of all, thank you for participating in our research today. Please make yourself comfortable
and ask let us know if you have any questions before the study begins. Please note that we are
not testing you. There are no “right” answers. Instead, we are testing the NAVAIR Career
Guidebook mobile application.
The NAVAIR CGB is an mobile application for prospective recruits and current employees of
NAVAIR. It was created by NAVAIR, the U.S. Navy Naval Air Systems Command, as a means
of prospective employee recruitments, new employee training, and current employee assistance
that can be used as a mobile application during convenient times for its users. Users can locate
and read information publicly accessible by downloading the mobile application for free. Users
can also place bookmarks or leave notes on selected pages, and then navigate to the selected
pages from the application’s homepage. That is what we are concerned with today, studying
the CGB as a means of recruiting, training and assisting new or current employees.
In order to understand how you feel about the CGB, as well as what you are thinking and doing
while completing this study, it is important that you feel free to do and say what you would
normally. All that we ask is that you read aloud and share your thoughts aloud. We may
occasionally ask you questions about what you are doing and thinking, but our main objective is
to observe the ways that you complete the tasks.
If at any time during the study you feel uncomfortable, need to use the restroom or just need to
leave, you should feel free to do so. Also, you may ask any questions that you wish. Please be
aware, however, that there might be some questions that we cannot answer during the study
because our answers might bias the way you would normally use the application. However,
please ask questions because these allow us to understand what you are experiencing. We will
gladly answer any questions that will not bias the study’s results, and we will try to answer all
your questions at the end of the study.
Scenario
For the purpose of this study, imagine you went to a career fair and met with someone from the
Navy who gives you a business card with information about the NAVAIR CGB mobile
application on it. You decided to install the application for the purpose of searching for the types
of opportunities and positions available for your skill sets.
In other words, please use the application to learn about career opportunities of interest to you
and to learn about what skills you may need in order to pursue those careers.
Please remember that we are not testing you or your performance level. We are using
your performance to rate the usability of the NAVAIR CGB.
Appendix D: Post-Test Questions for NAVAIR CGB Application Usability Study
1.When on a page, how would you do the following?
a. Can you find a job description here?
b. What knowledge is expected at the senior level of this position?
c. What level of education is necessary for each level of the position?
i. Where would you go to find the education requirements?
2. So once on this page, how would you find it later?
3. Suppose you had a question about this page. Is there a way to take notes while on this page?
4. Now look into the opportunities under comptroller. Save any of these pages that you are
interested in?
5.Look at your saved pages and return to one of those pages.
a. While in that page, take a note and look at other notes.
b. Look at other bookmarks
6.How would you find the meaning of unfamiliar acronyms?
7. Find what jobs are available in your discipline?
a. What are the requirements for these positions?
Rate the Difficulty of Each Task
1. Downloading and opening the mobile application.

Rate difficulty:
No difficulty a little difficult quite difficult extremely difficult N/A
2. Send feedback/ create an email to contact NAVAIR.

Rate difficulty:
3. Reach a specific page that you are interested in.

Rate difficulty:
4. Bookmarking a page.
Rate Difficulty:
5. Taking a note.
Rate Difficulty:
6. Return to home page from specific page.

Rate Difficulty:
7. Looking at your bookmarks and returning to one of the pages.
Rate Difficulty:
8. Answering questions about the engineering career options

Rate Difficulty:
Folder locations and navigation
9. Did you have any difficulty navigating, or finding where to go for information?
Yes/No
Rate how difficult:
10. What could be done to make this task less difficult?

Comments: _______________________________________________________
11. How useful do you think the NAVAIR CGB is to:

A .Recruiting New Employees
Never Sometimes Frequently Always
B. New Employee Training
C. Current Employees
12. Would you use the NAVAIR CGB to:

A. Lookup job opportunities
B. Complete Required Training
C. Seek Recommended Training
13. What is your general impression about the NAVAIR CGB mobile application?
14. What is your general impression about the producers and designers of the NAVAIR
CGB mobile application?
Appendix E: References and Guides for Building Mobile Applications
1. Mobile User Experience Guidelines and Recommendations:

https://www.digitalgov.gov/resources/mobile-user-experience-guidelines-and-
recommendations/
This website is a list of six guidelines to help assist in developing and designing a user-
friendly mobile application. It consists of many “dos” and “don’ts” that go into building a
user-friendly interface, allowing the audience to navigate through the mobile application
with ease.
2. Best Practices for Mobile Application Developers:

https://fpf.org/wp-content/uploads/Best-Practices-for-Mobile-App-Developers_Final.pdf
This PDF is great for working with privacy policies, especially since NAVAIR needs to
privatize as much as possible. It contains seven guidelines: Practice Privacy by Design,
Communicate Openly and Effectively, Make Your Privacy Policy Easily Accessible, Use
Enhanced Notice, Provide Users with Choices and Control, Secure Your User’s Data,
and Ensure Accountability.
3. Mobile app UX design: What strategies work and why

https://thenextweb.com/dd/2016/03/24/mobile-app-ux-design-strategies-
work/#.tnw_AWFpvTcA
An article that provides factors to make your mobile application successful, appealing,
and functional. This article dives into the details of User Interface Design, Mobile UX,
and Gesturization (swiping, pinching of a mobile application).

NAVAIR Final Report

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

NAVAIR Final Report

Uploaded by

Copyright:

Available Formats

Recommendation Report

A compilation of recommendations for the NAVAIR Career

Adrienne Somerville, NAVAIR (adrienne.somerville@navy.mil)

Charles Irons, NAVAIR (charles.irons.ctr@navy.mil)

Dorothy Cavanagh, NAVAIR (dorothy.cavanagh@navy.mil)

Xue Ding: xued@g.clemson.edu

Valerie Smith: vmsmith@g.clemson.edu

Ciara Marshall: ciara.m.marshall@gmail.com

Keith Sechrist: ksechri@g.clemson.edu

Dr. Tharon Howard: tharon@clemson.edu

Video evidence is referred to at the bottom of each recommendation within the

Gender 6 of 6 participants were male

Field of Study 5 mechanical engineering; 1 computer engineering

Career Plans 3 mechanical engineers; 1 software engineer; 1 manufacturing

Previous NAVAIR 0 of 6 participants

Phase 1: Participant Recruitment & Pre-Test Questionnaire

Phase 2: Participant Orientation

Phase 3: Performance Test

1) Installing and opening the application.

Phase 4: Post-Test Questions

Figure 1 Task Times

Comment Types by User:

General Navigation/Competency Organization: Competencies could be reorganized in a more

Exploration and Content

General Information: Positive- There was plenty of information under competencies.

Positive feedback: Keep the bookmark function.

directly from “My Notes” page by clicking on the button.

Appendix A: Consent Agreement (for distribution to participants)

Date: ____________ Signature: ___________________________________________

Initials: __________ Name (printed): _______________________________________

Date: ____________ Witness Signature: ____________________________________

Date: ____________ Test Administrator: ____________________________________

Appendix B: Pre-Test Questions for NAVAIR CGB Application Usability Study

Personal background information

7. What kind of cell phone operating system do you use?

Appendix C: Test Script

Appendix D: Post-Test Questions for NAVAIR CGB Application Usability Study

1.When on a page, how would you do the following?

a. Can you find a job description here?

b. What knowledge is expected at the senior level of this position?

c. What level of education is necessary for each level of the position?

i. Where would you go to find the education requirements?

5.Look at your saved pages and return to one of those pages.

a. While in that page, take a note and look at other notes.

b. Look at other bookmarks

6.How would you find the meaning of unfamiliar acronyms?

7. Find what jobs are available in your discipline?

a. What are the requirements for these positions?

Rate the Difficulty of Each Task

1. Downloading and opening the mobile application.

2. Send feedback/ create an email to contact NAVAIR.

3. Reach a specific page that you are interested in.

6. Return to home page from specific page.

8. Answering questions about the engineering career options

Folder locations and navigation

10. What could be done to make this task less difficult?

11. How useful do you think the NAVAIR CGB is to:

12. Would you use the NAVAIR CGB to:

1. Mobile User Experience Guidelines and Recommendations:

2. Best Practices for Mobile Application Developers:

3. Mobile app UX design: What strategies work and why

You might also like

Date: Signature: _______________________________

Initials: Name (printed): _____________________________

Date: Witness Signature: ________________________

Date: Test Administrator: ________________________