Ist626 Qms 1 8

1
IST626 Advanced Instructional Design

Abbreviated Design Document
CSUMB Online Tutorials
Prepared by the Quality Matters Standards Team (QMS):
Brian Birdsall, Christina Indalecio, Colleen Jackson,
Lisa Lindert, Elizabeth Rutlin, Jenna West
Client Need
The client, The School of Information Technology and Communication Design (ITCD),
at California State University Monterey Bay (CSUMB) has begun to offer a professional
development course to department faculty members, titled The Online Teaching Workshop. The
curriculum of the Workshop focuses on training instructors to develop coursework for students
via an online learning environment. This instruction is part of ITCDs launch of a new, fully
online degree completion program in Computer Science and Information Technology
(CSIT). This fully online program format is the first of its kind in the CSU system. Instructors
will receive certification that they can competently utilize technology to direct and support the
online learning environment. The majority of the faculty members who will be taking the
workshop have not previously taught online, and it is possible that they may never have taken an
online course themselves. In order to match the high standards of the CSU systems traditional
pedagogy in the online platform, the faculty must become familiar with the requirements of Cal
State Onlines Quality Online Learning and Teaching (QOLT) and Quality Matters (QM)
rubrics in order to apply them to their own online courses. The intention of the module is not
only to teach the faculty about the standards, but to illustrate the standards in practice by utilizing
them in the development of the professional development coursework.
Quality Matters Standards 1-8
Statement of Work
The Online Teaching Workshop is intended for CSUMB faculty who will be
incorporating online instruction into their curriculum delivery. The workshop is comprised of
five sections. Two of the sections are the focus of this statement of work: Quality Matters
Standards 1-4 and 5-8. Created in Adobe Captivate, the modules feature the following
instructional tools:
Audiovisual elements such as videos and narration
Interactive tools that generate automated responses and feedback based on data entry and
other methods
Images to illustrate content
Written Text to emphasize critical concepts
A variety of self-check activities, including multiple choice, drag and drop, and true or
false quizzes
Each Captivate module is 15-20 minutes long

The tools will be used in the modules as well as being the subject of the modules, as the
target audience needs to acquire the skills illustrated in order to create valuable online
instructional content.
Online Presence and Interaction
The goal of both modules is to apply the Quality Matters rubric to an online course that best
supports student engagement via online learning. The QMS 1-4 module will instruct learners
about the following standards: Course Overview and Introduction; Learning Objectives;
Assessment and Measurement; and Instructional Materials. The QMS 5-8 module will instruct
learners about the following standards: Course Activities and Learner Interaction; Course
Technology; Learner Support; and Accessibility and Usability. The two content modules were
developed separately, with the intention of delivering them to the learners as a single module.
Content Delivery
Online education is a great opportunity for busy adults who are unable to attend a
traditional university. California State University offers a variety of online courses, which are
created using rigorous standards. New programs are added yearly. At the end of the Online
Teaching Workshop, a student participating in this module will be able to elaborate about the
importance of the Quality Matters Standards, and the role the standards play in CSUMBs online
curriculum.
Current technologies in online education include access to e-books, multimedia
presentations, and audio and video conferencing. All of these platforms include images,
narrations, and audiovisuals. The software required for producing and sharing digital content
includes several types of software programs and applications, such as e-learning platforms or
Learning Management Systems, (LMS), screen-sharing online forum programs such as Zoom
and Blackboard Collaborate, and online resources such as YouTube and Atomic Learning.
Without the proper structure these elements can become overwhelming and their inclusions can
be more distracting to the learner than helpful in communicating content.
Crafting good learning objectives that drive the structure of the course, and relating
digital resources to the course objectives is an essential key that our modules will cover.
Writing assessments to match course objectives will also be a focus of the modules presented.
Overall we want to promote the best practices for developing an effective and engaging online
class with all of the Quality Matters Standards considered and utilized within the course.
Assessment
The Quality Matters modules will instruct the faculty how to develop a course for an
online environment. The modules will focus on creating lesson specific learning objectives
(rather than course level learning objectives), helping students who are struggling with
accessibility, and designing assessments that provide evidence that students successfully
achieved the learning outcomes set out for a given course. The module will be divided into two
sections, identifying and assessing QMS 1-4 and QMS 5-8. By the end of the complete module,
learners will be able to:
a) Create a course that includes a solid introduction and overview
b) Identify the benefits of using the QMS rubric
c) Create learning objectives that are supported by assessment and measurement
d) Identify ways to provide learner support
e) Identify learner interactions and activities that will support student learning
f) Provide effective accessibility and usability through the course technology
Learner Profiles
Information about learners was gathered in four ways: live interviews with two CSUMB
faculty (Brian Robertson and Hawra Rabaan), interviewed using Zoom, https://www.zoom.us/);
Google Form-based survey of faculty (Brian Robertson, Hawra Rabaan, and Donald Fischer, see
Appendix B and C); a conversation with Dr. Miguel Lara, the SME on this project; and the
Learner Support Analysis section of the ADD document created by MIST CSUMB students in
the initial design of the Teaching Online Workshop (Avina et.al, 2014).
Both faculty that were interviewed in a live session were non-tenured faculty, have
earned a Masters degree, and have taught at the college level for no more than two years. They
were both currently going through the Online Teaching Workshop using the modules already in
place and taking the course in a self-paced, rather than cohort-based, manner.These two faculty
members represent one of two groups that are the target audience for the course (Avina et.al,
2014). The other group of learners are tenured faculty, over 40 years of age who have earned
Ph.D degrees and have taught at the college level for over 6 years. Learners in both groups are
highly motivated to learn because they have self-selected themselves for this course based on a
self-assessment of their current readiness to teach online. They are participating voluntarily and
spending their free time to improve their skills, with no additional compensation.
Both Mr. Robinson and Ms. Rabaan are participating in the Online Teaching Workshop
to help them prepare to teach in CSUMBs online Computer Science Program. This was the
primary target audience for the course at the current time, but this could be expanded in the
future to include any faculty on the CSUMB campus (Dr. Lara, conversation).
The faculty learners interviewed were both motivated to participate in the Online
Teaching Workshop due to their desire to teach online more effectively. Both had experience
taking courses online as students but had varied experiences with teaching online. They felt that
the 20 hours (total course time including modules other than Quality Matters) allotted for the
course was reasonable, however they did value flexibility in content delivery so they might move
through the course according to their personal schedule and learning needs. They did value their
time and although 20 hours in 2 weeks seemed reasonable, any longer time would be seen as a
burden unless it was spread out over a longer period. It was also noted that the exercises in the
current Quality Matters module would be more valuable if they were related to a practical
application, such as guiding the learner to create deliverables for courses that the faculty were
developing as instructors. The ability to receive feedback on that work, while valuable, may
prove to be time-consuming, may not fit in a 2 week 20 hour time frame or allow for flexible,
asynchronous learning. They were comfortable with working with other faculty collaboratively
and receiving feedback from others if this were a component of the course.
With respect to the learners comfort with technology, the development team found that
the learners that were interviewed in person and via the survey supported the profile of a learner
that is rather tech-savvy. All interviewees were familiar with using computers for routine
communications (email, forums, videoconferencing). Comfort with mobile use was slightly less
than usage of a computer to perform basic tasks. All learners had access to a computer and most
had a smartphone and/or tablet as well (see Appendix C for more detail). They will access the
Online Teaching Workshop using all of these devices, but will most likely use a PC or Apple
computer rather than mobile devices to access course materials.
With respect to the learners comfort and experience with online learning there were
some areas where the learners were less experienced and less comfortable than their experience
and comfort with technology. Based on the survey given, learners had experience with teaching
and had previously taken a course or workshop that provided guidance in teaching and learning,
however not all had taken a course that provided guidance in online teaching and learning. The
survey also showed that all learners were familiar with LMS systems as a student and as an
instructor. The learners were less comfortable with writing learning objectives, understanding the
difference between learning outcomes and course objectives, writing assessments and aligning
learning objectives and course assessments. The idea of active learning was something that not
all learners surveyed were familiar with (see Appendix C for more detail). During interviews
there was greater uncertainty about the teaching and learning aspect of online teaching than the
technical aspect.
Finally, the survey probed the learners familiarity with resources available for students
and faculty with respect to accessibility and technology support. Learners were not as familiar
with resources for teaching students with disabilities and were similarly unfamiliar with the
accessibility guidelines for developing online courses that are accessible to all students. The
learners knowledge of the technical resources for faculty was greater than their knowledge of
resources for students.
Overall the learners interviewed via live interview or online survey are knowledgeable of
the challenges of teaching online and are motivated to take the Online Teaching Workshop to
help them be more effective instructors. They have the requisite technological background and
have the tools to use computers and software effectively to perform the tasks required for setting
up and administering the course, as well as creating course materials. The only small exception
to this is their discomfort with being recorded, which is common as being on camera is awkward
and unfamiliar to most people.
The learners in the Online Teaching Workshop appear to have a greater knowledge of
technology than knowledge of effective instruction and student support. The areas where the
learners are less certain is in the areas of: support of students with disabilities; technical support
for students; and creation of course materials and resources (See Appendix C). In the creation of
course materials and resources the learners appear to be able to create digital course materials,
but lack guidance in the structuring of their course through the construction of learning
objectives, understanding the difference between learning outcomes and objectives and aligning
their learning objectives to assessment. The Quality Matters rubric provides guidance in these
areas and it would be helpful to include many examples and guidance in the writing of learning
objectives, the distinction between outcomes and objectives and the construction of assessments
that are aligned with standards.
Another area where learners require support is in their knowledge of accessibility and
technical resources and support for learners (see Appendix C). This is an area that can also be
addressed by the Quality Matters Rubric modules. Support for learners was a great concern
expressed by the instructors in the interviews. They felt it was important for student engagement,
motivation, performance and retention. They felt it was a challenge to motivate students online
due to the lack of personal interaction and they were looking for support in this area.
Learner Support Analysis

The learner support analysis was performed in 2014 by a prior cohort of CSUMB MIST
students and is summarized below (Avina et.al, 2014). The learner support analysis was
compiled following consultation with Dr. Lara, who is the client for this project and for the prior
work.
A computer or mobile device that can access the Internet is required to participate in the
module and all learners have access either through their workplace or personal electronic
devices. CSUMB will host the LMS (iLearn) that is required for management and presentation of
the content. Learners will most likely access this learning module at home, or in a location
outside the workplace because the learners are using their leisure time to complete the modules.
The modules are designed to be accessed at any time faculty have free, however they are also
designed to be used within a two-week window with faculty assigned to cohorts. Although this
was the original intention the two faculty interviewed were working asynchronously so both
modalities are accommodated with the course design.
The entire Online Teaching Workshop is designed to take 20 hours with the QM module
taking 6 of those hours. Time is a valuable resource for faculty, however faculty will not be
monetarily compensated for this work and participation is entirely voluntary.
The initial analysis, design and development, and implementation of the QM learning
module will take place during the Spring Term B session of IST 626: Advanced Instructional
Design (MIST, CSUMB). Students in this course will be completing these tasks and passing the
completed modules to Dr. Lara for feedback and refinement. Dr. Lara will take the completed
projects and incorporate them into the current Online Teaching Workshop within iLearn. The
implementation and evaluation will be conducted by Dr. Lara and other faculty at CSUMB
following sufficient use of the module for instructor training. Students will not be monetarily
compensated for their work on this project, but they will get credit for graduation in the MIST
program. Faculty administering the module will receive their normal compensation. Additional
support for technology and accessibility are available through CSUMBs faculty resources and
will not require any specialized training of support staff.
Task/Work Analysis
Content and procedures
As stated previously the learners are a mix of masters and doctorate level
instructors/professors who are highly motivated to learn, tech-savvy and are familiar with LMS
systems. However the learners have a greater knowledge of technology than knowledge of
effective instruction and student support. The module will be divided into two sections,
identifying and assessing QMS 1-4 and QMS 5-8.
10
The learner will require access to a computer, the internet and a compatible browser to
complete and engage with the lesson. The module can be accessed at anytime and from any
location, however it would be beneficial to have a quiet learning environment to help the learner
focus and concentrate on the lesson. The module is an asynchronous lesson and does not require
group interaction or the learner to take the course on a specific day or time. However, the
learners are expected to complete the module within a two week / twenty hour timeframe.
The learners are not required to come to the workshop with prior knowledge of the course
subject matter. All instruction will cover relevant topics and the learners will be given progress
assessments throughout the module that require them to answer a series of questions prior to
advancing to the next section. Question types will include a combination of true or false,
matching, fill in the blank and multiple choice. The modules would culminate in an overall
assessment of understanding.
Goals and instructional objectives
The overall goal of the workshop is to teach faculty members how to utilize technology
to direct and support the online learning environment, as well as ensure they are familiar with the
requirements of Cal State Onlines Quality Online Learning and Teaching (QOLT) and the
Quality Matters (QM) rubrics, so they can apply them in their own online courses. The course
objectives will be achieved by mastery of the learning objectives listed below.
The modules will include instruction on the following areas:
Standard 1: Course overview and introduction. Upon completing this section,
the learner will understand the purpose and structure of an online course and be
able to answer questions on how to set up an organized course structure. The
11
learner will also establish course guidelines and clear expectations for online
students.
Standard 2: Learning objectives. The learner will develop outcomes that are
measurable and consistent with the course-level objectives. The learner will
understand the importance of and how to draw relationships between learning
objectives, competencies and course activities that are clearly stated and suited to
the level of the course.
Standard 3: Assessment and measurement. The learner will understand how

assessments should be developed to measure the stated learning objectives or
competencies, as well as how to state the course grading policy clearly. In
addition, the learner will understand how to develop assessment criteria that is
tied to the course grading policy.
Standard 4: Instructional materials. The learner will comprehend how and why
instructional materials contribute to the achievement of stated course and
module/unit learning objectives and competencies, and how the materials are to
be used for learning activities.
Standard 5: Course Activities and Learner Interaction. The faculty learner will
be able to produce curriculum that uses interactivity to engage the online student
and promotes the achievement of the stated learning objectives.
12
Standard 6: Course Technology. The learner will be able to identify technology

that supports the course learning objectives and enhances the learning experience.
The learner will understand which tools promote online student engagement and
active learning.
Standard 7: Learner Support. The learner will include information regarding

internal and external support resources as part of their course content, including
technical support and accessibility services offered by the hosting educational
institution.
Standard 8: Accessibility and Usability. The learner will be able to implement

organizational methods and develop course navigation that is simple and intuitive.
The learner will understand how to develop course content that includes
information concerning the accessibility of all elements included in the course
activities.
III. Design: Quality Matters Standards 1-8

A design template consistent with modules previously developed for the Online Teaching
Workshop was collaboratively designed for use with the Quality Matters Standards 1 - 8. The
learning modules will follow the organization, standards, and conventions as identified in the
following section.
Test Instruments
Testing will consist of progress quizzes and two cumulative knowledge exams native to
the eLearning module. Progress checks and opportunities to practice will be provided after each
13
new concept is presented.. The two cumulative knowledge exams will be presented at the end of
each sections lessons (QMS 104 and QMS 5-8). hen they have finished reviewing the content
and completed all progress checks for each section, learners will be prompted that they may
proceed forward to the knowledge exam if they are ready.
The final knowledge exam for each section will consist of 10 questions - ten relating to
Quality Matters Standards 1 - 4 and ten questions relating to Standards 5 - 8. Content will be
graded and reported to the corresponding Learning Management System. In order to receive a
passing score learners must achieve a minimum score of 80% on each. The following are
examples of format and feedback to be provided.
True/False
Multiple Choice
Matching
Fill in the blank
The final component in the testing plan for the Quality Matters instructional material will
be developed by the administrators of the Online Teaching Workshop, as it is outside the scope
of this development project. As part of the Learning Management System (LMS) on which the
QMS modules will be hosted, the administrators of the workshop will include feedback forums
and practical exercises related to the content delivered.
Organizational, Delivery, and Management Strategies
Content will be delivered via eLearning module that will be accessed within a Learning
Management System. The content for Quality Matters Standards 1 through 8 will be developed
using Adobe Captivate, packaged for SCORM delivery, and able to be viewed in HTML/Flash.
Content will be organized within the eLearning module according to learning-related sequencing
themes that builds on learners skills and knowledge. The Quality Matters Standards themselves
14
will be the guiding topic headers. In addition to sequencing content in relation to Quality Matters
Standards, the content developed for this training will consist of the following segments to
ensure learner orientation.
Navigational Guidance. The first slide will contain direction and information on how to
proceed through the module content. The guidance will be presented both as part of the narration
and onscreen prompt text.
Module Introduction. The introduction will contain the topics below.
Gain Attention: Primes the learner's mind to be open, receptive and inquisitive about new
information.
Overview: Whats in it for me?
Learning Objectives: What will I be learning?
Method/Media/GUI Overview: How will I be learning the new information?
Evaluation: How will I be evaluated?
Transition: Focuses the learner back to the topic for presentation.
Presentation of Content. This section will break down the topics that make up the
Quality Matters Standards content. Information will be chunked according to each of the
standards.
QMS Standard 1: Course Overview and Introduction

QMS Standard 2: Learning Objectives
QMS Standard 3: Assessment and Measurement
QMS Standard 4: Instructional Materials
QMS Standard 5: Learner Interaction and Engagement
QMS Standard 6: Course Technology
QMS Standard 7: Learner Support
QMS Standard 8: Accessibility
Progress Checks. These segments will allow the learner opportunities to recall content
prior to being tested. Techniques will include opportunities to review Quality Matters Standards,
15
identify whether standards have been achieved or not, and how to ensure standards have been
implemented. Progress checks will not be graded.
While the progress checks will be in quiz form rather than in an activity format, feedback
given will contribute to the learners understanding through the use of concept repetition. In the
case of an incorrect response, an explanation why the given response is inaccurate and a
description of the desired response will be given to reinforce the concept being tested.
Knowledge Exam. This segment will aim to test the performance of the student through
evaluation. This will be administered after the learner has reviewed all content, completed all
progress checks, and is ready to move forward to evaluation. As a cumulative test, the final
knowledge exam will contain questions previously experienced by the user during the progress
tests as well as new questions that address higher level understanding.
Instruction
All aspects of course design, development and implementation will adhere to the
following proven strategies.
Objective and Task Analysis

12 Principles of Multimedia Learning
Theories of Adult Learning
Merrills First Principles of Instruction
Quality Matters Standards - Higher Education Rubric
Content. Design of content will be based on this design document, a completed
Storyboard that will detail the content and function of all elements, pages and media, as well as
all identified references pertaining to Quality Matters Standards. Content will be delivered via
web based eLearning module. This media will provide a Graphic User Interface, navigation,
16
external resources, additional media elements where appropriate, and present content to the
learners in a linear fashion.
Development
Development of the Captivate lessons has been split into two teams addressing four
standards each (1-4 and 5-8). The teams have completed two eLearning programs that will be
used as part of the Quality Matters module in the Online Teaching Workshop. The two
Captivate lessons will be developed over an eight week period. Teams will provide a storyboard
with narration included for the client. A prototype will be provided to the SME to ensure the
design complements previous lessons created. After approval, the team will produce content,
interactions, and narration for the lessons. Usability testing will take place after completion of
the project and changes made to the lessons based on feedback from testing.
Implementation
The Captivate lessons will be embedded by the SME into the Online Teaching Workshop
for the learners to access. The intended date of access is September 1st, 2015. This module will
be part of the overall workshop that professors must take to convert their teaching methods from
face to face, to an online only class. Learners will access the course through iLearn and
complete modules based on the current format of the workshop. The workshop is designed to
help professors understand the best practices in converting content from face-to-face to online.
VI. Assessment
Summative evaluation
The summative evaluation is composed of two separate instances of evaluation. The first
consists of multiple-choice learner self-checks at the end of each instructional unit. These self-
17
checks are intended to keep the learner engaged with the instruction as well as to offer corrective
feedback if necessary. When the learner chooses the correct answer, the feedback will state that
the learner selected the correct answer, state why, and allow the learner to continue to the next
question or to the rest of the learning module if that was the last question. When the learner
chooses the incorrect answer, the feedback provided will inform the learner that their selection
was not correct, state why, and then ask the learner to choose again. This will repeat until the
learner chooses the correct answer.
The second evaluation will be a learning evaluation assessment which will consist of a
pre-test and a post-test, administered as part of the usability testing interviews. Both the pre- and
post-test assessments will entail answering the same set of questions. This will allow for an
evaluation of the learning effectiveness of the instructional module. Although a statistical
analysis of the testing results will be out of the scope of work for this project, we recommend
one be conducted at a later date by the administrators of the Online Teaching Workshop. A
statistical evaluation should be conducted comparing the pretest and posttest in order to assess
the statistical significance of the learning outcome. The pretest and posttest will be conducted
using Google Forms (Appendix D). This will allow consistent administration, collection of data,
and evaluation both with the present set of test subjects and additional subjects moving forward.
Learning Effectiveness Evaluation

The learning effectiveness test will be conducted at a future date to ensure that learning is
taking place. The effectiveness tests will consist of a pretest and posttest for each module. The
pretest and post-test will consist of the same assessment questions in order to accurately gauge
the learning that is expected to take place. These two tests will be analyzed to determine the
18
statistical significance of the learning. This section describes the protocols we recommend be
followed in order to ensure accurate results.
Prior to completing the module for Quality Matters 1-4 and 5-8 the learner will be asked
to take the pretest (Appendix D). Immediately following the pre-test the learner will participate
in the designated module. After the completion of the module the learner will be asked to take
the posttest. After completing the posttest the learner will be asked to fill out the usability form.
The information will be recorded and evaluated in order to better the instructional module. No
personally identifiable information will be collected with any assessment.
All assessments and instructions for the assessments will be accessed and delivered
through Google forms. A link to each of the specific assessments will be available within the
CSUMB iLearn system where the modules will be hosted.
Long Term Evaluation

Usability testing (appendix F) will not only be used as an initial assessment upon
completion of the learning module it will also be used as part of an effort to capture long term
evaluation data. The ability to know what changes are necessary as the module gets used more
frequently is important in order to keep up with the needs of the current learner. The raw data
files will be made available so that continual changes can be made at any time to the learning
module including narration, assessments, text, and images.
b.
Formative/usability evaluation
Formative assessments will progress in tandem with the design of the instruction and then
culminate with a usability assessment of the completed work. As a section of instruction is

completed, it will be tested for usability by the instructional design group. This will allow for
prompt corrections and modifications, keeping progress fluid to completion.
19
The final evaluation will be conducted as an assessment of the instructional design in its
entirety using Google Forms. This evaluation will be completed by participants who will be
directly involved with the future use of the instructional product. The feedback obtained from
these users will be used to correct and fine-tune the instruction, thus ensuring an error-free user
experience. These assessors, who are also future users, will also be asked to provide feedback
regarding the match between the stated learning objectives within the module and whether the
evaluators felt that those objectives were met. In addition to assessing instructional goals and
usability, questions regarding the principles of multimedia will also be evaluated (Appendix F).
Usability Summary
Usability testing was conducted on a preliminary version of both the QMS 1-4 module
and QMS 5-8 module. Because the number of participating users was limited to three, the
feedback received is helpful but not conclusive. Additional usability testing is recommended
with a larger test pool of participants.
Of the three subjects that participated in the preliminary testing, all three were college
educated males, ranging from early forties to late sixties in age. Subjects #1 and #2 come from a
professional education background, working as instructors at the college level. As professional
educators, Subjects #1 and #2 have had prior experience with the QMS Standards and were
familiar with the instructional content prior to reviewing the modules. Subject #3 works in the
high tech industry and has no background in professional education.
Testing Protocols
Each subject was given access to and completed a Pre-test using a Google Docs form
prior to experiencing the instructional modules. Subjects then went through the instructional
modules and completed a Post-test for each afterwards. The Pre-test and Post-test contained the
20
same questions in order to measure the impact of the instruction. A Usability Survey was
completed by each user after finishing the instruction and tests in order to capture direct
feedback on the user interface of the instructional modules.
Subjects #1 & 2 were observed remotely via a video conference. The subjects screens
were shared with the administrator in order to enable observation of mouse movements. Audio
and video of each of the subjects was also captured, and the subjects were encouraged to verbally
communicate their thought process as they moved through the instructional modules. Both
Subject #1 and Subject #2 went through the usability process in a quiet home office environment
familiar to them. Subject #3 was observed in person, also in a quiet home office environment.
Test Execution
Subjects #1 & 2 completed the Pre-test for QMS Module 1-4 and proceeded directly to
the Pre-test for QMS Module 5-8. Both subjects then proceeded to undertake the instructional
modules, again in sequence - QMS 1-4 followed directly by QMS 5-8. Upon completing the
instruction, the subjects completed both Post-tests and Usability Surveys, again in sequence.
Due to the procedure followed by Subjects #1 & 2, feedback was combined on the
Usability Surveys for the different modules. For example, a feature not included in QMS 1-4 was
commented on in the QMS 1-4 Usability Survey, instead of on the survey for QMS 5-8, where
the feature was included. For this reason, the testing procedure was altered slightly for Subject
#3, the last to go through the process. The intention was to isolate feedback to the specific
module under review.
Subject #3 completed all steps for QMS 1-4 in sequence (Pretest, instructional module,
Posttest and Usability Survey) before taking a break and repeating the sequence for QMS 5-8.
QMS 1-4 Results
21
Methodology. By assigning a value of 1 for each correct answer, the Pretest results
were compared with the Postest results as well as between each subject. A full statistical analysis
was not conducted due to the small data pool available.
QMS 1-4 results. Subject #1 and Subject #2 scored 85.71% on the Pretest (12 correct out
of 14 possible). Subject #3 scored 8 of 14, or 57.14%. After completing the instruction and
taking the posttest, Subject #1 increased his score by 7.14%, to 92.86%. Subject #2 maintained
the same score of 85.71%. Subject #3, the user with no professional educational experience and
therefore no prior exposure to the content, increased his score by 35.71%. This would indicate
that a user with no prior experience with the Quality Matters material would experience a
significant benefit from completing the QMS 1-4 module. Please see Charts 1 and 2 for an
illustration of results.
Chart 1: QMS 1-4 Test Scores
22
Chart 2: QMS 1-4 Improvement
Verbal feedback received from Subject #2 during the instructional module quizzes and
the Pre- and Post-tests indicates the Subject was aware of the correct answer based on the QMS
Standards, but disagreed with them. Answers were chosen based on personal belief rather than
content presented. At the time the usability tests were conducted, the feedback portion of the
quizzes had not been implemented. It is the opinion of this researcher that the incorporation of
informative feedback to quiz questions included in the module will increase the success rate of
the learners.
QMS 5-8 Results
23
Methodology. By assigning a value of 1 for each correct answer, the Pre-test results
were compared with the Post-test results as well as between each subject. A full statistical
analysis was not conducted due to the small data pool available.
QMS 5-8 results. Subject #1 and Subject #2 scored 85.71% on the Pre-test (6 correct out
of 7 possible). Subject #3 scored 3 out of 7, or 43%. After completing the instruction and taking
the Post-test, Subject #1 increased his score by 0%, maintaining the 85.71% score. Subject #2
increased their score by 14.29% to a score of 100%. Subject #3, the user with no professional
educational experience and therefore no prior exposure to the content, increased his score by
57% to achieve all correct answers. This would indicate that a user with no prior experience with
the Quality Matters material would experience a significant benefit from completing the QMS 58 module. See Charts 3 and 4 below for an illustration of results.
24

Usability Survey Results
Overall, both QMS modules were able to be accessed and completed by each of the
subjects. There were issues with the modules being accessed using the Firefox browser, with
users being notified that some content may not be able to be displayed within that medium.
Subjects expressed preference for consistent interactions (clicks, hovers, transitions) throughout
each of the courses and the desire to receive more direction on the conclusion of each slide and
when the learner is expected to move forward. Upon reaching the the summative evaluation
within the course modules, test subjects expressed their interest in receiving immediate feedback
on all questions and clarification on the number of correct questions in order to receive a passing
score. Another subject expressed that they felt the questions were repetitive in nature, and needed
to be further differentiated from one another.
25
Feedback received in the Usability Survey indicated that users found navigation of the
courses could be improved, and again referenced inconsistencies in interactions as part of the
confusion. Subjects also expressed that the automatic advance of slides was not preferable, as
they would rather have control and spend as much time as they need to review content. Overall,
users rated the instruction well, indicated that their attention was held, and that the pace was
acceptable. The results of the usability survey demonstrate the overall effectiveness of both the
QMS 1-4 and 5-8 learning modules, but also the areas that should be improved to better
accommodate the learner.
Conclusion
Learners from both QMS modules were able to access and improve their knowledge after
completing the content. Although the pool of subjects was too small to provide conclusive
evidence, the given data demonstrates that learners had a generally positive opinion about the
modules and the content. Feedback collected during usability and testing will be used and
implemented into course content, especially those referencing consistency, quiz feedback, and
navigability. The increase between Pre- and Post-test results indicate that the learners
demonstrated and increase in knowledge and that the content was beneficial in completing a final
QMS summative evaluation.
VII.
26
Appendices
Appendix A
General Learner Questions (in-person questions)
1. As an introduction, please describe your academic background and career path? What
path led you to this interview?
2. Have you ever taken a course online?
a. What did you like about it? Dislike?
3. Are you planning to convert one of your classes to an online format?
a. Next fall? Or next spring?
b. What is the topic of the class you plan to convert?
c. What motivated you to teach an online class?
4. Do you feel you currently have the technical skills to teach an online class?
a. What areas do you feel you may need more training?
b. Do you feel you need to upgrade your computer equipment to participate in an
online course or program of study?
27
c. Do you enjoy learning new technical skills?

d. Are you comfortable sharing your work and educational experience with your
peer learners?
5. How much time do you feel is appropriate for your training in online teaching and
learning?
6. How much time do you feel you have available for your training in online teaching and
learning?
7. When taking the Online Teaching Workshop, how will you allot and spend time when
you perform the modules?
8. Will you find it helpful to work with other faculty during the workshop?
9. Is it important to be able to work at your own pace through the online workshop?
10. What do you perceive as the most challenging about teaching an online class?
11. What do you feel is most different between an online class and a face-to-face class?
12. What do you feel are the advantages of online courses over face-to-face classes?
13. What do you want your students to know about online courses?
14. What are the things that you feel should be present in both online and face-to-face
instruction?
15. What do you perceive as the most challenging about teaching an online class?
16. What do you want your students to know about online courses?
17. What are some areas in which you feel online course designers could do a better job?
18. What design features and functions would you include in this course?
Appendix B
Learner Survey (taken from Google Forms, http://goo.gl/forms/VGvbdCveE1)
28
29
30
31
32
33
Appendix C
Summary of Responses to Google Form
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
Appendix D
Pre and posttest
53
54
Appendix E
Usability survey
55
56
57
58
References
Avina, C. Bettencourt, J., Bradley, A., Osterweil, A. Rincon, A., and Simmons, J. (2014)Ist 626
Abbreviated Design Document CSUMB Online Tutorials. California, United States. California State
University Monterey Bay.

Ist626 Qms 1 8

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ist626 Qms 1 8

Uploaded by

Copyright:

Available Formats

1

IST626 Advanced Instructional Design

Quality Matters Standards 1-8

Audiovisual elements such as videos and narration

Images to illustrate content

Written Text to emphasize critical concepts

Each Captivate module is 15-20 minutes long

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Learner Support Analysis

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Standard 3: Assessment and measurement. The learner will understand how

Quality Matters Standards 1-8

Standard 6: Course Technology. The learner will be able to identify technology

Standard 7: Learner Support. The learner will include information regarding

Standard 8: Accessibility and Usability. The learner will be able to implement

III. Design: Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

QMS Standard 1: Course Overview and Introduction

Quality Matters Standards 1-8

Objective and Task Analysis

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Learning Effectiveness Evaluation

Quality Matters Standards 1-8

Long Term Evaluation

culminate with a usability assessment of the completed work. As a section of instruction is

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Chart 1: QMS 1-4 Test Scores

Quality Matters Standards 1-8

Chart 2: QMS 1-4 Improvement

QMS 5-8 Results

Quality Matters Standards 1-8

Chart 3: QMS 5-8 Test Scores

Quality Matters Standards 1-8

Chart 4: QMS 5-8 Test Scores

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

c. Do you enjoy learning new technical skills?

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8

Quality Matters Standards 1-8