11/14/2012

AN OUTCOMES- AND TYPOLOGY-BASED QUALITY ASSURANCE FRAMEWORK

FOR PHILIPPINE HIGHER EDUCATION INSTITUTIONS

Approaches and Practices in the Philippines:

Monitoring and Evaluation - a Gateway to Improved Performance and Accountability

2nd Annual M & E Network Forum

NEDA - UNICEF
November 8, 2012, Manila

SCHOOLS TYPE THEMSELVES (HORIZONTAL TYPE)

University
Professional Institute
College

CHED EVALUATES SCHOOLS WITHIN HORIZONTAL TYPE

USING OUTCOMES-BASED INSTRUMENT (VERTICAL TYPE)
Autonomous
Deregulated
Regulated

CHED & ACCREDITATION AGENCIES EMPLOY OUTCOMES-

BASED ACCREDITATION SYSTEM FOR REGULAR MONITORING
AND UPDATING OF STATUS

1
 11/14/2012

HORIZONTAL TYPOLOGY

HEIs will be differentiated functionally along:

• the qualifications and corresponding competencies of

their graduates; (STUDENT OUTCOMES; LEARNING
COMPETENCIES OUTCOMES; GRADUATE ATTRIBUTES)

• the nature of the degree programs offered;

• the qualifications of faculty members;

• the types of available learning resources and support

structures available; and

• the nature of linkages and community outreach

activities.

HORIZONTAL TYPOLOGY
Universities
are operationally defined as follows:

• Academic degree programs should be comprehensive and manifest

the pursuit of new knowledge.

• There are at least 20 academic degree programs with enrollees, at

least 6 of which are at the graduate level. The presence of graduate
students manifests the training of experts, who will be involved in
professional practice and/or discovery of new knowledge.

• There is at least one doctoral program in 3 different fields of study

(disciplines or branches of knowledge) with enrollees.

2
 11/14/2012

HORIZONTAL TYPOLOGY
Universities
are operationally defined as follows:

• All graduate programs and at least 50% of baccalaureate programs

require the submission of a thesis/project/or research papers.

• There should be a core of permanent faculty members. All full-time

permanent faculty members and researchers have, at least, relevant
master’s degrees. All faculty members teaching in the doctoral
programs have doctoral degrees. All other faculty members should
have the relevant degrees, professional licenses (for licensed
programs), and/or relevant experience in the subject areas they
handle.

• At least 30 full-time faculty members or 20% of all full-time faculty,

whichever is higher, are actively involved in research.

HORIZONTAL TYPOLOGY
Universities
are operationally defined as follows:

Any one of these 2 conditions:

• Annual research cost expenditure for the past five years is equivalent
to at least PhP 75,000 x the number of faculty members involved in
research (including external grants, monetary value of research load
of faculty members, equipment, and similar expenses credited to
research); or

• At least 5% of full-time faculty members engaged in research have

patents, articles in refereed journals, or books published by
reputable presses in the last ten years (including the CHED-
accredited journals)

3
 11/14/2012

HORIZONTAL TYPOLOGY
Universities
are operationally defined as follows:

• Comprehensive learning resources and support structures allow

students to explore basic, advanced, and even cutting edge
knowledge in a wide range of disciplines or professions.

• Links with other research institutions in various parts of the world

ensure that the research activities of the university are functioning
at the current global standards.

HORIZONTAL TYPOLOGY
Universities
are operationally defined as follows:

• Outreach activities allow the students, faculty members, and

research staff to apply the new knowledge they generate to address
specific social development problems, broadly defined.

These minimum requirements for Universities—particularly the

numbers and percentages pertaining to academic degree programs,
faculty, and costs—should be reviewed by 2017, to see if these are
responsive to the development needs of the country.

4
 11/14/2012

HORIZONTAL TYPOLOGY
Professional Institutions
are operationally defined as follows:

• At least 70% of the enrollment (graduate and undergraduate levels) is

in degree programs in the various professional areas (e.g.,
Engineering, Health, Medicine, Law, Teacher Education, Maritime, IT,
Management, Communication, Agriculture, Forestry, and Fisheries).

• At least 60% of the academic degree program offerings are in the

various professional areas (e.g., Engineering, Health, Medicine, Law,
Teacher Education, Maritime, Information Technology, Management,
Communication, Agriculture, Forestry, and Fisheries) and have
enrollees.

HORIZONTAL TYPOLOGY
Professional Institutions
are operationally defined as follows:

• There should be a core of permanent faculty members, with at least

50% of full time permanent faculty members having the relevant
degrees, as well as professional licenses (for licensed programs)
and/or professional experience in the subject areas they handle.

All other faculty should have the relevant degrees, professional

licenses (for licensed programs), and/or professional experience in
the subject areas they handle (e.g. In the event a professional
institute has doctoral programs, all faculty members teaching in
these programs have doctoral degrees).

5
 11/14/2012

HORIZONTAL TYPOLOGY
Professional Institutions
are operationally defined as follows:

• Learning resources and support structures are appropriate to the

HEI’s technical or professional programs.

• There are sustained program linkages with relevant industries,

professional groups and organizations that support the professional
development programs. Outreach programs develop in students a
service orientation in their professions.

These minimum requirements for Professional Institutions should be

reviewed by 2017, to determine if these are responsive to the
development needs of the country.

HORIZONTAL TYPOLOGY
Colleges
are operationally defined as follows:

• At least 70% of undergraduate programs have a core curriculum that

develops thinking, problem solving, decision-making, communication,
technical, and social skills.

• There should be a core of permanent faculty members, with at least

50% of full time permanent faculty members having the relevant
graduate degrees in the subjects they handle. All other faculty should
have the relevant degrees, licenses (for licensed programs), and/or
experience in the subject areas they handle (e.g. In the event the
college has doctoral programs, all faculty members teaching in these
programs have doctoral degrees).

6
 11/14/2012

HORIZONTAL TYPOLOGY
Colleges
are operationally defined as follows:

• Learning resources and support structures are appropriate for the

HEIs’ programs.

• Outreach programs allow students to contextualize their knowledge

within actual social and human experiences.

These minimum requirements for Colleges should be reviewed by 2017,

to see if these are responsive to the development needs of the country.

A “MOVING TARGET” APPROACH

The unique political economic realities of Philippine higher
education reform and the results of three rounds of CHED
stakeholder consultations on typology-based QA make it
necessary to operationalize the proposed horizontal typology
within a moving targets framework. This is in order to
galvanize the country’s community of higher education
stakeholders to move the reform. Once the change process
has commenced and the quality assurance systems have
taken root in a critical mass of Philippine HEIs, significantly
higher normative targets will be implemented within a five
year period to further raise quality standards to the level
comparable to the academic norms for higher education in
the Asia-Pacific region.

7
 11/14/2012

VERTICAL TYPOLOGY

Vertical classification is based on the assessment of the HEI’s

Commitment to Program Excellence, the demonstration of excellent
student learning outcomes, and Institutional Sustainability and
Enhancement, the development of a culture of quality.

MANIFESTATIONS OF QUALITY

Program excellence is manifested through:

• accreditation

• Centers of Excellence and Development

• international certification.

8
 11/14/2012

MANIFESTATIONS OF QUALITY
Institutional quality is manifested through:

• institutional accreditation

• Institutional Sustainability Assessment (ISA), or revised

Institutional Quality Assurance through Monitoring and
Evaluation (IQuAME)

• other evidences in the areas of governance and

management, quality of teaching and learning, quality of
professional exposure, research, and creative work,
support for students, and relations with the community.

• institutionalization and documentation of

systems/processes

VERTICAL TYPOLOGY

There are three types of HEIs according to vertical classification:

1
Autonomous HEIs (by Evaluation) demonstrate exceptional institutional
quality and enhancement through internal QA systems, and
demonstrate excellent program outcomes through a high proportion of
accredited programs, the presence of Centers of Excellence and/or
Development, and/or international certification. In particular, they show
evidence of outstanding performance consistent with their horizontal
type, e.g., research and publications for universities; creative work and
relevant extension programs for colleges; and employability or linkages
for professional institutes.

9
 11/14/2012

VERTICAL TYPOLOGY

There are three types of HEIs according to vertical classification:

2
Deregulated HEIs (by Evaluation) demonstrate very good institutional
quality and enhancement through internal QA systems, and
demonstrate very good program outcomes through a good proportion
of accredited programs, the presence of Centers of Excellence and/or
Development, and/or international certification. In particular, they show
evidence of very good performance consistent with their horizontal
type.

VERTICAL TYPOLOGY

There are three types of HEIs according to vertical classification:

3
Regulated HEIs are those institutions, which still need to demonstrate
good institutional quality and program outcomes.

10
 11/14/2012

We need some numbers:

Program Excellence is given a maximum of 70 points;

institutional Quality 30 points.

Autonomous if score is >= 80.

Deregulated if score >= 65 and < 80.

Regulated of score is < 65

VERTICAL TYPOLOGY
Criteria for Commitment to Excellence (70%)
maximum of 70 points

Max points that

Criteria No. of points can be awarded
(points)
COE 10 per COE
60
COD 5 per COD

Local accreditation see next slides 60

International accreditation
10/program 40
(CHED recognized-mobility)

International certification 10/program 20

11
 11/14/2012

EQUATIONS TO DETERMINE POINTS FOR LOCAL ACCREDITATION

The points for local accreditation takes into account several

factors.

Proportion of accredited programs. It is the proportion of

accredited programs in relation to the total number of programs
covered by accreditation that is measured

Level of accreditation. There are increasing weights (values)

from Level I to Level IV.

Undergraduate/graduate programs. The weights for

undergraduate or graduate programs depend on HEI type and
the proportion of programs at the two levels.

EQUATIONS TO DETERMINE POINTS FOR LOCAL ACCREDITATION

Total points for local accreditation is the sum of undergraduate

and graduate components:

Accreditation = UG Accreditation + G Accreditation

The weights for the UG and G components depend on the

enrollment. The points for accreditation are based on the sum
of the ratios for the different accreditation levels, multiplied by
a value for the level.

12
 11/14/2012

EQUATIONS TO DETERMINE POINTS FOR LOCAL ACCREDITATION

EQUATIONS TO DETERMINE POINTS FOR LOCAL ACCREDITATION

13
 11/14/2012

EQUATIONS TO DETERMINE POINTS FOR LOCAL ACCREDITATION

Weights for Accreditation Levels. After two years, it is recommended that accrediting agencies
use the outcomes-based approach in its instruments, for which higher weights will be given.

Outcomes based
Inputs based
(Level III & IV)
Level IV 1.25 1.50
Level III 1.00 1.25
Level II 0.75
Level I 0.50

Criteria for Institutional Sustainability and Enhancement (30%)

maximum of 30 points
Max points that can be
Criteria No. of points
awarded (points)

Institutional accreditation
based on program accreditation 25
30
using instrument for type-based institutional Points to be aligned with ISA
accreditation
Category A: 30
IQUAME (Categories from 2005-2010) Category B: 25 30

Ave ≥ 2.75: 30
2.75 >Ave ≥ 2.50: 25
ISA(Revised IQUAME) 30
2.50 > Ave ≥ 2.00: 20
Six sigma, Baldridge, PQA (different kinds)
ISO 2014: 25
Institutional certification ISO 9001: 20 25

Additional evidence(type-based)
 Governance & Management
 Quality of Teaching &Learning
 Quality of Professional Max 3/key result area 15
Exposure/Research/Creative Work
 Support for Students
 Relations with the Community

14
 11/14/2012

THE POINT SYSTEM FOR VERTICAL TYPOLOGY

Classification Min No. of points

Autonomous by 80 points plus evidence of the following:

Evaluation
University By 2014:

1. The Institutional Sustainability Score or its equivalent ≥ 2.75.

2. At least 50 full-time faculty members or at least 30% of full-time faculty, whichever is higher, have been actively
engaged in scholarly work (research or creative work) in the last two years. (Evidence of this includes
completed/progress reports, approved research grants, presentation at conferences, books and anthologies, and
documented creative work.)

By 2017:

1. The Institutional Sustainability Score or its equivalent ≥ 2.75.

2. At least 50 full-time faculty members or at least 30% of full-time faculty, whichever is higher, have been actively
engaged in scholarly work (research or creative work) in the last five years. (Evidence of this includes
completed/progress reports, approved research grants, presentation at conferences, books and anthologies, and
documented creative work.)
3. At least 10% full-time faculty has patents or publications in refereed journals. Of these, at least 5% of full-time faculty
has publications in internationally indexed journals and/or books published in reputable academic presses in the last
five years.

THE POINT SYSTEM FOR VERTICAL TYPOLOGY

Classification Min No. of points

Autonomous by 80 points plus evidence of the following:

Evaluation
Professional By 2014:
Institution
1. The Institutional Sustainability Score (e.g., ISA) or its equivalent ≥ 2.75.
2. Any two of the following:
a. At least one program with licensure, or 20% of the school’s programs with licensure, whichever is higher, has a passing
rate that is higher than the national passing rate in board/licensure exams, in the last five years
b. At least two programs are accredited under internationally agreed upon criteria and procedures, which promote
professional mobility across national boundaries (e.g., accreditation under the terms of Washington Accord by
ABET or by the PTC as a probationary member of said Accord; EUR-ACE, etc.)
c. Over the last five years, at least 80% of its graduates were employed within the first two years of graduation.
d. Sustained linkage with industry as evidenced by working program(s) that significantly contribute to the
attainment of desired student learning outcomes and to the employability of its graduates.

By 2017:
1. The Institutional Sustainability Score or its equivalent ≥ 2.75.
2. Any two of the following:
a. At least one program with licensure, or 20% of the school’s programs with licensure, whichever is higher, has a
passing rate that is at least 1.1 times than the national passing rate in board/licensure exams, in the last three
years.
b. At least two programs are accredited under internationally agreed upon criteria and procedures, which guarantee
professional mobility across national boundaries (e.g., accreditation under the terms of Washington Accord by
ABET or by the PTC as a full signatory of said Accord; EUR-ACE, etc.).
c. Over the last five years, at least 80% of its graduates were employed within the first two years of graduation.
d. Sustained linkage with industry as evidenced by working program(s) that significantly contribute to the
attainment of desired student learning outcomes and to the employability of its graduates.

15
 11/14/2012

THE POINT SYSTEM FOR VERTICAL TYPOLOGY

Classification Min No. of points

Autonomous by 80 points plus evidence of the following:

Evaluation
College 1. The Institutional Sustainability Score or its equivalent ≥ 2.75.
2. At least 80% of all graduates were required as students to participate in a community-based research/public service/
extension program for a cumulative period of two years.
3. Over the last five years, at least 20% of faculty members were engaged in research and extension services that
contribute to instruction and/or community development.

THE POINT SYSTEM FOR VERTICAL TYPOLOGY

Classification Min No. of points

Deregulated By 65 points plus evidence of the following:

Evaluation
University By 2014:

1. The Institutional Sustainability Score or its equivalent ≥ 2.50.

2. At least 30 full-time faculty members or at least 25% of full-time faculty, whichever is higher, have been actively
engaged in scholarly work (research or creative work) in the last five years.

By 2017:

1. The Institutional Sustainability Score or its equivalent ≥ 2.50.

2. At least 30 full-time faculty members or at least 25% of full-time faculty, whichever is higher, have been actively
engaged in scholarly work (research or creative work) in the last five years.
3. At least 7% full-time faculty has patents or publications in refereed journals.

16
 11/14/2012

THE POINT SYSTEM FOR VERTICAL TYPOLOGY

Classification Min No. of points

Deregulated By 65 points plus evidence of the following:

Evaluation
Professional By 2014:
Institute 1. The Institutional Sustainability Score or its equivalent ≥ 2.50.
2. Any two of the following:
a. At least one program with licensure, or 20% of the school’s programs with licensure, whichever is higher, has a passing
rate that is at least equal to the national passing rate in board/licensure exams, in the last five years.
b. At least one program accredited under internationally agreed upon criteria and procedures, which promote
professional mobility across national boundaries (e.g., accreditation under the terms of Washington Accord by
ABET or by the PTC as a probationary member of said Accord; EUR-ACE, etc.).
c. Over the last five years, at least 70% of its graduates were employed within the first two years of graduation.
d. Sustained linkage with industry as evidenced by working program(s) that significantly contribute to the
attainment of desired student learning outcomes and to the employability of its graduates.

By 2017:
1. The Institutional Sustainability Score or its equivalent ≥ 2.50.
2. Any two of the following:
a. At least one program with licensure, or 20% of the school’s programs with licensure, whichever is higher, has a
passing rate that is higher than the national passing rate in board/licensure exams, in the last three years.
b. At least one program is accredited under internationally agreed upon criteria and procedures, which guarantee
professional mobility across national boundaries (e.g., accreditation under the terms of Washington Accord by
ABET or by the PTC as a full signatory of said Accord; EUR-ACE, etc.).
c. Over the last five years, at least 70% of its graduates were employed within the first two years of graduation.
d. Sustained linkage with industry as evidenced by working program(s) that significantly contribute to the
attainment of desired student learning outcomes and to the employability of its graduates.

THE POINT SYSTEM FOR VERTICAL TYPOLOGY

Classification Min No. of points

Deregulated By 65 points plus evidence of the following:

Evaluation
College 1. The Institutional Sustainability Score or its equivalent ≥ 2.50.
2. At least 70% of all graduates are required to participate in a community-based extension program for a cumulative
period of two years.
3. Over the last five years, at least 15% of faculty members were engaged in research and extension service that
contributes to instruction and/or community development.

17
 11/14/2012

THE LONG-TERM GOAL

The long-term goal is to have the majority of HEIs:

• Progressively improving their level of attainment of

their desired program student learning outcomes
through outcomes-based accreditation and evaluation;
and

• Implementing an established internal quality assurance

system and undergoing institutional assessment
preferably using a standard type-based instrument,
such as ISA, which can be used by both the accrediting
agencies and CHED.

Both the outcomes-based accreditation system and the ISA

are the tools of M&E in higher education.

WHERE CHED IS NOW:

• CMO on Outcomes- and Typology-Based QA Framework

done with hearings; about to be issued; IRR drafted;

• Outcomes-based curricula and related matters about to

be formulated by CHED’s Technical Panels;

• ISA instrument and outcomes-based accreditation to be

discussed with accreditation bodies.

18
 11/14/2012

OUTCOMES-BASED ACCREDITATION
Paradigm shift from inputs-based to outcomes-based
accreditation in the last decade. In engineering and
computing, the Washington Accord members and ENAEE
members have adopted the new paradigm.

Inputs-based accreditation looks at output also as when they

look at board exam results and when they interview alumni
and possibly employers. But by and large it is a bean-counting
approach.

The US-based Accreditation Board for Engineering &

Technology (ABET) itself is managing the shift gradually. They
have mainly added new criteria to the existing ones, namely,
Program Educational Objectives, Student (Learning) Outcomes
and Continuous Quality Improvement (CQI).

OUTCOMES-BASED ACCREDITATION

OUTCOMES-BASED ACCREDITATION CRITERIA

• Program Educational Objectives

• Student Outcomes
• Students
• Faculty and Support Staff
• Curriculum
• Facilities and Learning Environment
• Leadership and Institutional Support
• Industry-Academe Linkage
• Continuous Quality Improvement (CQI)

19
 11/14/2012

OUTCOMES-BASED ACCREDITATION

Program Educational Objectives

• consistent with the mission and vision of the institution

• documented and published
• reflects the particular field(s) of practice and the associated
area(s) of specialization
• reflects the desired characteristics and/or capabilities of the
graduates after a few years of their career following graduation
• reflects the anticipated career destinations of graduates
• reflects the needs of the appropriate external constituencies.
• existence of a formal and documented process to develop,
review and monitor the program educational objectives – a
process that is periodic and that ensures and demonstrates that
the objectives are based on the needs of the program’s various
stakeholders with inputs from them.

OUTCOMES-BASED ACCREDITATION

Student Outcomes

• foster the attainment of the program educational objectives

by the graduates
• based on international standards
• established and documented
• evidence that graduates of the program possess the
attributes of the student outcomes by the time of
graduation

20
 11/14/2012

OUTCOMES-BASED ACCREDITATION

ABET Student Outcomes – International Standards

Baccalaureate degree programs must demonstrate that graduates have: 

(a) an

ability to apply knowledge of mathematics, science, and applied sciences 
(b) an
ability to design and conduct experiments, as well as to analyze and interpret data 
(c)
an ability to formulate or design a system, process, or program to meet desired needs

(d) an ability to function on multidisciplinary teams 
(e) an ability to identify and
solve applied science problems 
(f) an understanding of professional and ethical
responsibility 
(g) an ability to communicate effectively 
(h) the broad education
necessary to understand the impact of solutions in a global and societal context 
(i) a
recognition of the need for and an ability to engage in life-long learning 
(j) a
knowledge of contemporary issues 
(k) an ability to use the techniques, skills, and
modern scientific and technical tools necessary for professional practice.

OUTCOMES-BASED ACCREDITATION

Continuous Quality Improvement (CQI)

• Existence of a recorded process for assessment and evaluation of

the student outcomes.
• Existence of a recorded process for assessment and evaluation of
the program educational objectives.
• evidence that results of the evaluation of student outcomes and
results of the evaluation of program educational objectives are
utilized to make changes in the program processes such as course
syllabi, curriculum, and any other aspect of the program to
improve the degrees to which the student outcomes and program
educational objectives are achieved
• existence of feedback to and from all concerned stakeholders on
the achievement of undergraduate students for the student
outcomes and graduates for the achievement of the program
educational objectives.
• existence of a Continuous Quality Improvement program with
adequate supporting resources.

21
 11/14/2012

OUTCOMES-BASED ACCREDITATION

Inputs-based criteria had been useful to an extent in starting

schools on the journey towards quality by pointing out strengths
and weaknesses in resources and systems. However, the systems
and mechanisms for CQI are not well-defined.

Outcomes-based education and accreditation go hand-in-hand

naturally with CQI.

OUTCOMES-BASED ACCREDITATION
Other Terms
Term Definition
Used
Program Broad statements that describe the career
Goals, outcomes,
Educational and professional accomplishments that the
standards
Objectives program is preparing graduates to achieve.
Student Statements that describe what students are
Outcomes, expected to know and be able to do by the Objectives,
Graduate time of graduation that enable them to meet standards, goals
Attributes the program educational objectives.
Standards, rubrics,
Specific, measurable statements identifying
Performance specifications
the performance(s) required to meet the
Criteria outcomes, metrics,
outcome; confirmable through evidence.
objectives
a set of categories developed from the
performance criteria that define and describe
progression toward meeting important
components of work being completed,
Rubric
critiqued, or assessed. (Allows faculty to
assess student work in a way that identifies
the progress students are making toward
achieving the performance criteria.)

22
 11/14/2012

OUTCOMES-BASED ACCREDITATION

Other Terms
Term Definition
Used
Processes that identify, collect, analyze, and report
Assessment Evaluation
data that can be used to evaluate achievement.

Process of reviewing the results of data collection

Evaluation and analysis and making a determination of the Assessment
value of findings and action(s) to be taken.
Surveys used for either program educational
Assessment objectives or student outcomes; Any assessment
tools methods for which there are established protocols
(focus groups, portfolios, simulations, etc.)

EXAMPLE
Program Educational Objective: Graduates will exhibit effective
communications skills.

Student Outcomes: By the time of graduation, students will:

•demonstrate effective written communication skills. .

•demonstrate effective oral communication skills.

Performance criteria (indicators) for written communication skills:

•. Student provides adequate detail to support his/her

solution/argument.
•. Student uses language and appropriate word choice for the
audience.
•. Student work demonstrates an organizational pattern that is
logical and conveys completeness.
•. Student uses the rules of standard English.
Source: Rogers, Gloria, Assessment101 Assessment Tips, Community Matters, ABET
Monthly Newsletter, September 2006.

23
 11/14/2012

Direct assessments

• exams, quizzes, demonstrations, and reports.

• provide for the direct examination or observation
of student knowledge or skills against measurable
learning outcomes.
• provide a sampling of what students know and/or
can do.
• provide strong evidence of student learning.

Source: Rogers, Gloria, Assessment101 Assessment Tips, Community Matters, ABET

Monthly Newsletter, August 2006.

However,

not all learning can be measured in a direct way. For

example, a desired outcome of a course may be to create
more positive student attitudes toward mathematics (or
writing, or team work), which are difficult to assess using
direct methods.

Source: Rogers, Gloria, Assessment101 Assessment Tips, Community Matters, ABET

Monthly Newsletter, August 2006.

24
 11/14/2012

Indirect assessments

• ascertain the perceived extent or value of learning

experiences.
• assess opinions or thoughts about student knowledge or
skills.
• provide information about student perception of their
learning and how this learning is valued by different
constituencies.
• not as strong an evidence as direct measures because
assumptions must be made about what exactly the self-
report means.
• useful in that it can be used to measure certain implicit
qualities of student learning, such as values, perceptions,
and attitudes, from a variety of perspectives.
• in the absence of direct evidence, assumptions must be
made about how well perceptions match the reality of
actual achievement of student learning.
Source: Rogers, Gloria, Assessment101 Assessment Tips, Community Matters, ABET
Monthly Newsletter, August 2006.

Multiple Assessment Methods

• all assessment methods have their limitations and contain

some bias
• A meaningful assessment program would use both direct
and indirect assessments from a variety of sources
(students, alumni, faculty, employers, etc.).
• This use of multiple assessment methods provides
converging evidence of student learning.
• Indirect methods provide a valuable supplement to direct
methods and are generally a part of a robust assessment
program.

Source: Rogers, Gloria, Assessment101 Assessment Tips, Community Matters, ABET

Monthly Newsletter, August 2006.

25
 11/14/2012

Method Direct Indirect Method Direct Indirect

Exit and Other

X Locally Developed Exams X
Interviews

Simulations X External Examiner X

Written Surveys,
Behavioral Observations X X
Questionnaires

Archival Data X Portfolios* X

Focus Groups X Oral Exams X

Performance Appraisal X Standardized Exams X

* A portfolio is a purposeful collection of student work that demonstrates student achievement relative to specific competencies.
Portfolios can provide for the direct measure of student learning for the purpose of program improvement.

Source: Rogers, Gloria, Assessment101 Assessment Tips, Community Matters, ABET

Monthly Newsletter, August 2006.

Rubric

• an assessment tool used to measure students' work.

• a scoring guide that seeks to evaluate a student's
performance based on the sum of a full range of criteria
rather than a single numerical score.
• a working guide for students and teachers, usually handed
out before the assignment begins in order to get students
to think about the criteria on which their work will be
judged.
• can be created for any content area including math,
science, history, writing, foreign languages, drama, art,
music, etc...

Source: Rogers, Gloria, Assessment101 Assessment Tips, Community Matters, ABET

Monthly Newsletter, September 2006.

26
 11/14/2012

Example of a rubric:
2 - Progressing to 1 - Below
4 - Exceeds Criteria 3 - Meets Criteria
Criteria Expectations
Inconsistent or
Provides ample Provides adequate Some details but may
few details that
supporting detail to supporting detail to include extraneous or
Content may interfere with
support solution/ support solution/ loosely related
the meaning of
argument. argument. material.
the text.

Organizational Organizational Little evidence of

Little completeness &
pattern is logical & pattern is logical & organization or
wholeness, though
Organization conveys conveys completeness any sense of
organization
completeness & & wholeness with few wholeness &
attempted.
wholeness. lapses. completeness.

Uses effective
Uses effective Limited & predictable Limited or
language; makes
language & vocabulary, perhaps inappropriate
engaging,
appropriate word not appropriate for vocabulary for the
appropriate word
choices for intended intended audience & intended audience
Style choices for audience
audience & purpose. purpose. & purpose.
& purpose.

Consistently follows Generally follows the Generally does not Does not follow
the rules of rules for standard follow the rules of the rules of
standard English. English. standard English. standard English.

Source: Rogers, Gloria, Assessment101 Assessment Tips, Community Matters, ABET

Monthly Newsletter, September 2006.

Note that each category contains a gradation of levels of

completion or competence with a score assigned to each level
and a description of what performance criteria need to be met
to attain the score at each level.

The use of rubrics when scoring student work provides the

program with valuable information about how students are
progressing and also points to specific areas where students
need to improve.

For example, when a faculty member is grading a student’s

paper, she can also score the paper for the student’s writing
skills using the rubric provided. The scores obtained by each
student can be aggregated and used for program assessment.

Source: Rogers, Gloria, Assessment101 Assessment Tips, Community Matters, ABET

Monthly Newsletter, September 2006.

27
 11/14/2012

RESULTS OF SCORING STUDENT WORK AGAINST A RUBRIC

two sections of a course that has a total of 50 students

This not only demonstrates how many students meet the expected standard of “meets criterion” but also
the number who exceed standard and the number that are making progress. These data could also be
presented by the percentage of students who fall into each category.

Source: Rogers, Gloria, Assessment101 Assessment Tips, Community Matters, ABET Monthly
Newsletter, October 2006.

GRADES AS A MEASURE OF STUDENT LEARNING OUTCOMES?

Grades represent the extent to which a student has successfully
met an individual faculty member’s requirements and expectations
for a unit or course. Because many factors contribute to an
assigned grade, it is almost impossible to make inferences about
what a student knows or can do solely by looking at that grade.
In outcomes assessment at the program level, the primary
question that needs to be answered is, “To what extent do
students demonstrate the anticipated learning outcomes?” The
focus of program assessment is on providing evidence that
students can demonstrate knowledge or a skill that is directly
linked to specific performance criteria1 that define the program
outcomes. Grades per se are relative measures and generally do
not represent specific aspects of learning. More often, they reflect
performance on multiple concepts.
Source: Rogers, Gloria, Assessment101 Assessment Tips, Community Matters, ABET Monthly
Newsletter, December 2006.

28
 11/14/2012

Classroom Assessment

Within each course, a faculty member makes decisions about which

topics will be included based on the learning outcomes that students are
to achieve and how the course contributes to the rest of the curriculum.
Choice of topics depends on a number of factors — whether the course
is required or an elective, whether it is applied or theoretical, length of
course, characteristics of the course’s students, etc.

Once the topics have been chosen, the faculty member must decide
which concepts to focus on for each topic. This includes a decision about
the level of learning expected for each of the concepts. It is at the
‘concept’ level where assessment data are collected as evidence of
student achievement of the topic areas. The faculty member assigns
each student a grade for the course based on overall performance on
each concept for each topic.

Source: Rogers, Gloria, Assessment101 Assessment Tips

Program Assessment

The process of institutional/program assessment begins with consulting constituents

about attributes that employers/graduate schools would expect graduates to
demonstrate (educational objectives). Based on this understanding, it is necessary to
determine the outcomes that students will need to demonstrate by the time of
graduation to achieve the educational objectives. Each outcome must be defined in
terms of measurable performance criteria that create a unifying focus of program
teaching/learning strategies and data collection. The table below compares the
similarity between classroom and program assessment structure. In both cases, data
are taken at the concept/performance criteria levels and inferences are made about
student learning related to the topic/outcome.

Notice the similarity in the structure of the dimensions of learning moving from
general to specific learning attributes. Assessment (data collection) takes place at the
point where the outcomes are defined and measurable — at the ‘concept’ level in
classroom assessment and at the ‘performance criteria’ level in program assessment.
Once the assessment has been made, inferences can be made about the extent of
student learning at the topic/outcome levels.
Source: Rogers, Gloria, Assessment101 Assessment Tips

29
 11/14/2012

CLASSROOM vs PROGRAM ASSESSMENT

It is important to note that, in classroom assessment, faculty

are assessing individual students to provide feedback on their
performance in the form of grades. In program assessment,
faculty are collecting data from students to provide feedback
to the program on its performance. In the example below, the
performance criteria being used to assess ‘ethical
responsibilities’ may never appear in a course in ethics being
taught by a philosopher. The question is not “How do we
measure ethics or lifelong learning?” The question is “What
do we want our students to demonstrate so that we are
confident that they are meeting the outcome?” By creating a
few measurable performance criteria, the rest of the
assessment process falls into place.

Source: Rogers, Gloria, Assessment101 Assessment Tips

Source: Rogers, Gloria, Assessment101 Assessment Tips

30
 11/14/2012

OUTCOMES-BASED EDUCATION

• Development of program educational objectives

• Development of student learning outcomes
• Identification of a limited number of performance
indicators for each outcome
• Alignment of curriculum with learning outcomes
(curriculum mapping)
• Learner-centered systems to deliver the curriculum
• Classroom assessment and program review systems
• Evaluation (strengths, weaknesses, effectiveness)
• Use of evaluation data to make improvements (curriculum,
classroom strategies, the program educational objectives,
the student outcomes…)

31
 11/14/2012

THE CQI PROCESS / SYSTEM

A CQI process involves a clear understanding of

• mission (who you are, what you do, who you serve),
constituents (those who have a stake in the quality of the
‘product’);

• objectives (what one is trying to achieve);

• outcomes (what is being produced to meet the objective);

• processes (internal practices to achieve the outcome);

• facts (data collection);

• evaluation (interpretation of facts); and

• action (feedback to improve processes).

THE CQI PROCESS / SYSTEM

In a CQI process,

• each outcome must be defined in measurable terms

(performance criteria3);

• processes (e.g., curriculum, co-curriculum) need to be

examined and aligned to understand how they contribute
to the desired outcome;

• data should be efficiently and systematically collected to

see if the performance criteria that define the outcome
are being met.

32
 11/14/2012

THE CQI PROCESS / SYSTEM

A CQI system requires the data collection process to be

based on fact (direct measures of student performance), not
supposition or opinion (e.g., self-reported, anecdotal data).

Rubrics are appropriate for many of the data collection

methods but are not required in all situations to have a good
process.

Institutional Sustainability Assessment (ISA)

Core Indicators and Criteria
KRA 1: GOVERNANCE AND MANAGEMENT

Criterion: The institution’s governance arrangements

demonstrate probity, strategic vision, accountability, awareness
Core Indicator: Governance
and management of risk, and effective monitoring of
performance.
Criterion: The institution’s management, financial control, and
Core Indicator: Management quality assurance arrangements are sufficient to manage
existing operations and to respond to development and change.
Criterion: The institution has enabling features such as the use
of Information and Communication Technology (ICT) for
Indicator: Enabling Features more efficient and effective management; and a viable,
sustainable and appropriate resource generation strategy to
support its development plans.

33
 11/14/2012

Institutional Sustainability Assessment (ISA)

Core Indicators and Criteria
KRA 2: QUALITY OF TEACHING AND LEARNING

Criterion 1: Program Approval. The institution sets the objectives and

Core Indicator: Setting and
Achieving Program Standards
CORE INDICATOR: FACULTY
PROFILE
Core Indicator: Appropriate
Learning Resources
learning outcomes of its programs at appropriate levels, and has
effective mechanisms to ensure that its programs achieve those
objectives and enable students to achieve the intended outcomes
(including board passing rates).
Criterion 2: Program Monitoring and Review. The institution has
effective arrangements for monitoring the effectiveness of its
programs.
Criterion 3: Action to Strengthen Programs. The institution takes
effective action to address weakness, build on strengths, and
to enhance performance by the dissemination of good
practice.
Criterion: The institution has an adequate number of faculty with the
appropriate expertise and competence to teach the courses offered by
the institution.
Criterion: The institution makes effective use of learning resources,
such as library resources, laboratories, and information and
communications technology, to support student learning.

Institutional Sustainability Assessment (ISA)

Core Indicators and Criteria
KRA 3: QUALITY OF PROFESSIONAL EXPOSURE, RESEARCH, AND CREATIVE WORK

Criterion: The institution has programs that allow students

Indicator: Professional to practice their learned competencies in view of their future
Exposure careers, such as programs for practicum, internship, on-the-
job training (OJT), and case writing (for graduate level).
Criterion: The institution has a research community of
Indicator: Research faculty, students and postdoctoral research workers that
Capability fosters and supports creative research and other advanced
scholarly activity.
Criterion: The institution has programs that promote
Indicator: Creative Work
creative work in the arts and/or innovation in science and
and/or Innovation
technology.

34
 11/14/2012

Institutional Sustainability Assessment (ISA)

Core Indicators and Criteria
KRA 4: SUPPORT FOR STUDENTS

Criterion 1: Recruitment, Admission, and Academic

Indicator: Equity and Access
Core Indicator: Student
Services
Support. The institution is effective in recruiting, admitting,
supporting, and graduating students, including those from
indigenous groups, the handicapped, low-level income
classes, foreign students, and other special groups.
Criterion 2: Student Scholarships. The institution operates
effective arrangements to direct scholarships and study
grants on merit to support the most able students on
programs that develop competences needed to support the
Filipino economy and to enable the country to compete in
global labor markets.
Criterion: The institution has programs for student services,
to support the non-academic needs of the students.

Institutional Sustainability Assessment (ISA)

Core Indicators and Criteria
KRA 5: RELATIONS WITH THE COMMUNITY

Core Indicator: Relevance of
Programs
Indicator: Networking and
Linkages
Indicator: Extension Programs
Criterion: The institution offers programs that take into
consideration the social, cultural, economic, and
developmental needs of the country at local, regional, and
national levels, as well as the need for the country to compete
effectively in global markets.
Criterion: The institution is valued as a partner by other higher
education institutions; professional, government, and/or non-
government organizations; and industry, within the Philippines
and internationally.
Criterion: The institution is valued by its local community as a
provider of extension programs that are responsive to the
needs of the community for people empowerment and self-
reliance.

35
 11/14/2012

ISA Indicators by HEI Type

Core – Core indicator; Req – Required indicator; Indic – Indicator

Indicator Professional Institute College University

Governance and Management

Governance Core Core Core

Management Core Core Core

Enabling Features Indic Indic Indic

Quality of Teaching and Learning

Setting and Achieving Program Core Core Core

Standards
Faculty Profile Core Core Core

Appropriate Learning Resources Core Core Core

ISA Indicators by HEI Type

Core – Core indicator; Req – Required indicator; Indic – Indicator

Indicator Professional Institute College University

Quality of Professional Exposure,
Research, and Creative Work
Professional Exposure Req Indic Indic

Research Capability Indic Indic Req

Creative Work and/or Innovation Indic Req Indic

Support for Students

Equity and Access Indic Indic Indic

Student Services Core Core Core

36
 11/14/2012

PROFESSIONAL
UNIVERSITIES COLLEGES
INSTITUTES
OUTCOMES-BASED CQI,
ISA, M & E

MORE RATIONAL
DEPLOYMENT OF
RESOURCES

OUTCOMES OUTCOMES OUTCOMES

IMPROVED
PERFORMANCE

CLEARER ACCOUNTABILITY FOR

ENHANCING STRENGTHS &
ADDRESSING WEAKNESSES
COMMUNITY/INDUSTRY

THANK YOU!

37