Reports

Communicating
about Quality

Quality
Communicating

Dr Tara Ryan,
Tblisi, June 2018
Agenda
09:30 – 10:00 Registration of participants
10:00 – 10:15 Welcome and speed networking
10:15 – 11:15 The path to the report – the SER and the big picture
11:15 – 11:30 Coffee break
11:30 – 13:00 Analysing an SER
13:00 – 13:45 Lunch
13:45 – 15:00 The panel visit and report writing.
15:00 – 15:15 Coffee break
15:15 – 16:30 Decisions and Standards
Style for today!
• Intended learning outcome – that when you (next) participate in a
you will have increased confidence and comfort to grapple with the
standards as manifested in a particular HEI and come to a fair,
evidence-based judgement, which reflects the particular environment
• Learning outcomes with action verbs – question, read, analyse,
critique, reflect, discriminate, judge, assess,
• Stories rather than technical facts – leave the legalities and
technicalities to National Center For Educational Quality
Enhancement (EQE)
• Definitions
Republic of Ireland – Quality and
Qualifications Ireland
Some ROI context
• No programme accreditation – managed internally by autonomous
HEIS
• No institutional accreditation
• Legislative Framework for the review of the effective of quality
assurance
 The path to the report
The Funnel =
The Institute
Artefacts
and
Meetings
Standards &
Documents
Criteria
Reflection

The Panel Decisions

and
Judgements
The Report
 The Route to the Report
• Criteria and Standards Established nationally
• Process Established nationally – broadly follows typical international model
1) Institution undertakes Self Evaluation and writes a report
2) Panel convened to analyse the SER, decide what questions and information is required
to see if the SER addresses and meets the standards
3) The site visit where the panel work together to seek the information and evidence to
complement the understanding gained from the SER
4) The panel makes and agree their conclusions on the meeting of standards and the
associated evidence
5) The secretary writes the REPORT, using the template
6) The Agency okays the report, and sends it to the HEI for factual verification
7) The panel make any required changes and send back
8) End
Critical Self Analysis – A Self
Evaluation Report
What is a Self-Evaluation
In higher education quality management, a self-evaluation is process of
systematic and critical self-analysis leading to judgments and/or
recommendations regarding the quality of either an institution or a
study programme.
Self-evaluation is a collective reflection providing an opportunity for
quality enhancement.
The resulting report can serve as a provider of information for a review
team, an external peer-review panel appointed to conduct an
independent, external evaluation against the same criteria.
 What are we doing in this process?
• Arguably there is an overarching question we are trying to
answer
• A report – whether a self-evaluation or the result of an external
evaluation – is an answer to a question, as well as sets of
questions
• The same principles apply to the writing of the HEI Self
Evaluation as to the External panel’s Evaluation
• It is about reflecting on something important and valuable and
making judgements, which may have multiple functions
• Formal compliance
• Confidence enhancing
• Identifying areas for improvement
Are we any good at what we do?
Are you good at your job?
Is your University any good?
Is your Agency any good?

How do you know?

Quality Assurance
• QA is “merely” doing a good job, by
• Knowing what my job is or what the mission of my HEI is
(knowledge of purpose)
• Knowing what good is (knowledge of standards and
expectations – linked to values)
• Being able to demonstrate that I do a good job or that my
HEI is a good HEI (key outcomes and performance indicators)
• So, in an evaluation the implicit question is
• Do you know how good you are?
An Unexamined Life is not worth living…
“[A]s Socrates demonstrated in his own life (and death) being fully
human … can be extremely challenging.
In a world of abiding uncertainty and complexity one can recognise a
certain attraction in not examining too much, for too long in life. Thus
the allure of those who offer to provide clear answers, simple
directions, precise instructions (whatever) so that you may set aside
examination and merely comply, or unthinkingly follow custom and
practice – perhaps living a conventionally moral life rather than an
examined ethical life. One can easily imagine how pleasant an
unexamined life might be.”

Dr Simon Longstaff, Sydney

 Developing an effective self-evaluation process
1. Consider the criteria (each criterion has an intent, a requirement to be met) What
is the purpose of the evaluation – what key questions is the evaluation seeking to
answer?
2. Identify the relevant information/data which needs to be collected to ensure the
criteria can be effectively responded to, and the optimum mechanisms for
collecting it
3. Identify the relevant stakeholders and how they should be consulted and their
feedback incorporated into effectively responding to the criteria
4. Map to standards by which it can be demonstrated the intents are minimally met
5. Agree the participants in the consultation, drafting, reflection processes and their
roles
6. Establish what judgements and decisions can be made and what evidence is
required to support these
7. Revise according to feedback received
8. Finally prepare results in clear and agreed formats
What do you see – Analysing the
SER?
Where do you look?
 Receiving the SER as a Panellist
• Prepare Prepare Prepare.

• Analyse the SER and the supporting materials provided by the

HEI.

• The Report Writing Starts Here.

On receiving the SER
• It is essential to read carefully including all of the appendices
• Challenges include – too much information, unfamiliar context,
different meanings of terms from those you are used to
• Try to find a coherent picture – what is important to the HEI – what
are they trying to tell you?
• Make notes as you read, spotting strengths, weaknesses, questions.
• It is very common to have a negative reaction to an SER, which is
erased at a site visit
• People matter
But

• Prior to the review the Team is discouraged from making any final
judgements.
• First impressions, whether positive or negative, should be the basis of
further discussion and evidence collecting throughout the pre site
visit and site visit.
• It is important to ensure that the initial list of comments and
impressions are tested and addressed throughout the review process
and are answered before the end of the Site Visit.
Be systematic
Sample SER
• Read 1-5 only!
• What questions would you ask of this HEI?
• What issues are arising?

• Let’s put them on a flip chart

Coffee break
Back to SER
• Read 6-7
• Are these similar questions?
 Standard Sample SER Focuses
• have a clear focus and purpose for the self-evaluation
• present a primarily evaluative rather than descriptive narrative
• provide an honest evaluation of institutional strengths and weaknesses
• incorporate broad consultation with internal and external stakeholders especially learners
• provide a sufficiently rigorous, systematic, evidence-based and comprehensive evaluation meeting all
of the objectives and criteria
• add value, minimise unnecessary overheads and assist in building capacity (i.e. it is not simply a paper
exercise or undertaken solely to satisfy a regulatory requirement)
• demonstrate evidence of leadership at all relevant levels within the institution and involvement of
relevant staff
• enhance understanding and ownership of quality assurance processes within the institution
• give appropriate consideration to the environment of the institution
• integrate with and build upon other related management processes where relevant (e.g. strategic
planning, operational management, internal audit, etc.)
• result in recommendations for improvement which the institution will factor into future plans
Sample questions when reading the SER for the first time might
include:
• Who wrote it?
• Who approved it? (governance and leadership)
• Who was on the development team?
• Were a range of staff, learners and stakeholders consulted?
• How long did it take to develop?
• How has it been disseminated within the institution?
• How well balanced is it between description and analysis?
• Is there evidence of genuine self-criticism and self-reflection?
• Is there any evidence of significant shortcomings or issues of concern in relation to the ESG or Standards for Compliance?
• Is there any evidence of significant shortcomings or issues of concern in relation to the management of quality assurance
and enhancement?
• Is there evidence of benchmarking against national standards and similar institutions?
• Is there evidence of benchmarking against international standards and similar institutions?
• Is there evidence of compliance with any regulatory requirements, including meeting qualifications’ standards?
• Is there evidence of the use of information and narrative sources of evidence?
• Is there evidence of commitment to a quality culture?
• Does it explicitly identify issues that the institution would like the Team to explore?
Back to SER

• Read 8-10
• What do you think of the findings?
There was nothing so very remarkable in that; nor did
Alice think it so very much out of the way to hear the
Rabbit say to itself "Oh dear! Oh dear! I shall be too
late!" (when she thought it over afterwards it
occurred to her that she ought to have wondered at
this, but at the time it all seemed quite natural); but,
when the Rabbit actually took a watch out of its
waistcoat-pocket, and looked at it, and then hurried
on, Alice started to her feet, for it flashed across her
mind that she had never before seen a rabbit with
either a waistcoat-pocket, or a watch to take out of it,
and burning with curiosity, she ran across the field
after it, and was just in time to see it pop down a
large rabbit-hole under the hedge.
In another moment down went Alice after it, never
once considering how in the world she was to get out
again
Reading an SER Pitfalls
• Over-reliance on the compliance standards
• Being swamped by the volume
• Going down seeming evidentiary rabbit holes
• Getting hung up on a hobby horse – also applies to panels
• Joint Awards
The Mapping Tool
• Use the mapping tool to discern your initial thoughts.
• Mapping Tool gives an opportunity for comments, queries
and concerns arising from first impressions and may offer
a base line ‘scoring’ of the Institution’s effectiveness in
relation to the core areas.
• The analysis will assist the process of individually and
then collectively identifying general themes, issues and
areas for further investigation or clarification.
• Beware of requesting more documents.
• This process will result in a shared list of issues that will
form the basis of discussions at Site Visit.
• These will be important to the Story for the report.
Lunch!
The Panel
The Characters
• The fluffy humanities professor who keeps going off the point
• The boring business professor who keeps obsessing about
commercialisation
• The superficial QA nerd who wants to know about some obscure
regulation or even worse, the lawyer….
• The engineer who everyone loves, because they ‘do process’
• Everyone has to work together
The Chair
• Leadership
• Guidance – to give and know when to seek
• Trouble-shooting
• Allocation of tasks
• Making sure everyone has a voice and no one is over burdened
• Corralling hobby horses
The Secretary, that is you
• Busy
• Don’t be shy
• Be nerdy
• Remember, you will be writing down the decisions – make sure you
have evidence to support the decision
• Do you understand and agree with the dialogue – a lot is happening,
make sure you synthesise your thoughts so that you are able to tell a
coherent story
The Secretary and Writing
• Start writing as soon as possible
• As possible, includes before the site visit – you can have the report
template pre-populated with the basic key descriptive information on
the HEI
• Distinguish between analysis and evidence
• Connect decisions to the analysis and supporting evidence
• Always substantiate your decisions
• Evidence, analysis and conclusions should be mutually supportive.
Chair role to work with panellists
• Start structuring thoughts into questions and themes – do you have a
theme that may help give overall coherency to the ensuing report?
• Consider evidence that might assist and whether the documentation
provided is sufficient
• Consider whether the draft agenda is structured in such a way as to make
it easy to source answers to questions and also to triangulate those
answers
• Check protocols around documents which need translation between
languages
• Use the mapping tool provided where it is helpful
At the Site Visit
Questions to Ask
Next 4 slides taken directly from - www.mahatma.am
1. HEI
2. External Stakeholders
3. Teaching Staff/Admin
4. Students
For the HEI
Will there be the managerial and administrative support needed?
Does the qualification fits well with the HEI’s strategic plans, priorities &
USPs …
Are there are senior managers who champion this particular study
programme …
Will learning resources & facilities be made available …
Has market research has shown the scale and nature of student demand …
Is there a business plan which demonstrates sustainability for the HEI &
affordability for students
Has a risk assessment been done: is all well?
How would key ‘external’ stakeholders … current & potential employers, professional
associations, public bodies, government agencies, international organisations, media
… be persuaded that this degree/institution was vital to the public interest?

• Have they been involved and listened to throughout the planning processes …
• Is employability & enterprise are embedded in the curriculum and assessments ---career planning,
placements, internships, business planning …
• Are work-based learning activities are available for part-time students …
• Does the study programme see higher education as local and global …
• Will the achievements of students will be showcased in the media …
• Are our Key Performance Indicators (KPIs)informed by external views of what these should be …
• Can high ‘value for money’ can be shown
• Are external stakeholders are involved in many aspects of the degree …
• Will professional associations will recognise the awards …
• Have national/international benchmarks have been applied …
• Does the programme/instituion makes good use of technology and emphasise digital literacy …
The programme/teaching team
Does the programme/teaching team work together?

• Is there a shared vision about the kind of Community of Practice (micro-culture)

that the team wants to build for and with our degree students …
• Is the study programme is progressive, and more than the sum of its parts
(modules) …
• Do the teachers, administrators and supporters of learning all work as a team …
• Do individuals understand their distinctive roles and responsibilities …
• Have similar programmes elsewhere have been studied and advice has been
welcomed from others …
• Have any training and support needs have been identified and addressed …
• Is the success of the study programme will be professionally important to the
team …
• Is the study programme is supported by extra-curricula events/activities …
Prospective Students
What is known about them?
• What are the details of the demographics, the nature of demand …
• What are student needs, knowledge & abilities, backgrounds &
experiences, expectations, passions & interests …
• What is the level of student preparedness for postgraduate study, learning
styles, language abilities …
• What are students’ personal, academic & career aspirations …

Have they been involved in discussions about the design of programmes?

• patterns, place, pace, topics, modes of study and assessment …
• role of technology-supported learning (distance, online, intensive) …
• internships, placements, work-based learning assessments …
Panel Members: At the end before you leave each
other….

• Agree the degree of compliance with each standard with key word, core
argumentation and evidence pointers
• Agree Commendations and recommendations
• Agree timing for communication with each other – make sure everyone is
available when you need them to sign off on the report
• Double check addresses
• Leave no decision unmade
• Check that everyone agrees
• Check that you have the evidence to support your decisions
What is the point of a report?
Writing the Report and Communicating to multiple audiences
Making Judgements to Share with Various
Audiences
The Decisions Available:
• Fully Compliant with the Standards
• Substantively Compliant with the Standards
• Partially Compliant with the Standards
• Not Compliant with the Standards
Available Decisions
• Full or substantial compliance may be impossible for some HEIs,
owing to restrictions placed on them by the very nature of their work
and/or legislation
• Context should to be duly explained in the review report.
An Example of How EQAR uses judgements
And Telling a Coherent Story
• What is the context and evidence for the decisions made by
the panel
• Common Themes in Reviews
• Governance
• Formal communication
• Appointment of people
• Who are the decision-makers – are they the ones formally nominated to
make decisions, how do you know
• Standard of Research
• Degree of Internationalisation
Be Positive
• Help to improve, not undermine
• Recognise the volume of work undertaken – both on paper and in
teaching, learning and research and service to society
• Recognise the diversity of perspective
Report Challenges
• Consistency
• Of panel engagement
• Of panel judgement
• Find and specifically agree core language that sends consistent
messages around strengths and weaknesses
• Use agreed terminology that move between all sections, but
without excessive ‘forumla’ approach
• Make it real, reflect good stories across various standards
Intelligibility
• Criteria and templates and Training – help improve
consistency, but may over-emphasise an evaluative system at the
expense of the people and institutions who are providing education
opportunities for learners and contributing ideas and vision for
society
• Think about the many accidental discoveries of academics, penicillin!
• Jargon free, but it is a technical exercise too
Exercise
What was the decision?
Steps
1. Read page 1, Standard 2.1
2. Read page 2, a report of a panel on how a Georgian HEI addresses
this standard
3. Questions
1. What recommendations would you make to the HEI
2. What level of compliance would you ascribe
The Standards
Quantitative and Qualitative
Standards
• They are the way we label “good”
• They are the boxes for “goodness”
Culture of Standards
• This impetus to produce monitorable norms for judging attainment is
an example of what Max Weber referred to as “rationalisation”, the
legalistic tendency to govern life by bureaucratic means. (William Keenan)
East and West – the QA caricatures
Compliance with rules ?
The Vertical and the Horizontal of Criteria
• Professor Ryan vs Dr Finnerty
 Things to Remember
• Some standards are proxies – and proxies for which there can be
challenging research
• E.g. If we operate a system of learning outcomes, i.e. we focus on what a
student will learn, why are we very definite about the inputs –
• Is the lower student-to-staff ratio (SSR) linked to high quality of education in higher
education institutions?
• Is it fair to use SSR as an indicator to assess the quality higher education?
• Is SSR a key determinant of learning outcomes?
• Is Caltech a good HEI because its SSR is 3:1 or??
• Be reasonable and transparent in your decision-making
• Be crystal clear on your evidence
• Try not to be reductionist
Standards and Compliance - The dangers

If someone can't see the wood for the trees (in British English), or can't
see the forest for the trees in (American English), they are very involved
in the details of something and so they do not notice what is important
about the thing as a whole.
The evaluation may not extend to the underlying values represented by
the defined linear/mechanistic standards for which compliance is sought.
A linear mechanistic approach can fail badly on a simple intellectual level
 Some Lithuanian examples - Research
In order to ascertain the relevance of research (applied research)
and/or art activities, it is necessary to analyse the following: Legislation
• alignment of research (applied research) and/or art activities • 3. Professional bachelor’s study
with the institution’s mission and strategic documents;
programmes of the first cycle may be
• alignment of research (applied research) and/or art activities
(and cycle 3 study programmes) with the priorities of the carried out by colleges
national and/or regional economic, cultural and social
development; • Bachelor’s study programmes of the
• impact of academic, social and business partners on the first cycle – by universities.
research (applied research) and/or art activities of the higher
education institution. • Study programmes awarding a degree
In order to ascertain the international links of research and/or of the second cycle may be carried out
art activities of the universities and their alignment with the by universities.
provisions of the European Research Area, it is necessary to
analyse the following: • Doctoral studies may be carried out by
• alignment of the higher education institution’s strategic universities or universities together
documents relating to research and/or art activities with the
priorities of the European Research Area; with research institutes.
• participation in international research and/or art projects;
• researchers’ and/or artists’ international mobility and the
impact of the visiting researchers and artists on the research
and/or art activities of the higher education institution.
 Governance – Panels differ
Article 20. Council of a state higher education institution - A
management body of a higher education institution shall be the • When is a
council.
• The Council shall consist of 9 or 11 members.
council not a
• The statute of the higher education institution shall fix an exact number of the council members.
council?
One member of the council shall be appointed by the representation of students in accordance • When you are
with the procedure laid down by it, and in the absence of such – by a general meeting
(conference) of students; other members of the academic community shall, in accordance with
between
the procedure laid down by the higher education institution, appoint respectively not more than councils and
four or five members. two members or, if the council consists of 11 members, three members when you are
shall be appointed by the teaching staff and the research staff; four or five members respectively, running an
who do not belong to the staff and students of the higher education institution, shall be selected,
appointed and recalled in accordance with the procedure laid down by the senate (academic
election?
council), with one member from them being selected, appointed and recalled in accordance with
the procedure laid down by the representation of students. These four or five members shall be
selected through open competition and appointed upon the evaluation of the assessment of the
candidates, carried out by the Council of Higher Education.
Balance: Evidence and Trust
• “In any field of endeavour where evaluation of quality is involved, as
it clearly is with academic performance, there is always scope for
discernment – a “sense”, a “nose for”, an “intuitive understanding” or
suchlike. Weber speaks of verstehen in such contexts, that is, the
empathetic insight that allows us to identify layers of subjective
quality below the external features of phenomena and experience.”
William Keenan
Criterion-based Decision Making and your gut
• Use the lens of the standards and criteria. This is the formal purpose of the
evaluation – they must be addressed
• But also use the lens of “common sense”
• Try reading the SER to get a feel for the story the institutions is trying to tell
• What is their message
• What is the world they live in
• What are the pressures they are under
• What do they think is important
• What do they want to achieve
• You are a panellist because you have experience – are there good stories
being told, are there things that need improvement?
• And how do these thoughts intersect with the criteria
Thank you
Resources and References
EXETER , DANIEL J. et al (2010) Student Engagement in Very Large Classes: The Teachers’ Perspective.
Studies in Higher Education 35 (7): 761 – 775
www.mahatma.am

William Keenan, Letter to THES, June 2018