Hatland Conlon Evaluation Plan For Chicago Center 1

Running head: EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM
Evaluation Plan for the Chicago Center Program

December 1, 2015
Anne Conlon, Marie Hatland
Loyola University Chicago
EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM
Table of Contents
Evaluation Plan for the Chicago Center Program ........................................................................... 4
Program Context ......................................................................................................................... 5
Goals and Objectives ................................................................................................................... 6
Program Design ........................................................................................................................... 7
International House.................................................................................................................. 8
UNIV 110: Exploring the United States through Chicago. ...................................................... 9
Excursions to Major U.S. Cities .............................................................................................. 9
Setting and Context ................................................................................................................... 10
Stakeholders and Organization.................................................................................................. 10
Evaluation Rationale ..................................................................................................................... 12
Logic Model .................................................................................................................................. 13
Inputs ......................................................................................................................................... 14
Outputs ...................................................................................................................................... 14
Outcomes ................................................................................................................................... 15
Assumptions .............................................................................................................................. 16
External Factors......................................................................................................................... 17
Evaluation Approach .................................................................................................................... 18
Strengths and Weaknesses ........................................................................................................ 19
Outcome Specifics..................................................................................................................... 19
Quantitative Approach .................................................................................................................. 21
Research Design ........................................................................................................................ 21
Survey Instrument ..................................................................................................................... 22
Procedures and implementation ................................................................................................ 25
Validity and Reliability ............................................................................................................. 26
Analysis ..................................................................................................................................... 27
Qualitative Approach .................................................................................................................... 29
Focus Group Participants .......................................................................................................... 29
Focus Group Procedures ........................................................................................................... 31
Focus Group Moderators ........................................................................................................... 32
Implementation Plan ................................................................................................................. 33
Focus Group Analysis ............................................................................................................... 33
Validity and Reliability ............................................................................................................. 34

Timeline ........................................................................................................................................ 35
Budget ........................................................................................................................................... 36
Reporting....................................................................................................................................... 37
Next Steps ..................................................................................................................................... 37
References ..................................................................................................................................... 39
Appendix A: UNIV 110 Syllabus .................................................................................................. 42
Appendix B: Los Angeles Excursion Itinerary ............................................................................. 45
Appendix C: Logic Model for Loyola University Chicagos Chicago Center ............................. 46
Appendix D: Survey Matrix.......................................................................................................... 47
Appendix E: Pre-Test Survey ....................................................................................................... 54
Appendix F: Post-Test Survey ...................................................................................................... 59
Appendix G: Focus Group Email Invitation ................................................................................. 65
Appendix H: Focus Group Consent Form .................................................................................... 66
Appendix I: Focus Group Protocol ............................................................................................... 68
Appendix J: Focus Group Coding Construct Map & Matrix ........................................................ 72
Appendix K: Focus Group Coding Rubric ................................................................................... 73
Appendix L: Timeline ................................................................................................................... 74
Appendix M: Budget..................................................................................................................... 75
Appendix N: Presentation ............................................................................................................. 76
Evaluation Plan for the Chicago Center Program

The Chicago Center at Loyola University Chicago (LUC) provides a study abroad
experience for international students during a semester, academic year, or summer session of
study, with Chicago as the host city. In an effort to introduce short-term international students to
American culture and society, Chicago Center students are required to enroll in a customized
seminar on the history and culture of the city of Chicago, UNIV 110 Exploring the United States
through Chicago; maintain residency in the cross-cultural residence hall, the International
House; and participate in two travel excursion trips per semester to major U.S. cities.
Additionally, the Chicago Center offers optional program components through the International
House, such as cross-cultural events and the Global Mentorship program (Chicago Center,
n.d.).
This evaluation approach is centered on the Chicago Centers desire to determine if the
programming is achieving its primary goal: to provide a high-quality academic immersion into
American society. This formative evaluation plan will be able to supply concrete information
regarding recommended improvements for the program. The evaluation plan explains the
Chicago Centers context, goals and objectives, the program design, stakeholders, and
anticipated student developmental outcomes. Understanding the scope of the Chicago Center
program is essential when considering the overall evaluation approach. This mixed methods
evaluation plan looks specifically at students experiences in each of the program components as
well as their general intercultural competency development throughout a semester of
participation in the Chicago Center program. This evaluation has been designed deliberately to
be both useful and feasible in anticipation of its implementation.
Program Context
Although the United States is the leading educational destination of globally mobile
students, higher education institutions often overlook academic and social integration of the
international student population (Glass, Buus, & Braskamp, 2013, p. 3). Prior to the
development of the Chicago Center program, international students interested in studying at
LUC needed to fit into one of three specific categories:
1. Full degree-seeking international students
2. Direct exchange students in which LUC has specific relations to allow exchange students
to participate in short-term study
3. Students who pursue English as a Second Language (ESL) coursework only
Recognizing a need for academic opportunities for short-term international students who
seek academic courses other than ESL, Patrick Boyle, Vice Provost of Academic Centers and
Global Initiatives, began to develop the Chicago Center in 2009. Boyle also remarked on an
increased need in developing a short-term program since higher education institutions across the
world have improved greatly, rendering a decreased value in an American degree (personal
communication, September 11, 2015). In fact, the market share of international students in the
United States has dropped from 23% to 17% in the last ten years, which may imply a decrease in
desirability for an American degree (Glass et al., 2013).
The Chicago Center was conceptualized to maintain the legacy of LUCs international
academic centers: the John Felice Rome Center (JFRC), the Beijing Center (TBC), and the
Vietnam Center (P. Boyle, personal communication, September 11, 2015). LUC has maintained
a tradition of providing study abroad opportunities to American students at these centers since
the inception of JFRC in 1962. The Chicago Center was modelled heavily after TBC, a center
that incorporates quality academics and unique excursions to provide students with a Jesuit
education and introduction to the culture and society of China (P. Boyle, personal
communication, September 11, 2015).
The Chicago Center was officially launched in Fall 2011 with an enrollment of five
students, all of whom were recruited from the University of International Business and
Economics (UIBE) in Beijing, China. LUC has maintained a partnership with UIBE; most
notably, UIBE houses LUCs academic center, TBC. Enrollment has continued to grow each
semester and has noted a diversification of the student nationalities, although Chinese and
Brazilian students have remained a stronghold in the most recent semesters.
Goals and Objectives
The Chicago Center intends to provide a high-quality academic immersion into American
society through a study abroad experience at LUC. The program is emboldened by the
theoretical framework of Kolbs Experiential Learning Theory (ELT), which is defined as the
process whereby knowledge is created through the transformation of experience (Kolb, 1984, p.
41). This learning style is represented in four stages: (1) Concrete experience, in which a student
has an experience; (2) Reflective observation, in which the student reflects on an experience; (3)
Abstract conceptualization, in which the student learns from the experience; (4) Active
experimentation, in which the student implements what they have learned (Kolb, 1984). By
applying ELT to the Chicago Center program, one can better understand and improve upon how
students learn abroad and aid in creating evolving education programming (Brandauer &
Hovman, 2013, p. 107). Positive rhetoric urges that student development occurs merely through
participation in a study abroad program (Brandauer & Hovman, 2013; Engberg, Jourian, &
Davidson, 2015; Vande Berg, Paige, & Lou, 2012). However, there is increasing literature,
which stresses the importance of intentionality in a study abroad program design in order to
achieve developmental outcomes (Brandauer & Hovman, 2013; Engberg et al., 2015; Vande
Berg et al., 2012). Through participation in the Chicago Center program, students are expected
to become more immersed in American culture and develop a higher intercultural competency
through interaction with fellow Chicago Center students from different national
backgrounds. Programmatic components such as mentoring, incorporating cultural content,
reflecting on intercultural experiences, and engaging with the culture have been highlighted as
ideal features of a study abroad program (Engberg et al., 2015, p. 2). The Chicago Center has
implemented certain highlighted programmatic components, as detailed in the following section.
Little research exists that explores an intentional study abroad program for international
students in the U.S.; the vast majority of study abroad literature focuses on the American student
abroad. The lack of information and research on study abroad programs for international
students in the U.S. reflects the uniqueness of the Chicago Center program.
Program Design
Chicago Center students are required to participate in three main components of the
program:
1. Residency at the International House, a cross-cultural living community on Loyolas
Lake Shore Campus (LSC)
2. Enrollment in the semester course, UNIV 110: Exploring the United States through
Chicago with instructor Michael Hines, a doctoral candidate in the Cultural and
Educational Policy Studies program
3. Two trip excursions to major U.S. cities each semester: Washington DC and Los Angeles
during Fall semester; New York City and New Orleans during Spring semester
In addition to the three main components, students are urged, but not required, to
participate in the Global Mentorship Program in order to engage more with American
students. The Chicago Center recently restructured the design of this mentorship program
because many American partners were not fulfilling their agreements in the past years. Chicago
Center students are able to enroll in most undergraduate academic courses and are able to join
student groups and activities of their choosing on campus.
The Chicago Center launched a summer session program in 2015, although the
developmental outcomes of this session are unclear since students are only required residency at
the International House. Chicago Center students pay subsidized semester tuition at Loyola
University Chicago of $13,000 per semester. This fee covers full-time enrollment in courses,
housing at the International House, and two trip excursions to major U.S. cities. LUC discounts
tuition significantly in order to attract participants (P. Boyle, personal communication,
September 11, 2015). Non-Chicago Center students generally pay around $23,000 per semester
for tuition and housing as of Fall 2015.
International House. This residence hall houses Chicago Center students, short-term
direct exchange students, and several domestic American students. The Chicago Centers main
office is located on the first floor of the International House. In an effort to provide a crosscultural experience in housing, the International House hosts various culture-specific events such
as Chinese New Year Event, Brazilian Carnivale Night, and Japanese Culinary
Night. Additionally, the International House hosts American-specific events such as Halloween,
Thanksgiving, and Valentines Day celebrations. There is entertainment space on the main floor
with a kitchen, dining room, and a lounge area. Chicago Center students are placed intentionally
with roommates of different nationalities in order to foster a cross-cultural experience. While a
few international students have American roommates, most do not due to the low number of
American students living in the International House.
UNIV 110: Exploring the United States through Chicago. This required course is
designed to introduce key aspects of Chicagos history and culture to Chicago Center students
(Appendix A). Through exploration of the city of Chicago, the course aims to strengthen
English language communication and promote cross-cultural collaboration in the classroom. The
course also features expert guest speakers to present on salient topics such as Chicago politics
and music. In addition to the in-classroom experience, several site visits to various Chicago sites
are required such as a trip to the Chicago History Museum and the Art Institute of Chicago. For
the final class project, students are required to reflect on their experience in the U.S. and present
their findings. Reflective activities are critical pedagogical tools within study abroad (Engberg
et al., 2015, p. 3).
Excursions to Major U.S. Cities. Jason Obin, Director of the Chicago Center, and
Rebecca Kahn-Witman, Chicago Center Graduate Assistant, design and plan the excursion
trips. Supplemental LUC staff is asked for extra support during excursion trips with the
increasing number of Chicago Center students. The excursion trips were conceptualized to
broaden Chicago Center students understanding of American culture.
The excursions are characterized by historical site visits and cultural activities
representative of the U.S. city. For example, the excursion to Los Angeles includes a visit to
Warner Bros. Studio and Hollywood to introduce students to the importance of the TV industry
in the city (Appendix B). In an effort to have students engage cross-culturally, hotel roommates
are chosen intentionally so students are housed with a roommate of a different
10
nationality. Students are also required to interact with different nationalities at group meals in
order to encourage conversations in English.
Setting and Context
The Chicago Center is based on LUCs Lake Shore Campus (LSC) at the International
House and uses the city of Chicago as a backdrop for exploring American culture. While
students can enroll in the program for a semester, academic year, or summer session, most enroll
for one semester. Enrollment in the Chicago Center has been increasing since its inception in
2011. Fifty-one students are currently enrolled in the Chicago Center for Fall 2015. There are
25 Brazilian students, which account for the largest nationality group in the program. The Brazil
Scientific Mobility Undergraduate Program (BMSP), a scholarship program implemented by the
Brazilian government, explains the influx of Brazilian participants.
In addition to Brazilian students, there are 22 Chinese students. Chinese students have
historically remained a stronghold in program enrollment. Significant recruiting efforts have
been focused on University of International Business and Economics (UIBE) and Jilin Huaqiao
Foreign Language Institute (HUAWAI) in China. Additionally, there are four South Korean
students from Sogang University in Seoul, a Jesuit university with which LUC has a partnership.
Stakeholders and Organization
Stakeholders are various individuals and groups who have a direct interest in and may
be affected by the program being evaluated or the evaluations results (Fitzpatrick, Sanders, &
Worthen, 2011, p. 9). The primary stakeholders are those directly involved with the program
including the policy-makers, funders of the program, the program facilitators and staff, the
intended beneficiaries, and those who may be disadvantaged by the program. The secondary
stakeholders are those who have an indirect interest as what happens in the program affects them.
11
The primary stakeholders of the Chicago Center include: Jason Obin, Director of the
Chicago Center and the International House; Patrick Boyle, Vice Provost of Academic Centers
and Global Initiatives; and Michael Hines, instructor of the Chicago Center Seminar. Obin
reports directly to Boyle of the Office of the Provost. Boyle reports directly to the Interim
Provost, Dr. Samuel Attoh. LUC has recently experienced a change of leadership in the Office
of the Provost. The previous Provost, Dr. John Pelissero, was in full support of the Chicago
Center program; it is unclear if Attoh supports the program or not.
Jason Obin is the only full-time staff member who currently coordinates the Chicago
Center and the International House. He is assisted by numerous Graduate Assistants and Student
Living Assistants: Samantha Ng, International House Graduate Assistant; Rebecca KahnWitman, Chicago Center Graduate Assistant; Eddie Buggie, International House Student Living
Assistant; Taiwo Adefiyiju, International House Student Living Assistant; Tyler McLoughlin,
International House Student Living Assistant. These individuals coordinate and manage various
aspects of the Chicago Center and the International House like programming cultural events at
the International House and at various sites in the city of Chicago.
Although there is currently no direct report, the Chicago Center is associated with the
Office for International Programs (OIP) and collaborates with several OIP offices. There have
been discussions about integrating the Chicago Center into OIP, but the program will remain
under Boyle for the foreseeable future (P. Boyle, personal communication, September 11,
2015). The Chicago Center works and collaborates with several LUC departments: English
Language Learning Program (ELLP) for English as a Second Language (ESL) courses and
student activities; International Students & Scholars (ISS) for immigration and visa services; and
Academic Advising for international student advising. Indirect stakeholders include Beth Lair,
12
Associate Director of ELLP; Steven Fehr, Program Coordinator of ELLP; Tami Renner,
Associate Director of ISS; and Akeya Peterson, International Student Academic Advisor. These
individuals assist the Chicago Center students in essential aspects such as enrollment in
academic courses, ESL courses, and visa paperwork.
There are other secondary stakeholders, who are affected by the program but have no
direct involvement. Chicago Center students interact with American students in the classroom
and organizations, which may help American students develop intercultural skills. Furthermore,
Chicago Center students families and friends from their home country may be affected by the
experience and may develop growth in intercultural competency. Additionally, this ripple-out
effect touches both the American and the related international societies as students and those
with whom they interact may help foster intercultural understanding and acceptance.
Evaluation Rationale
The Chicago Center has not been evaluated previously. Patrick Boyle and Jason Obin
recognize a need to implement an evaluation (P. Boyle, personal communication, September 11,
2015). The Chicago Center has faced significant changes since 2011 such as a stark increase in
enrollment and the development of the International House. Enrollment has increased nearly 10
times since its debut in Fall 2011. Stakeholders intend to limit student enrollment in the program
to about 50 per semester (P. Boyle, personal communication, September 11, 2015). The
predominant focus of the Chicago Center currently is to diversify the represented countries in its
student population. Jason Obin will focus on recruiting in countries other than China and South
Korea (J. Obin, personal communication, September 24, 2015). Jason Obin recently attended the
European Association of International Conference in Scotland in an effort to recruit more
European students.
13
In addition to diversifying the student population, the stakeholders desire to recruit more
American students to reside in the International House (J. Obin, personal communication,
September 24, 2015). This has proven difficult since the International House is considered offcampus housing and is not within the umbrella of Residence Life at LUC. LUC students are
required to live in on-campus housing during their first and second year. This requirement
prevents American students from residing in the International House during those years.
An evaluation of the Chicago Center study abroad program for international students is
necessary in order to determine if its programmatic components provide a sufficiently high
quality academic immersion program that fosters intercultural competency in its students
(Deardorff, 2011). Beyond determining this developmental outcome, we also wish to evaluate if
there are any areas of the program that could be strengthened or enhanced. While there are many
aspects to running such a program, we are looking at three required components as well as the
two optional components that the center facilitates.
Logic Model
A logic model provides a way for the evaluator to begin to describe the program, its
components, and sequences (Fitzpatrick, Sanders, & Worthen, 2009, p. 296). It lays out the
situation, the intervening activities, and the outputs from these activities that lead to the desired
goal and can give some idea about assumed causal linkages between these components
(Fitzpatrick et al., 2009, p. 311). Besides bringing overall clarity to the situation, it also helps
point to what questions to ask in the assessment instrument. Thus, we have developed a logic
model to assist us in evaluating the Chicago Center (Appendix C).
14
Inputs
The inputs in this program are all of the stakeholders including administrators,
facilitators, class instructor, graduate student assistants and both international and American
students. The instructor for UNIV 110 makes all the arrangements for guest speakers as well as
site visits and relies on students public transportation U-passes for getting to the sites. In
addition, the Chicago Center director and graduate assistant makes all travel arrangements
including arranging travel, tour guides, hotels, and meals for the travel excursions to two major
U.S. cities. The budget and the International House are also important elements to this
program.
Outputs
The outputs are the interventions or the program components of the Chicago Center. The
international students are required to take a seminar focused on American culture through the
lens of Chicago, two excursion trips during the semester to other major U.S. cities, and live in
the International House. They also have options to participate in American and cross-cultural
events held at the International House and/or a voluntary Global Mentorship program with an
American mentor. The outputs also include the participants of each activity besides the
international students. The primary people in the Chicago seminar are the instructor and
graduate teaching assistant. They also have guest speakers for the class. The excursions involve
the center director, support staff, the graduate assistants, and local tour guides. The director, a
graduate assistant, and three Student Living Assistants, who all work with all of the students
living there including American students and international students participating in other
programs, support the International House. The graduate assistant is responsible for planning the
cultural events at the International House though the others may participate in these events. A
15
graduate assistant, who recruits American students to be mentors, manages the Global
Mentorship program.
Outcomes
We have three levels in our outcomes. The short-term outcomes are specific outcomes
we envision occurring as a result of each intervention. We anticipate two outcomes for both the
required seminar and the excursions. First is that students would observe the cultural character
of neighborhoods and several major cities. We also would anticipate them to display open
mindedness and cultural sensitivity by working in cross-cultural groups (see UNIV 110 syllabus,
Appendix A; Jackson, 2004). Residency in the International House with other American and
international students should help them synthesize intercultural awareness of American and other
national cultures with their own (Brandauer & Hovmand, 2013; Pedersen, 2010; Vande Berg,
Paige, & Lou, 2012). For the American and cross-cultural events at the International House,
participation in the events would further their knowledge, understanding, and respect for other
cultures as well as a pride in sharing their own cultural events (Brandauer & Hovmand, 2013;
Engle & Engle, 2004). The Global Mentorship program would be expected to expand their
awareness of culture and norms of communication in the United States (Brandauer & Hovmand,
2013; Deardorff, 2011; Jackson, 2004). This would include what to say and how to act in
particular situations (Watson, Siska, & Wolfel, 2013).
The medium-term outcomes are more general in nature and should be a natural
developmental outcome resulting from the short-term outcomes. As students complete their
immersion experience at the end of the semester, we expect that they would display growth, selfconfidence, and courage (Brandauer & Hovmand, 2013; Hotta & Ting-Toomey, 2013; Jackson,
2004). We also anticipate them demonstrating greater cross-cultural awareness and appreciation
16
with increased exposure to American culture and the various cultures represented in the
International House (Deardorff, 2011). These two outcomes would contribute to them exhibiting
intercultural competence through the ability to observe, respect, and adapt to cultural difference,
as described by Engle and Engle (2004) as the definition of intercultural competence.
The final long-term outcome is increased intercultural competency through a high quality
academic immersion program into American society. Intercultural competency is considered a
long-term outcome as we expect students to continue to increase or develop their intercultural
competency further after they leave the Chicago Center program. This would occur when they
return home, where we anticipate they will integrate what they have learned by living in another
culture into their everyday lives through new habits and attitudes. However, through exhibiting
attainment of the medium-term outcomes, students will have already accomplished the purpose
and goal of Chicago Center in providing a high quality academic immersion program into
American society as displayed through their increased growth and development in intercultural
competency.
Assumptions
We have several primary assumptions in developing this evaluation. The first
assumption is we believe the current elements of the program offered at the Chicago Center will
produce a successful immersion experience for the international students. We further assume
that the planned components for the immersion experience will be successful regardless of the
cultural background of the students and/or the cultural mix of students participating within any
given semester. Within this program, we are also taking for granted that the facilitators of the
various components are adequately trained and are providing the level of academic rigor
necessary for the program to be able to increase the students intercultural competency.
17
Admission into the program requires students to have a minimal score of 550 on the
paper and pencil version, 213 on the computer version, or 79 on the internet based (iBT) version
on the Test of English as a Foreign Language (TOEFL). Based on these scores, we trust that
their language skills are sufficient to fully engage in and comprehend all of the immersion
activities as well as our questions in our evaluation.
We presume that these students will be fully engaged in the activities offered and that
their journaling will be reflective along with drawing comparisons to their own national
culture. If students make minimal efforts and/or they spend all of their free time socializing with
others from their own country and speaking their own language, they would thwart a successful
immersion experience.
External Factors
External factors are elements beyond our control that could affect the results of the
evaluation. One factor is the background cultures of the international students. These students
are not from a homogenous culture. Depending on how similar or different they are to American
culture, it would influence the level of intercultural competence development the students could
expect to attain and influence the outcome of this evaluation (Hotta & Ting-Toomey, 2013).
Even though the international students pay a fee that covers tuition, program travel, room,
and board, it does not cover the full expense of the program. If the university were to reduce the
budget for the Chicago Center and/or the International House, it may require cancellation of
certain activities.
If there was any change in key personnel, namely the Chicago Center Director, the
instructor for UNIV 110, or the Vice Provost of Academic Centers and Global Initiatives, the
program and/or its specific activities could change. In addition, if there was a change in the
18
upper administration of Loyola University Chicago, such as the Provost or President, university
priorities could change. If global initiatives were no longer a priority, the Chicago Center
program could be eliminated in order to shift funds to new priorities.
There are also general external factors that could influence the international students
experience and therefore the assessment. Bad weather could cause another U.S. city and/or local
excursion to be cancelled. Extreme weather causing extensive physical damage to the university
could alter the program considerably for a particular semester, as accommodating adjustments
had to be made. Another element that could affect these students experience could be global
events, which could alter their views of one another thereby interrupting their intercultural
development.
Evaluation Approach
We are using a formative process and outcome evaluation approach. We will assess the
intercultural competency development of the international students in the Chicago Center
program to determine if the program is meeting the intended outcome of providing a high quality
academic immersion experience. We will also assess the process through examining the
students experiences of the individual components to determine the nature of delivery and the
successes and problems encountered as they relate to the students developmental outcome
(Fitzpatrick et al., 2011, p. 26). Since any decision made about the program resulting from this
evaluation will be to improve it, it will be a formative evaluation, which according to Fitzpatrick,
Sanders, and Worthen (2011) is the purpose of this evaluation approach. Fitzpatrick et al. further
stated that outcome assessments in a formative evaluation are concerned with describing,
exploring, or determining changes that occur in program recipients (p. 26). We will examine
the short-term and medium-term outcomes specifically of the five components used by the center
19
for the immersion experience to determine if each international student in the program has
progressed in developing intercultural competency. In addition, our intention is that the survey
results and information gathered from the students experiences through focus groups provide
feedback to the program director for ways to enhance the program further based on overall
findings.
Strengths and Weaknesses
One strength is that we will be able to survey most, if not all, the international students
each semester, as they are required to attend certain events and all of them have one class
together. This allows us the opportunity for hearing from almost everyone on their
experience. In addition, there are new groups of students each semester. If the center conducts
this evaluation each semester, the program can obtain rapid feedback from any improvements
made and gain cumulative results relatively quickly.
While the goal is to diversify the nationalities present each semester, the current group is
more than 50% Brazilian students, with only four students from South Korea, and the balance
from several areas of China. This is an anticipated weakness of our evaluation as these
background cultures are likely to have an impact as cultural similarities and dissimilarities to a
host culture affect intercultural competency development (Hotta & Ting-Toomey, 2013). With
so few different cultural backgrounds, there is the potential that the feedback may be skewed and
not lead to a well-developed and balanced program for future groups with different cultural
backgrounds.
Outcome Specifics
Our approach breaks the activities into two broad groups. The UNIV 110 seminar and the
two excursions are about a broad exposure to American culture as expressed through three major
20
U.S. cities with their museums, monuments, history, and culturally diverse
neighborhoods. Through observation, reflective journaling, and joint projects, we anticipate the
students will display open-mindedness and cultural sensitivity across cultures. At a more
personal level, the students are living among several different nationalities in the International
House, experiencing each others cultural events, and having an opportunity to engage with an
American student mentor. With these more personal experiences, we foresee that they will
expand their awareness of U.S. culture and norms of communication as well as their crosscultural appreciation. Additionally, we will examine how well they synthesized their
intercultural understanding between American culture and their own native culture.
These component-specific outcomes will lead to overarching outcomes as they move
further into their semester. It takes courage to interact in a foreign language and the related
culture. As students become more comfortable in the American culture and speaking English,
we anticipate their self-confidence and independence increasing as they begin initiating
conversations, participating in informal activities, and generally being more adventurous
(Jackson, 2004). Thus, we would expect to see growth in independence, self-confidence, and
courage as supported by study abroad immersion studies of Korean students in the United
Kingdom and American students in Denmark (Brandauer & Hovmand, 2013; Jackson,
2004). As they demonstrate greater cross-cultural awareness and appreciation, they would
exhibit intercultural competence through the ability to observe, respect, and adapt to cultural
difference (Engle & Engle, 2004, p. 231). These outcomes make up the definitions of
intercultural competency and point to the accomplishment of the program goal of a high quality
academic immersion into American society. We do not expect that everyone will develop at the
same speed or to the same level. Students develop individually based on their background,
21
stimulus, and own level of personal development. However, seeing developmental progress still
meets the desired goal.
Quantitative Approach
With the intention to evaluate the effectiveness of the programmatic components in
providing a high quality academic immersion experience that fosters intercultural competency, a
quantitative methods approach has been strategically structured in order to best align with the
purpose of the evaluation of the Chicago Center. With a clear and concise quantitative approach,
an evaluation of the Chicago Center will provide enlightening information in how students
experience growth over the course of the semester.
Research Design
In order to assess process and outcomes of the Chicago Center, we designed the approach
as a pre-experimental longitudinal study. A pre-experimental design is best suited to our study
since it studies a single group and provides an intervention during the experiment (Creswell,
2009, p. 156). The Chicago Center provides an educational intervention or treatment to shortterm international students during their semester abroad at LUC. The three required components
of the Chicago Center are residency at the International House, enrollment in UNIV 110, and
participation in two excursions to major U.S. cities. The optional components included are the
cross-cultural events sponsored by the International House and the Global Mentorship program.
For this evaluation, the Chicago Center program components as a whole are considered the
intervention.
Creswell (2009) also noted that a pre-experimental design does not have a control group
to compare with the experimental group (p. 158). A pre-experimental or longitudinal design
requires that the same group be tested over a time. We have chosen this design because we
22
specifically want to see the students development in intercultural competency. Unlike our
research design, other designs such as quasi-experimental and true experimental designs require
a control group (Creswell, 2009). While using a control group was an option if we wanted a
different experimental design, we decided against it as not being feasible given the resources and
time constraints.
This pre-experimental longitudinal research design intends to survey all Chicago Center
students enrolled for a semester of study. We will administer census sampling since we intend to
survey the entire population (Wholey et al., 2010). Census sampling is best used with small
populations, which aligns well with Chicago Center enrollments. Stakeholders such as Patrick
Boyle do not want to expand the Chicago Center beyond 50 students in future semesters (P.
Boyle, personal communication, September 11, 2015). Chicago Center students will be assessed
during pre-semester orientation and the last class session of UNIV 110, which reflects the
longitudinal nature of the research design. While participation in the surveys is voluntary, it will
be integrated into orientation and UNIV 110 materials. We anticipate 100% response rate since
the pre-test and post-test surveys will be administered when their attendance is
required. However, there is a chance that there will be a lower response rate due to issues such
as attrition.
Survey Instrument
We created localized pre-test and post-test surveys for the target audience of the Chicago
Center in order to assess the effect of the programmatic components over the course of a
semester. The pre-test and post-test surveys utilize Likert scale and multiple-choice
questions. The survey explores various aspects of the program such as the component process
and developmental outcomes. As demonstrated in the survey matrix, many questions are rooted
23
in the logic model (Appendix C). The logic model concisely indicates intended short-term,
medium-term, and long-term developmental outcomes of the Chicago Center program. The
Global Perspective Inventory (GPI) has provided a framework in production of our pre-test and
post-test surveys (Braskamp, Braskamp, & Engberg, 2014). GPI was in part created in order to
investigate whether changes in the way students view their knowing, their sense of self and
relationships with others after studying abroad for a semester (Braskamp et al., 2014, p.
15). GPI is especially useful since the Chicago Center seeks to provide an education abroad
experience for short-term international students in the U.S. The Chicago Center has not
implemented any surveys to investigate whether the programmatic components as a whole are
encouraging development for participating students though students do complete the regular
course evaluation for UNIV 110.
The pre-test survey and the post-test survey differ in various ways as noted in the survey
matrix (Appendix D); however, the content of many of the questions remain the same. The pretest survey (Appendix E) is composed of eight Likert scale questions with multiple parts and five
multiple-choice questions. The pre-test survey will take approximately seven minutes to
complete. There is brief introduction that frames the context of the survey, followed by a request
to provide their LUC Student Identification number. Their Student Identification number will be
useful in matching pre-test and post-test surveys to the same individual. Within the pre-test
survey, students are asked about their perceptions of English skills, living in the U.S.,
intercultural skills, interacting with Americans, cross-cultural interactions, and participation in
program components. These are posed as Likert scale questions and consist of either four-point
or five-point scale responses. Though disputed amongst scholars, Wholey et al. (2010)
suggested that four-point scales motivate students not to respond neutrally, so we used a
24
combination of four-point and five-point scale responses. These Likert scale questions reflect
the short, medium, and long-term goals of the Chicago Center program. The multiple-choice
questions investigate prior visits to the U.S.; and whether credits transfer to their home
institution. The final three multiple-choice questions ask what region of the world the student is
from; where their roommate is from; and their gender identification. These demographic
questions are limited to region rather than specific country. We intentionally decided to use
regions in order to preserve the confidentiality of the student. For example, in certain semesters
there has only been one student from one specific country. We recognize the limitations in
asking for region, but hope that this will secure student confidentiality.
The post-test survey (Appendix F) is composed of 12 Likert scale questions with multiple
parts and five multiple-choice questions. The post-test survey will take approximately 10
minutes to complete. Similarly, to the pre-test survey, the post-test survey questions reflect the
short, medium, and long-term goals of the Chicago Center program. Questions about their
perceptions of English skills, living in the U.S., intercultural skills, interacting with Americans,
cross-cultural interactions, and participation in program components are included in the post-test
survey. These questions are repeated in pre-test and post-test in order to assess growth in these
areas after completion of the intervention. In addition to identical questions, we developed
unique questions specifically for the post-test survey. These unique questions aim to investigate
student experience in UNIV 110, U.S. city excursions, residency at the International House,
cross-cultural events, and the Global Mentorship program. There is also an emphasis on
reflection of the Chicago Center program as a whole. These questions will provide insight into
student experience upon completion of programmatic components. The multiple-choice
questions explore social interactions and communication with family and friends from their
25
home country. Demographic questions that were included in the pre-test survey have been
excluded from the post-test survey since their LUC Student Identification number will match
students pre-tests and post-tests.
Procedures and implementation
Anne Conlon, co-writer of this evaluation and assistant moderator, will distribute both the
pre-test and post-test surveys on paper. Paper rather than web surveys will be used as students
are so new to campus at the time of orientation and may not have registered their computers with
the university in order to have internet access yet. Participation in the survey will be integrated
into the programmatic structure of the Chicago Center. Although participation in the survey is
not mandatory, participation is expected. The pre-test survey will be distributed before the start
of the semester, and the post-test survey will be distributed during the last class session of UNIV
110. Creswell (2009) noted that one can bolster the response rate by taking advantage of
captive audiences (p. 271). Both the orientation and the last session of UNIV 110 are
mandatory, which lends to a captive audience.
Pre-test surveys will be distributed at Chicago Center orientation before the start of Fall
or Spring semester in order to best avoid influence by programmatic interventions. For instance,
Chicago Center students are required to participate in the first excursion of the Fall semester in
mid-August before semester courses start. In addition, student residency at the International
House is considered as a programmatic intervention. By implementing the pre-test survey during
orientation, there is less of a chance that the student will have experienced programmatic
interventions. In the event that a student is unable to attend orientation, there is a makeup
session available where the pre-test survey may be distributed.
26
Post-test surveys will be distributed during the last session of the required course, UNIV
110. Chicago Center students are required to attend this final session of UNIV 110 in order to
present their final projects. Anne Conlon will distribute the post-test surveys in order to remain
consistent and will be listed as the main contact for the evaluation.
Pilot tests will be administered to a small group of Chicago Center students the semester
prior to full implementation. Doing it the prior semester allows us access to the same student
population and the same timing for administering the pilot tests. One of our main concerns in
constructing the pre-test and post-test surveys is the word choice. Since our target audience
comprises international students, it is important to consider the varying levels of English
comprehension. In addition, there may be words used in the surveys that are confusing to ESL
students. Chicago Center students, through the pilot testing, will be able to provide feedback on
the wording of the surveys, which could be a great assistance in modifying the pre-test and posttest surveys.
Validity and Reliability
We anticipate two possible validity/reliability issues with the assessment. The first is in
the pretest. When international students first arrive, they may overestimate their intercultural
competency, as they have not yet really experienced the immersion, often referred to as the
ceiling effect. We will try to address this by asking the students in the post-test to think back
and indicate how they were at the beginning of the semester as well as currently on the same
questions. Another possible weakness is testing bias, which is that participants remember their
responses from the pre-test when completing the post-test and may indicate improvement
because it is expected (Creswell, 2009; Porter, 2011; Schuh et al., 2009). To address this, the
27
instructions stress that we are evaluating the program and not their personal progress and stress
the importance of answering the questions honestly.
If attrition occurs during the semester, it could skew the results also. If a student
completes the pre-test survey but does not complete the post-test survey, there will be incomplete
data. This could be especially damaging if the student represents a group with lower overall
cultural competency, since the post-test survey demonstrates growth experienced after the
intervention.
Analysis
The purpose of the quantitative portion of this evaluation is to understand process and
developmental outcomes as a result of participation in the Chicago Center program. Paired
sample t-test and a cross-tabulation with a simple chi-square test analysis will be utilized to
evaluate the survey data. Survey data will be entered into statistical software called Statistical
Package for the Social Sciences (SPSS; Schuh et al., 2009). SPSS is noted as particularly welldesigned for the analysis of survey data (Schuh et al., 2009, p. 157). Once entered in SPSS,
descriptive statistics will measure mean, median, and mode, which will show where students as a
group fell on the scale for each outcome and overall as well as showing growth or improvement
when comparing the pre-test and post-test measures. Frequency statistics will represent student
population demographics such as the students region of origin and gender. Both descriptive and
frequency statistics will demonstrate how the target audience responded as a whole to the
surveys (Huck, 1974; Schuh et al., 2009).
The paired sample t-test will demonstrate whether there is a significant change in process
and developmental outcomes. By implementing the paired sample t-test, it will be easy to
determine which process and developmental outcomes garnered significant growth from pre-test
28
to post-test. This will help determine whether there is significant growth and the most impactful
components of the Chicago Center program. Significance is determined based on the mean
differences when the p-value is less than .05 as we would be able to reject the null hypothesis
that there is no significant difference between the pre-test and post-test.
A cross-tabulation and a simple chi-square test will assist in conducting a deeper analysis
of data. The cross-tabulation allows independent categorical variables such as region of origin
and gender to be compared to specific processes or developmental outcomes to see if there is a
relationship. Schuh et al. (2009) noted that cross-tabulation is the simplest, but still effective,
method for analyzing the relationship between discrete variables (p. 153). The chi-square test
will provide the opportunity to investigate the region of origin as well as by gender to see
whether there are significant differences amongst student experiences, which could influence
future adjustments to the program (Schuh et al., 2009). Another comparable option is to use an
analysis of variance (ANOVA) test, which allows us to compare mean levels of a continuous
dependent variable across the levels of a main effect (a categorical variable) (Schuh et al.,
2009, p. 148). With our Likert scale questions functioning in a continuous manner and our
independent categorical variables having three or more sub-categories with each variable,
ANOVA could be used to see if there is a significant difference between the sub-categories. If
we discovered significant differences, we would run a Tukey post-hoc test to determine between
which groups the significant difference existed and how large it was (Engberg, 2015a). This
could also be particularly useful when looking at the regions of origin given the cultural
differences that could be influencing it.
29
Qualitative Approach
In addition to the quantitative approach, a qualitative approach will further deepen the
evaluation of the Chicago Center program. According to Wholey et al. (2010), qualitative data
allows understanding that goes beyond numbers and statistical inference (p. 429). A
qualitative approach will add critical depth to the evaluation in addition to the important data
received in the quantitative approach. As detailed in the following sections, the qualitative
approach envisions implementation of a focus group in order to better understand how Chicago
Center students make meaning of their immersion experience during their semester in
Chicago. According to Schuh and Associates (2009), the goal of the focus group is to explore
peoples perceptions and feelings (p. 92). The focus group will follow themes covered in the
logic model (Appendix C) similar to our pre-test and post-test survey questions; however, the
focus groups allow for a specific focus on process-related questions, which were difficult to
incorporate by the structure of the surveys. By implementing a focus group, we will be able to
better explore critical aspects of the Chicago Center program in students own words.
Focus Group Participants
The target population for the focus group will consist of Chicago Center students who
completed the pre-test survey. The Chicago Center student population consists of a diverse array
of students from different origins, academic disciplines, and cultures. With such a unique
population, implementing a focus group will provide deeper insight to programmatic components
experienced across various represented cultures.
Evaluators should aim to acquire six to eight students for each focus group. Six to eight
participants aligns with recommended practice for noncommercial topics according to Schuh et
al. (2009) though there is some difference of opinion among scholars. Two focus groups should
30
be conducted during each evaluation period. While it is recommended to hold enough focus
groups until redundancy is reached or no further insights are gained, we are limiting the number
of focus groups to allow for response rates in this small population size and the limited resources
of the Chicago Center (Krueger & Casey, 2010). Incorporating a focus group into the Chicago
Center structure each semester will allow helpful feedback for evaluators and stakeholders.
All enrolled Chicago Center students will be invited to participate in the focus group. In
order to reach Chicago Center students, we will announce the opportunity to participate in a
focus group several weeks prior to implementation in UNIV 110. We will also send a
personalized email (Appendix G) to all enrolled students who completed the pre-test
survey. Maximum variation sampling will be implemented so the focus group benefits from not
only the typical students, but also the students who are adapting extremely well and not so
well. This strategy is ideal with small samples as it allows us to see the diversity of the full
group within the smaller sample (Engberg, 2015). The instructor for UNIV 110 will help
determine which participants would give us a maximum variation of those who volunteered. If
the full range of students did not volunteer, appropriate students will be approached directly and
encouraged to participate.
Students will be incentivized to participate in a focus group with a $15 gift card to the
Loyola University Bookstore as well as a pizza dinner. According to Schuh et al. (2009),
incentives increase the perceived reward, minimize the perceived cost, and increase the trust
(p. 98). Additionally, providing an incentive has been found to slightly increase the response
rate (Schuh et al., 2009). The email invitation highlights the incentives of participation as well
as the purpose for conducting a focus group. Email recipients are notified that the focus group is
an opportunity to share their immersion experiences in the Chicago Center program (Appendix
31
G). Once a student replies YES to the email, subsequent emails will be sent as reminders for
participation if they are selected. Otherwise, they will receive an email thanking them for
considering participation.
Focus Group Procedures
The focus group will be guided by the focus group protocol (Appendix I). Moderator,
Hongying Chen, will guide the focus group with assistance of the assistant moderator, Anne
Conlon. The choice in moderators will be further detailed in the following section. First,
participants will arrive and will be invited to grab some pizza and engage in informal
conversation to put them at ease. Once settled, the moderator will give a brief introduction and
explanation of the purpose of the focus group. Anne Conlon will pass a consent form to each
student that details the purpose, procedures, and confidentiality information (Appendix
H). Students will then be asked to introduce themselves and share their name, country of origin,
academic discipline, and home university. Although many of the students may already know
each other, this allows students to become more familiar with each other.
The focus group protocol consists of nine questions, each of which includes follow-up
probing questions. Each question in the focus group protocol is designed to deepen
understanding of student experiences throughout the various components and activities of the
Chicago Center program. The focus group protocol consists of five segments, which investigate
student experiences living in the International House, travel excursions to major U.S. cities,
UNIV 110, cross-cultural interactions, and interactions with Americans. These five segments
assist in achieving the overall goal of the focus group, which intends to better understand how
students make meaning of their experiences through the various Chicago Center components
(Schuh, 2009, p. 130). In addition to the five segments, the probing questions in each segment
32
should elicit thoughtful responses. Hongying Chen will be cognizant of the dominant voices of
the group and encourage quieter students to share their experiences.
Anne Conlon, the assistant moderator, will be responsible for digital audio recording the
content of the focus group. In addition to audio recording, Anne Conlon will take notes on the
process during the focus group, which may elicit important information on group dynamics that
would otherwise be unknown in the recording. Digital audio recording will be convenient during
the focus group and relatively unobtrusive for participants.
Focus Group Moderators
Hongying Chen, a psychologist at LUCs Wellness Center, will be the focus group
moderator. At the Chicago Center orientation, Hongying Chen presents on culture shock and
other issues associated with living in a foreign country. By having a moderator from outside of
the Chicago Center program, students may feel more open in sharing honest opinions during the
focus group. Furthermore, students may feel more comfortable with Hongying Chen since she is
originally from China.
In addition to Hongying Chen as the moderator, a co-author of this evaluation, Anne
Conlon, will act as an assistant moderator in order to oversee various aspects of the focus
group. As Administrative Assistant to Patrick Boyle, a primary stakeholder, Anne Conlon has a
particular advantage in assisting in the evaluation of the Chicago Center. She does not have any
sort of authoritative role like other graduate assistants associated with the Chicago
Center. Chicago Center students may know Anne Conlon since she acts as staff support on
excursions to major U.S. cities. As a co-author of the evaluation, Anne Conlon also has a
particular advantage in knowing and understanding the evaluation plan and approach.
33
Implementation Plan
The focus group will be scheduled during the final month of Fall and Spring
semesters. This allows students to have participated in many of the programmatic components
such as the two excursions to major U.S. cities, the majority of UNIV 110, and several months
experience of living in the International House. By situating the focus group during the final
month of the semester, there is a better chance for a higher participation rate than if it were
scheduled after the end of the semester. Many Chicago Center students leave immediately for
travel back to their country of origin after the end of the semester (E. Buggie, personal
communication, October 13, 2015).
The focus group will last approximately 90 minutes, will start at 6:00 p.m., and provides
students with pizza and drinks. Choosing a time in the late afternoon will be more convenient
for students since undergraduate courses generally occur during the day. Additionally, the focus
group will take place in the private staff room of the International House. This room provides
adequate seating and is convenient for Chicago Center students. This location will be familiar
and comfortable for students since they reside in the International House.
A pilot test will be conducted the semester prior to full implementation of the evaluation
with a small group of Chicago Center students. Since the students only stay one semester, this
will not affect the limited pool of students for the actual evaluation. By completing a pilot test,
moderators will gain experience in leading the focus group. It will also demonstrate the
effectiveness of the focus group protocol (Appendix I).
Focus Group Analysis
Analysis will occur through a process of transcription, descriptive coding, and data
analysis. Anne Conlon and the International House Graduate Assistant will transcribe the digital
34
audio recording. The International House Graduate Assistant will be able to offer additional
assistance in the evaluation process. The transcription process will be less time-consuming with
two individuals rather than one.
Descriptive coding will be completed in Microsoft Word looking for key words in the
proper context. Both Anne Conlon and the International House Graduate Assistant will code the
focus group information separately, which is a way of testing inter-rater reliability. Testing
inter-rater reliability is a way of checking the interpretations of data (Rogers & Goodrick,
2010, p. 446). A coding rubric (Appendix K) was developed to assist in the coding procedure of
the evaluation. Descriptive coding will be utilized in the coding procedure, which identifies
topics or issues in the text (Rogers & Goodrick, 2010). Codes have been developed to represent
major anticipated themes such as cross-cultural experiences (CROSS) and experiences with
American culture (AMCULT). Eight codes are listed in the coding rubric, and will greatly assist
Anne Conlon and the graduate assistant in transcribing the large amount of data. Additionally, a
coding rubric will be essential in maintaining organization during this procedure.
After the focus groups are transcribed and coded, the data will be ready for further
analysis. Throughout the coding process, there is a chance that other major themes will arise that
were unanticipated on the coding rubric (Appendix K). If this is the case, it is important to note
these major themes as well as opinions that do not fall into a specific category.
Validity and Reliability
In order to ensure validity, triangulation, a method meant to overcome the potential bias
that can arise from the use of a single method will be implemented (Rogers & Goodrick, 2010,
p. 446). In order to implement triangulation, data will be triangulated from the pre-test survey
and the post-test survey. By implementing triangulation, the evaluation will have a more holistic
35
understanding of the results uncovered. Triangulation will also find points of both convergence
and divergence.
In addition, using multiple evaluators like Anne Conlon and the International House
Graduate Assistant will assist in ensuring further validity. There is a possibility that evaluators
will have a personal bias when transcribing and coding data. Anne Conlon especially has a
vested interest in the Chicago Center since she works directly for Patrick Boyle. However,
having two individuals code will help to identify any discrepancies during the coding process
(Rogers & Goodrick, 2010, p. 445). Additionally, member checking will be implemented, which
checks the interpretations of data with participants (Rogers & Goodrick, 2010). The participants
will be contacted via email for member checking once the themes have emerged from the
analysis.
Furthermore, there is an inherent limitation in-group composition depending on who
volunteers to participate in the focus group. Maximum variation is ideal within each focus
group. However, it is possible that students who are not adapting well will not be interested in
participating, even though the focus group could provide vital information for improving the
program. It is also possible that the moderator inadvertently asks a leading question even though
the provided probes in the protocol are not leading.
In order to prevent bias, particular emphasis was placed on selecting an appropriate
moderator and assistant. The Chicago Center has limited staff, so Hongying Chen and Anne
Conlon are appropriate choices since they are not directly involved in the program.
Timeline
A timeline has been developed to ensure proper implementation of the evaluation
(Appendix L). The timeline includes pilot testing for the Spring 2016 semester, with the intent
36
of implementing a complete phase of the evaluation starting in Fall 2016. Through pilot testing
in Spring 2016, the evaluators will be able to make adjustments to any details before complete
implementation in Fall 2016. The summer will allow time to make the proper adjustments to the
evaluation plan, if needed. The evaluation has been prepared to be implemented each semester
beginning in Fall 2016; however, the timeline only details the evaluation plan through January
2017. A similar timeline could easily be replicated each semester, with dates being adjusted
accordingly. The timeline will need to be strictly adhered to; otherwise, there may be a delay in
preparing the final report for the Chicago Center and the Office of the Provost.
Budget
A budget has been developed with estimates for complete implementation of pilot testing
and semester evaluation for Spring 2016 and Fall 2016 (Appendix M). It is essential to remain
transparent in the costs associated with conducting an evaluation so that the Chicago Center can
budget accordingly. The budget includes paper printing associated with the evaluation, such as
pre-test and post-test surveys, and the final, spiral-bound report that will be presented to
stakeholders at the end of the evaluation. Additionally, costs associated with focus group
implementation are included. In order to create an incentive for students to participate in the
focus group, $15 gift card to the Loyola Bookstore as well as costs associated with a pizza dinner
are also included. The focus group costs are where the bulk of the costs associated with the
evaluation are focused. Adjustments may be made in future implementations of the
evaluation. The evaluation budget benefits greatly from its association with LUC. The audiorecording device, SPSS software, and room reservation at the International House are all free of
charge. Furthermore, LUC staff and graduate assistants will be transcribing and coding the
evaluation; therefore, no additional staff will need to be hired as a result of the evaluation.
37
Reporting
The findings from both the quantitative and qualitative portions of the assessment will be
presented in a written report that includes summarization of results in both an executive
summary as well as a more detailed narrative format. The content of the quantitative portion of
the written report will compare the results from the pre-test and post-test surveys, discuss
demographics of the sampling, and responses across regions of origin, as well as indicate if the
level of student growth was significant. Pie charts for frequencies and bar graphs for Likert scale
questions will provide visual representation of the results throughout the narrative. The five
major segments included in the focus group protocol (Appendix I) will organize the qualitative
portion of the narrative. Direct quotations from focus group participants will be included in this
report. It will be important to include contrasting opinions, which many not fall into a specific
segment. A construct map will be developed to clearly show the connection between the codes
and the five segments (Appendix J). The executive summary will be presented to the Office of
the Provost and the narrative report to the Director of the Chicago Center.
Next Steps
By implementing this evaluation plan for the Chicago Center program, there are
important next steps to consider moving forward. The intent of this evaluation is to provide
insight into how the programmatic components are effective or ineffective. On receipt of the
final report, it is essential to begin conversations concerning potential program improvements.
Since the evaluation approach is organized on a semester-basis, it should be relatively easy to
implement changes and perform subsequent evaluation. This will assist the Chicago Center in
continued development and evolution since its inception in 2011.
38
Since this evaluation is broad in assessing all programmatic components of the Chicago
Center, there is an opportunity to narrow the evaluation in subsequent evaluations. For example,
if a particular component of the program were assessed as ineffective, it would be wise to
investigate further by a more narrow evaluation on that particular programmatic component.
Additionally, there is an opportunity to expand the evaluation to a quasi-experimental or
true experimental design by considering the experiences of Chicago Center students with ELLP
students as a control group to determine whether or not the Chicago Center program is effective
in fostering development. ELLP students share similar characteristics to Chicago Center
students such as being short-term international students but do not undergo the same
programmatic components that are included in the Chicago Center program. This would be a
summative evaluation that would determine whether or not the programmatic components are
more effective in development for international students.
Overall, this evaluation plan provides an opportunity for the Chicago Center to continue
its innovative development. The Chicago Center is a unique program, which considers the
development and immersion of international students on an American campus. Watson, Siska,
and Wolfel (2013) indicated pre-immersion and post-immersion testing adds academic rigor to
the study abroad process and helps students quantify and discuss the benefits of their learning
experiences (p. 71). The program can only benefit from undergoing assessment and the
continued development of this study abroad program for international students.
39
References
Brandauer, S. C., & Hovmand, S. (2013). Preparing business students for the global workplace
through study abroad. Journal of International Education in Business, 6(2), 107-121.
doi:http://dx.doi.org/10.1108/JIEB-05-2013-0018
Braskamp, L. A., Braskamp, D. C., & Engberg, M. E. (2014, August). Global Perspective
Inventory (GPI): Its Purpose, Construction, Potential Uses, and Psychometric
Characteristics. Retrieved from GPI: https://gpi.central.edu/supportDocs/manual.pdf
Chicago Center (n.d.). Retrieved from http://www.luc.edu/oip/chicagocenter/
Creswell, J.W. (2009). Research design: Qualitative, quantitative, and mixed method
approaches. Thousand Oaks, CA: Sage Publications.
Deardorff, D. K. (2011). Assessing intercultural competence. New Directions for Institutional
Research, 2011(149), 65-79.
Engberg, M.E. (2015a). ANOVA test and post-hoc tests (PowerPoint slides). Higher Education,
Loyola University Chicago.
Engberg, M.E. (2015b). Sampling approaches (class handout). Higher Education, Loyola
University Chicago.
Engberg, M.E., Jourian, T.J., & Davidson, L.M. (2015). The mediating role of intercultural
wonderment: connecting programmatic components to global outcomes in study abroad.
Higher Education: The International Journal of Higher Education Research, 1-17.
Engle, L., & Engle, J. (2004). Assessing Language Acquisition and Intercultural Sensitivity
Development in Relation to Study Abroad Program Design. Frontiers: The
interdisciplinary journal of study abroad, 10, 219-236.
40
Fitzpatrick, J., Sanders, J., & Worthen, B. (2011). Program evaluation: Alternative approaches
and practical guidelines (4th Ed.) New York: Longman.
Glass, C. R., Buus, S., & Braskamp, L. A. (2013). Uneven experiences: whats missing and what
matters for todays international students. Chicago, IL: Global Perspective Institute, Inc.
Hotta, Jean, & Ting-Toomey, Stella. (2013). Intercultural adjustment and friendship dialectics in
international students: A qualitative study. International Journal of Intercultural
Relations, 37(5), 550-566.
Huck, S. W., Cormier, W. H., & Bounds, W. G. (1974). Reading statistics and research (pp. 74102). New York: Harper & Row.
Jackson, J. (2004). Language and cultural immersion: An ethnographic case study. RELC
Journal, 35(3), 261-279.
Kolb, D.A. (1984). Experiential learning: experience as the source of learning and
development. Prentice-Hall, Englewood Cliffs, NJ.
Krueger, R. A., & Casey, M. (2010). Focus Group Interviewing. In J. S. Wholey, H. P. Hatry, &
K. E. Newcomer (Eds.), Handbook of Practical Program Evaluation (3rd ed.; pp. 378402). San Francisco, CA: Jossey-Bass.
Pedersen, P.J. (2010). Assessing intercultural effectiveness outcomes in a year-long study abroad
program. International Journal of Intercultural Relations, 34(1), 70.
Porter, S. R. (2011). Do college student surveys have any validity?. The Review of Higher
Education, 35(1), 45-76.
Rogers, P. J., & Delwyn, G. (2010). Qualitative Data Analysis. In J. S. Wholey, H. P. Hatry, &
K. E. Newcomer (Eds.), Handbook of Practical Program Evaluation (3rd ed.; pp. 429453). San Francisco, CA: Jossey-Bass.
41
Schuh, J. H. & Associates (2009). Assessment methods for student affairs. San Francisco, CA:
Jossey-Bass.
Vande Berg, M., Conner-Linden, J., & Paige R.M. (2009). The Georgetown Consortium project:
interventions for student learning abroad. Frontiers: The Interdisciplinary Journal of
Study Abroad, 18, 1-75.
Vande Berg, M., Paige, R., & Lou, K. (2012). Student learning abroad : What our students are
learning, what they're not, and what we can do about it (First ed.). Sterling, Virginia:
Stylus Publishing, LLC.
Watson, J. R., Siska, P., & Wolfel, R. L. (2013). Assessing Gains in Language Proficiency,
Cross-Cultural Competence, and Regional Awareness During Study Abroad: A
Preliminary Study. Foreign Language Annals, 46(1), 62-79. doi:10.1111/flan.12016
Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.) (2010). Handbook of Practical Program
Evaluation (3rd ed.). San Francisco, CA: Jossey-Bass.
42
Appendix A: UNIV 110 Syllabus

LOYOLA UNIVERSITY CHICAGO
UNIV 110.001: Exploring the United States through Chicago
Location: Room 105, Corboy Law Center, Water Tower Campus
Instructor: Michael Hines
Phone: (815) 275-7834
Email: mhines2@luc.edu
FALL TERM 2015
Time: Fridays 4:15p.m.-6:30p.m.

T.A.: Rebecca Kahn Witman
Phone: (215) 264-5616
Email: rkahnwitman@luc.edu
Office Hours: by appointment

Course Description
The city of Chicago is a place of diverse cultures, rich history, and vibrant urban life. Using the city as a lens, this
course seeks to explore a wide range of contemporary American issues, including politics, history, race, class, and
culture by examining how they play out in the Windy City. Chicago Center students will strengthen their
speaking, writing, analysis, and presentation skills through the course content, which includes formal lectures, guest
speakers, and a number of outings that allow us to move beyond the campus and explore the sights and sounds of
Chicago.
Learning Outcomes
The following represent the objectives or learning outcomes we will accomplish throughout the course:
Students will identify key events, people, and places that have impacted the growth and development of the
city of Chicago.
Students will strengthen their presentation skills by presenting a short analysis of an article from a local
news outlet.
Students will strengthen their written communication through journaling and editing assignments.
Students will apply basic research techniques in order to complete a collaborative project focusing on the
history and culture of one of Chicagos historic neighborhoods.
Students will display an attitude of open mindedness and cultural sensitivity by working in cross cultural
groups throughout the semester.
IDEA Standards
The following represent the IDEA standards chosen for this course. These standards will be assessed at the end of
the semester through the course evaluation, which will help determine the progress we made in achieving our
learning outcomes.
Gaining factual knowledge (terminology, classifications, methods, trends).

Developing skill in expressing myself orally and in writing.
Acquiring skills in working with others as members of a team.
Gaining a broader understanding and appreciation of intercultural activity.
43
Required Materials
Please bring writing utensils and a notebook to every class, as well as a folder to keep any important information
you receive. There are no required texts, as all course readings will be made available via Sakai, but it is the
responsibility of the student to use Sakai to manage and print these materials.
Assignments
Students will be graded on a number of different assessments that seek to build their skills in writing, editing, and
presenting.
Forums: 25%
In the News Presentation: 10%
Midterm Assignment (Chicago Neighborhood Project): 25%
Final Presentation: 25%
Participation: 15%
Total: 100%
Assignment Descriptions
Forums: Forums are 5 small assignments given throughout the semester. For each you will log into the course site
for UNIV 110 on Sakai, then find the forums label, located on the menu running down the left of the screen. Once in
there you will select that weeks forum. I will have posted topics with questions or prompts for you to respond to
under each. You should be ready to:
-Pick one of the topics listed under that weeks forum.
-Read what Ive written.
-Write 250-500 words in which you answer the questions for that topic.
-Lastly, take the time to read a response from another classmate. Once you are done write 50-100 words
tell them what you think about their response (Did you agree? Disagree? Do you want to add something
to what theyve said? Does their response make you change your mind or shed new light on the issue?)
In the News Presentation: An individual presentation. Each student will sign up to bring in an article related to a
particular topic covered in class. Students will explain the content of the article, provide a visual aid (either ppt. or
Prezi), and answer questions from the class about the topic.
Midterm Assignment (Chicago Neighborhood Project): Group project group centering on a neighborhood in
Chicago. Groups will create a short research paper describing the neighborhood, as well as a video guide
showcasing important places within the neighborhood to show to the class.
Final Presentation: Each student will complete a final 5 minute speech at the end of the semester in order to reflect
and share their thoughts and reactions to their experience in the U.S.. This assignment will center on choosing an
object from the semester that best represents some aspect of their time here, and explaining what the object is and its
significance.
Participation: Composite score based on attendance (including timeliness), and willingness to take part in class
discussions and activities. Students who miss more than 1 class without being excused by the instructor will have
their grade lowered a half a letter (Ex. A to A-) each time.
Grading Scale
Assignments will be graded based on the following scale:
A= 92-100
B= 84-91
C= 76-83
D= 67-76
F= 66-below
44
Class Expectations
Students are required to complete all assignments by the due date. Late assignments will be assessed a penalty of
one letter grade per day unless otherwise arranged with the instructor.
Attendance is expected at every class. If you are unable to attend a class meeting or outing for any reason, it is your
responsibility to notify the instructor beforehand, explain your situation, and ask that you be excused.
Since this course is built on forming a community of learners, your participation and input are absolutely key to your
learning as well as the success of the class as a whole. You are expected to come to class each week prepared with
any readings or assignments, and ready to take an active part in class and small group discussions. In order to create
an environment where everyones voice can be appreciated and heard, cell phones must be silenced and away, and
laptops must be closed during class unless used specifically for note taking. Failure to appropriately use technology
will affect your participation grade.
Schedule
The following is a schedule for each week with important assignments and locations. Any changes to this schedule
will be communicated through email and sent far in advance of the class date.
Aug 28 : Introduction to Course / Chicagos History
th
Sept 5 : Field Trip: Chicago History Museum

th
Sept 11 : Chicago Neighborhoods / Introduction to Neighborhood Project (F1 due)

th
Sept 18 : Chicagos Cultural Scene: Literature, Art, and Architecture

th
Sept 26 : Field Trip: Art Institute of Chicago

th
Oct 2 : Schools and the City: Education in Chicago

nd
(F2 due)
Oct 9 : Green Chicago: Urban Space and Environment in Chicago

th
Oct 17 : Field Trip: LUREC

th
Oct 23 : Chicago Eats!!! Food and Culture in the Windy City

rd
(F3 due)
Oct 30 : Midterms Presentations (Chicago Neighborhood Projects due)

th
Nov 6 : Field Trip: Taste Bud Tour

th
Nov 13 : Big City Bosses: Politics in Chicago*

th
(F4 due)
Nov 20 : The Price of Doing Business: Entrepreneurship in Chicago

th
Nov. 27 : NO CLASS!!! THANKSGIVING!!!

th
Dec 4 : The Chicago Blues: Music and the Soul of the City *
th
Dec 11 : Final Presentations (Final Due)

th
*indicates guest speaker or expert will be present
(F5 due)

Appendix B: Los Angeles Excursion Itinerary
45
Los Angeles Itinerary
October 2-6, 2015

The Chicago Center Loyola University Chicago
MEETING PLACE & TIME: Loyola Welcome Center on October 2 at 5:00pm
HOTEL: Hotel Carmel, Third Street Promenade, 201 Broadway, Santa Monica, CA 90401
Friday, October 2
AFTERNOON
Depart Loyola Welcome

Center at 5:00pm
EVENING
Dinner on your own at

airport
Depart ORD flight UA1557

at 8:50pm
Arrive LAX at 11:14pm
Transport to hotel
Saturday, October 3
Sunday, October 4
Monday, October 5
Tuesday, October 6
AFTERNOON
Dolby Theatre Tour (home

of the Academy Awards!)
1:30pm
TCL Chinese Theatre &

Hollywood Tour 3:00pm
Group Movie 3:45pm
Transport to hotel
AFTERNOON
Lunch on your own &

Universal Studios 12:00pm6:00pm
AFTERNOON
Lunch on your own &

Grammy Museum 1:004:00pm
Griffith Observatory 4:305:30pm
AFTERNOON
Transport from Hollywood to

airport 1:00pm
Depart LAX flight UA438

4:30pm
EVENING
Dinner on your own in

Santa Monica/Free time
EVENING
Group dinner at Stella Barra

in Hollywood 7:00pm
Free time
EVENING
Group dinner at Blue Stove

Restaurant 7:30pm
Free time/Walk back to

hotel
MORNING
Breakfast on your own
Meet in hotel lobby at

9:00am
Getty Museum 10:00am
Group Lunch at the Getty

11:30am-12:30pm
MORNING

8:00am
Beach Yoga 8:30am

11:00am
MORNING

7:00am
Warner Bros. Studio Tour

9:15am-12:00pm
MORNING
Hotel checkout 8:30am
El Pueblo de Los Angeles

9:00am-10:00am
Group breakfast at Mr.

Churro
Madame Tussauds Wax

Museum 10:30am
EVENING
Arrive ORD at 10:32pm;

arrive at Loyola
approximately 11:30pm
46
Appendix C: Logic Model for Loyola University Chicagos Chicago Center
47
Appendix D: Survey Matrix

Chicago Center's Immersion Program
Survey Construct Map and Matrix
Pre- and Post-Immersion
Logic Model
Component
Demographics
Short-Term:
Observe the
cultural
character of
neighborhoods
and several
major cities
Short-Term:
Display open
mindedness
and cultural
sensitivity by
working in
cross-cultural
groups
Short-Term:
Synthesize
intercultural
awareness of
US & own
cultures
through
reflective
journaling
Short Term:
Expand
awareness of
US culture and
norms of
communication
SQ #
Top
Answer
Type
Question
Student I.D. number
Text
Likert
8j
Confidence in ability to observe cultural differences in

excursions
8a
Confident in being open-minded around different cultures
8c
Confident in classes with international students
8e
Confident in interacting with different cultures
8g
8h
Confident in understanding my professors' expectations

Confident in ability to get along with cross-cultural
roommates
8i
Confident in ability to work with other cultural groups
7a
While journaling, list daily events
7b
While journaling, being insightful in your experiences
7c
While journaling, comparing experiences to own culture
7d
While journaling, comparing own culture to culture in other

major U.S. cities
4b
Interacting with Americans in class
4c
8b
8d
8f
Interacting with Americans outside of class

Confidence in being in classes with Americans
Confidence in interacting with the American culture
Confidence in understanding American culture
Answer
Options
4-point: Not
Confident Very Confident
Likert
4-point: Not
Likert
5-point
Frequency:
Never - Very
Often
Likert
4-point: Not
Important
Very Important
Likert
4-point: Not
Logic Model
Component
Short Term:
Participate in
celebrations of
various
holidays of US
& represented
countries
Medium-Term:
Display growth
in
independence,
self-confidence,
and courage
Medium-Term:
Exhibit
intercultural
competence
through the
ability to
observe,
respect, and
adapt to
cultural
difference
Medium-Term:
Demonstrate
greater crosscultural
awareness &
appreciation
Process:
Chicago
Seminar
(UNIV 110)
48
Pre- and Post-Immersion continued

SQ #
Question
5b
Interacting with others from different cultural backgrounds
2a
2b
2c
2d
Initiate conversations in English with Americans with ease

Often push myself outside of my comfort zone
Look forward to exploring off-campus areas on my own
Interact with people outside of the university with ease
2e
Feel immersed in American culture
2e
Adapting to the American culture
3a
Observing aspects of American culture
3b
Observing aspects of non-American cultures
5d
Answer
Type
Answer
Options
Likert
4-point: Not
Important Very Important
Likert
4-point:
Strongly
Disagree Strongly Agree
Likert
4-point:
Strongly
Likert
4-point: Not
Likert
4-point:
Strongly
Likert
4-point: Not
Informal cross-cultural friendships
3c
Appreciate American culture
3d
Appreciate non-American cultures
6a
Participating in Chicago excursions
6b
Participating in the Chicago seminar (UNIV 110)
49
Pre- and Post-Immersion continued

Logic Model
Component
Process: Travel
excursion to 2
major cities
Process: Global
Mentorship
program
Process:
Residency in
the
International
House
Process:
Cultural events
at the
International
House
General
(Confounding
Variables)
SQ #
6c
4d
Question
Participating in two major city travel excursions
Interacting with Americans in the mentorship program
4a
Interacting with Americans at the International House
5a
Cross-cultural interactions at cultural events in the

International House
1a
1b
1c
1d
Speaking English
Writing English
Reading English
Listening and understanding English
Answer
Type
Answer
Options
Likert
4-point: Not
Likert
4-point: Not
Likert
4-point: Not
Likert
Likert
4-point: Not
4-point: Major
Weakness
Major Strength
50
Pre-Immersion Only
Logic Model
Component
General
(Confounding
variables)
SQ #
Question
12
Prior stays in the U.S.
13
Loyola credits count towards degree back home
15
Region roommate is from
Demographics
14
Region student is from
16
Gender
Answer
Type
Multiple
Choice
Multiple
Choice
Multiple
Choice
Multiple
Choice
Multiple
Choice
Answer
Options
Never - Years
Yes / No
51
Post-Immersion Only
Logic Model
Component
Short Term:
Observe the
cultural
character of
neighborhoods
and three
major cities
Short Term:
Display open
mindedness
and cultural
sensitivity by
working in
cross-cultural
groups
Short-Term:
Expand
awareness of
US culture and
norms of
communication
Medium-Term:
Display growth
in
independence,
self-confidence,
and courage
SQ #
Question
9j
Confidence in ability to observe cultural differences in

excursions
9a
Confident in being open-minded around different cultures
9c
Confident in classes with international students
9e
Confident in interacting with different cultures
9g
9h
Confident in understanding my professors' expectations

Confident in ability to get along with cross-cultural
roommates
9i
Confident in ability to work with other cultural groups
9b
Confidence in being in classes with Americans
9d
Confidence in interacting with the American culture
9f
Confidence in understanding American culture
12c
More independent
12d
More self-confident
12e
More courage
Answer
Type
Answer
Options
Likert
4-point: Not
Likert
4-point: Not
Likert
4-point: Not
Likert
4-point:
Strongly
52
Post-Immersion Only continued

Logic Model
Component
Long-Term:
Develop
intercultural
competency
through high
quality
academic
immersion into
American
Society
Process:
Chicago
Seminar
(UNIV 110)
Process: Travel
excursions to 2
major cities
Process: Global
Mentorship
program
Process:
Residency in
the
International
House
SQ #
12f
Question
Answer
Type
Answer
Options
Likert
4-point:
Strongly
Strong intercultural skills
12h
Immersion experience all that I had hoped for
12i
Satisfied with my overall immersion experience
11a
Importance of the Chicago seminar (UNIV 110)
Likert
12a
Aware of the cultural character of neighborhoods
Likert
4-point: Not
4-point:
Strongly
11b
Importance of the major city travel excursions
Likert
12b
Observed major cultural differences in the travel excursion

cities
Likert
4-point: Not
4-point:
Strongly
Likert
4-point: Not
11e
11c
Importance of the Global Mentorship program
Importance of living in the International House
Likert
4-point: Not
53
Post-Immersion Only continued

Logic Model
Component
Process:
Cultural events
at the
International
House
General
(confounding
variables)
SQ #
11d
13
Question
Importance of cultural events at the International House
15
Most of my social interactions are with . . .

Stay connected with family and friends back home through
Skype
Stay connected with family and friends back home through
social media
16
Number of ESL classes taken
17
Number of regular university classes (not ESL or UNIV 110)
14
Answer
Type
Likert
Answer
Options
4-point: Not
Multiple
Choice
Americans Others from

multiple
countries
Multiple
Choice
Never Monthly
Multiple
Choice
0-3
1-4
54
Appendix E: Pre-Test Survey
Evaluation of the Chicago Centers Immersion Program

Pre-Immersion
We are doing an evaluation to see if the Chicago Center meets its goal of providing a high
quality academic immersion into American society. The survey is to measure where you
currently are. While this is voluntary, we hope that you will help us so we can make
improvements to the program for future students.
Please answer the following questions as honestly as possible. You will not cause offense or
display disrespect if you give frank or critical answers. All information you give will remain
confidential, but please give the last six numbers of your Loyola student identification number so
we can link your responses in this survey to the survey we will implement at the end of the
semester.
_________________________
Student identification number
1. Please rate the strength of your English skills

Major Somewhat Somewhat Major
Weakness Weakness Strength Strength
a. Speaking English
b. Writing English
c. Reading English
d. Listening and understanding English

2. Please rate your level of agreement or disagreement
about living in the U.S.
55
Strongly Somewhat Somewhat Strongly

Disagree Disagree Agree
Agree
a. I initiate conversations in English with

Americans with ease
b. I often push myself outside of my comfort zone
c. I look forward to exploring off-campus areas on

my own
d. I interact with people outside of the university

with ease
e. I am adapting to the American culture
f. I feel immersed in the American culture

about your cross-cultural skills

Agree
a. I observe aspects of American culture
b. I observe aspects of non-American cultures
c. I appreciate American culture
d. I appreciate non-American cultures
4. Please rate the level of importance you place on

interacting with Americans in the following ways:
Not
Important
Somewhat
Important
Very
Important Important
a. At the International House
b. In class
c. Outside of class
d. In the mentorship program

the following cross-cultural interactions:
56
Not
Important
Somewhat
Important
Important
Very
Important
a. Attending cultural events put on by the

International House
b. Interacting with others from different

cultural backgrounds
c. Developing informal cross-cultural friendships
Not
Important
Somewhat
Important
Important
Very
Important
a. The Chicago excursions
b. The Chicago seminar (UNIV 110)
c. The two major city travel excursions

participating in the following activities:
7. How often do you include the following in your

journal:
Never
Seldom Sometimes
Often Very Often
a. Listing daily events
b. Reflecting about my experiences
c. Comparing my experiences to my own culture
d. Comparing my own culture to cultures observed

in other major U.S. cities

8. How confident do you feel in the following
situations:
Not
Confident
57
a Little
Confident
Somewhat
Very
Confident Confident
a. Being open-minded around different cultures
b. Being in classes with Americans
c. Being in classes with international students
d. Interacting with the American culture
e. Interacting with different cultures
f. Understanding American culture
g. Understanding my professors expectations
h. My ability to get along with my

cross-cultural roommate
i. My ability to work with other cultural groups
j. My ability to observe cultural differences in

excursions
For the following questions are multiple choice, please choose the answer that best describes
your situation.
9. I would count prior stays in the U.S. in . . .
a) Have never visited before
b) Days
c) Weeks
d) Months
e) Years
10. My Loyola credits from classes count toward my degree back home
a) Yes
b) No
c) Do not know
11. I am from the following region . . .

a) Asia/Pacific Islands
b) Middle East
c) Africa
d) Europe
e) North America
f) Central America
g) South America
12. My roommate in the International House is from the following region . . .

a) Asia/Pacific Islands
b) Middle East
c) Africa
d) Europe
e) North America
f) Central America
g) South America
13. I identify with the following gender . . .

a) Female
b) Male
c) Other
Thank you for your time
58
59
Appendix F: Post-Test Survey
Evaluation of the Chicago Centers Immersion Program

Post-Immersion
We are doing an evaluation to see if the Chicago Center has met its goal of providing a high
quality academic immersion into American society. This survey is to measure your intercultural
skills from your immersion experience. While this is voluntary, we hope that you will help us so
we can make improvements to the program for future students.
Please answer the following questions as honestly as possible. Since we are evaluating the
quality of the immersion program, do not be concerned if you feel there was no growth in some
areas. You will not cause offense or display disrespect if you give frank or critical answers. All
information you give will remain confidential, but please give the last six numbers of your
Loyola student identification number so we can link your responses in this survey to the survey
you took at the beginning of the semester.
_________________________
Student identification number
1. Please rate the strength of your English skills

Major Somewhat Somewhat Major
Weakness Weakness Strength Strength
a. Speaking English
b. Writing English
c. Reading English
d. Listening and understanding English
60

about living in the U.S.
Agree
a. I initiate conversations in English with

Americans with ease
b. I often push myself outside of my comfort zone
c. I look forward to exploring off-campus areas on

my own
d. I interact with people outside of the university

with ease
e. I am adapting to the American culture
f. I feel immersed in the American culture

about your cross-cultural skills

Agree
a. I observe aspects of American culture
b. I observe aspects of non-American cultures
c. I appreciate American culture
d. I appreciate non-American cultures

interacting with Americans in the following ways:
Not
Important
Somewhat
Important
a. At the International House
b. In class
c. Outside of class
d. In the mentorship program

Very
Important Important

the following cross-cultural interactions:
61
Not
Important
Somewhat
Important
a. Attending cultural events put on by the

International House
b. Interacting with others from different

cultural backgrounds
c. Developing informal cross-cultural friendships
Not
Important
Somewhat
Important
a. The Chicago excursions
b. The Chicago seminar (UNIV 110)
c. The two major city travel excursions

participating in the following activities:
Very
Important Important
Very
Important Important
7. How often do you include the following in your

Journal:
Never
Seldom Sometimes
Often Very Often
a. Listing daily events
b. Reflecting about my experiences
c. Comparing my experiences to my own culture
d. Comparing my own culture to cultures observed

in other major U.S. cities
8. How confident do you feel in the following

situations:
Not
Confident
a Little
Confident
Somewhat
Very
Confident Confident

8 (continued) Your level of confidence

Not
Confident
62
a Little
Confident
Somewhat
Very
Confident Confident

j. My ability to observe differences in cultural

excursions
9. Think back to early in the semester and indicate

what you think your confidence level actually was
then:
Not
Confident
a Little
Confident
Somewhat
Very
Confident Confident

j. My ability to observe differences in cultural

excursions
63
10. How often did you participate in the following

activities during the semester:
Never
Seldom Sometimes
Often Very Often
a. Global mentorship program
b. Cultural events at the International House
c. Student organizations
d. Community service
11. How important were of each of the following

Chicago Center components in your overall
Learning:
Not
Important
Somewhat
Important
Very
Important Important
a. The Chicago seminar (UNIV 110)
b. Major city travel excursions
c. Living in the International House
d. Cultural events at the International House
e. The Global Mentorship program
12. Having completed my immersion experience . . .

Agree
a. I am aware of the cultural character of neighborhoods
b. I observed major cultural differences in the travel

excursion cities
c. I am more independent
d. I am more self-confident
e. I have more courage
f. I have strong intercultural skills
g. My immersion experience was all I had hoped for
h. I am satisfied with my overall immersion experience
64
For the following questions are multiple choice, please choose the answer that best describes
your situation.
13. Most of my social interactions in Chicago have been with . . .

a) Americans
b) Others from my own country
c) Others from one other country
d) Others from multiple other countries
14. I stay connected with my family and friends back home through Skype . . .
a) Never
b) Daily
c) Weekly
d) Monthly
15. I stay connected with my family and friends back home through social media . . .
a) Never
b) Daily
c) Weekly
d) Monthly
16. The number of English-as-a-Second-Language (ESL) classes I took . . .

f) Zero
g) One
h) Two
i) Three
17. The number of regular university classes (not ESL or UNIV 110) I took . . .
a) One
b) Two
c) Three
d) Four
Thank you for your time!
65
Appendix G: Focus Group Email Invitation
Dear ___________,
Are you interesting in receiving a $15 Loyola Bookstore gift card? As a member of the Chicago
Center, we would like to hear about your experiences in the different components of the Centers
program. Would you be willing to meet for 90 minutes with 6-8 other students to discuss your
immersion experience? The focus groups will be informal with both general and some more indepth questions about the program components.
We are interested in your honest opinion of the immersion program. During the focus group,
pizza dinner will be provided, and at the end of the focus group, you will receive your $15
Loyola Bookstore gift card.
Please respond to this email with the subject line: YES and your country of origin. If more
students are interested in participating than we need, we will contact you with specific details
and timing of the focus group if you are selected to be part of one of them. Otherwise, we will
email you thanking you for your willingness to participate. We look forward to hearing from
you!
Sincerely,
The Chicago Center
66
Appendix H: Focus Group Consent Form
CONSENT TO PARTICIPATE IN THE CHICAGO CENTER FOCUS GROUP
Project Title: Chicago Centers Immersion Program

Evaluator:
Introduction:
You are being asked to take part in a focus group to help inform the Chicago Center about your
thoughts and experiences about their immersion program. You are being asked to participate
based on your response to our email.
Please read this form carefully and ask any questions you may have before deciding whether to
participate in this focus group.
Purpose:
The purpose of the focus group is to gather information about students experience in the
immersion program.
Procedures:
If you agree to participate in the focus group, you will be asked to respond to a series of
questions that will last for approximately 90 minutes. You are encouraged to respond openly and
honestly to the questions asked of you, although you should only respond to questions you feel
comfortable doing so.
Risks/Benefits:
There are no known risks involved in participating in this study, although discussions concerning
the immersion program may be sensitive for some individuals. Benefits may include
improvements to the immersion program and a greater understanding of whether and how
intercultural competence is transmitted to students through different components of the program.
Confidentiality:
Your name will not be associated with your responses in the focus group reports. We will
compile a report of basic themes and share it with you prior to finalizing it. The information
gathered in todays focus group will only be shared with members of the Chicago Center staff.
Voluntary Participation:
Participation in this focus group is voluntary. Even if you decide to participate, you are free to
leave at any time without penalty.
Contacts and Questions:
If you have questions about the focus group, you can contact Anne Conlon at aconlon1@luc.edu.
Statement of Consent:
Your signature below indicates that you have read and understood the information provided
above, have had an opportunity to ask questions, and agree to participate in this focus group.
You will be given a copy of this form to keep for your records.
____________________________________________
Participants Signature
Date ____________
____________________________________________
Evaluators Signature
Date ______________
67
68
Appendix I: Focus Group Protocol

Preliminaries
Good evening. We appreciate you taking time out of your busy day to participate in our Chicago
Center focus group. Please help yourselves to the pizza if you have not done so already. We
will begin shortly.
Introduction: Hi, my name is ___________ and I look forward to having a conversation with you
today. For the next 90 minutes, we will discuss your experiences in the Chicago Center
program. Specifically, we will discuss your experiences and feelings about living at the
International House, taking the course UNIV 110: Exploring the U.S. through Chicago, the
one/two excursions to major U.S. cities, and your cross-cultural interests and interactions as well
as your interactions with Americans. (Name of scribe), will be our recorder for our conversation
today. She will be taking notes about general ideas and main points that are brought up.
However, there will be no names attached to the comments, so please feel free to honestly
address the questions wherever you feel comfortable. In addition, we are recording our session
today by audio tape.
Purpose of the Focus Group: The reason that we are conducting a focus group is to better
understand your experiences as a student in the Chicago Center program. The Chicago Centers
mission is to provide a program of high-quality academic immersion into American society to
increase your intercultural competence. Please feel free to share as much or as little information
as you would like. There is no right or wrong answer. This is a safe space and all of your
comments will remain confidential. We also ask that you do not repeat any information you hear
today during this focus group. Please keep in mind that any comments that you have will help us
improve the Chicago Centers program for future students. Do you have any questions?
Consent Form: Before we go on, we are distributing a consent form. Please review it, and if you
are comfortable with it, please sign it and return it to me so we can begin.
Focus Group Discussion

Participant Introductions: (approximately 10 min.)
1. Lets begin by going around and introducing ourselves. Please tell us your name, where
you are from, your home university, and your major. (If everyone knows each other, just
ask for their home university and major.)
2. Can you briefly share with us why you chose to enroll in the Chicago Center program?
Thank you for sharing. I now want learn about your feelings and experiences regarding each
separate component of the Chicago Centers immersion program.
69
Segment 1: Living at the International House (approximately 10 min.)

Lets begin with the International House
1. Now that you have been living in the International House for a while, can you describe
what your experience has been like living in the house?
Probes:
What do you like most about it?
What do you like least about it?
Tell us about your experience with the Student Living Assistants.
What, if anything, should be changed at the International House?
I appreciate your comments so far.
Segment 2: Travel Excursions to Major U.S. Cities (approximately 10 min.)
Now I would like to move on and talk about your excursion(s) to other major U.S. city(s). (plural
needed for Fall semester and singular for Spring)
1. Tell us what it is like to explore this city/these cities?
Probes:
What did you like most about it?
What did you like least about it?
What was it like on the tours?
Did you notice anything that you found particularly interesting?
What, if anything, did you learn from the excursion(s)?
How was it having a new roommate?
What, if anything, should be changed in these excursions?
Thank you, what you are sharing is very useful.
Segment 3: UNIV 110: Exploring the U.S. through Chicago (approximately 10 min.)
Lets move on now to talk about the required course you are taking - Exploring the U.S. through
Chicago.
1. Tell us what its like to be in this course?
Probes:
What is one thing you learned in the class that stands out or you will remember
after the experience is over?
Tell us about having class and homework in English.
Tell us what you think about the professor.
Tell us what it was like to work in cross-cultural groups.
Tell us about your class outings.
What, if anything, should be changed in this course?
70
We are grateful for you sharing your thoughts and experiences with us so far. While we have
covered the main parts of the Chicago Center program, we would now like to talk about what it
has been like in interacting with Americans and others from different cultures.
Segment 4: Cross-Cultural Interest and Interactions (approximately 15 min.)
Lets begin with your cross-cultural interests and interactions.
1. Tell us what it was like to interact with non-American cultures.
Probes:
What, if anything, have you learned by interacting with others from nonAmerican cultures?
Tell us about challenges, if any, for you in interacting with others from nonAmerican cultures.
What affect, if any, has it had on you living and working on assignments in crosscultural situations?
2. Tell us about your cross-cultural interests
Probes:
Did your cross-cultural interests change? If so, how?
Tell us what you thought about the cross-cultural events at the International
House.
What do you like most about the cross-cultural events?
What do you like least about the cross-cultural events?
Segment 5: Interacting with Americans / Participation in the Global Mentorship Program
(approximately 15 min.)
Good. Now lets talk about interacting with Americans.
1. Tell us what it is like for you interacting with Americans
Probes:
What has been most challenging?
What has been most rewarding?
What do you think has been your biggest accomplishment in interacting with
Americans?
What, if anything, have you found most confusing in interacting with Americans?
2. If any of you participated in the Global Mentorship program, tell us what that has been
like for you
Probes:
What, if anything, has been most challenging with the program?
What, if anything, has been most rewarding with the program?
Is there anything that should be changed in the program?
71
You have all been most helpful. We are almost done.
Conclusion (approximately 10 min.)

It takes a great deal of courage to study abroad in an immersion program and youve probably
had to push yourselves outside of your comfort zones.
1. Is there anything more that the Chicago Center could have done to support you in this
immersion experience?
2. Is there anything else you would like to add that would help us improve Chicago Centers
immersion program for future students?
(If time permits, the moderator can re-cap the main themes of the evening.)
That concludes our conversation. We thank you very much for taking your time to share your
thoughts and experiences with us. Your feedback is very important to us because without it, we
would not be able to fully understand what we are doing well and what needs improvement. As
you leave, you will each receive a Loyola Bookstore gift card as a token of our appreciation on
behalf of the Chicago Center.
72
Appendix J: Focus Group Coding Construct Map & Matrix

Chicago Center's Immersion Program
Focus Group Coding Construct Map and Matrix
Logic Model
Intervention
Construct
Code
Meaning
Theme
CLASS
Class
CROSS
Cross-Cultural
Related to UNIV 110

Regarding non-American cultural
interactions
Process: Travel excursions

to 2 major cities
EXCUR
Excursion
Related to travel excursions to major U.S.

cities
Process: Residency in the

International House
IHSE
International
House
Related to living in the International House
Process: Chicago Seminar

(UNIV 110)
Process: Cultural events at

the International House
CEVNTS
Cultural Events
CROSS
Cross-Cultural
American
Culture
AMCULT
Process: Global
Mentorship program
Process: General overarching all the
interventions
GLBL
AMER
AMCULT
Global
Mentorship
American
American
Culture
Related to the Cultural Events put together

by the International House
Regarding non-American cultural
interactions
Related to American culture
Related to the Global Mentorship program

Regarding interactions with Americans
Related to American culture

Appendix K: Focus Group Coding Rubric
Construct
Code
Meaning
IHSE
International
House
CROSS
CrossCultural
EXCUR
Excursion
Theme
Comments related to living in the International House
Comments regarding interactions with others from different
countries (excluding interactions with Americans)
Comments related to travel excursions to major U.S. cities
CLASS
Class
Comments related to UNIV 110
AMER
American
Comments regarding interactions with Americans
AMCULT
American
Culture
Global
Mentorship
Cultural
Events
Comments regarding the American culture
GLBL
CEVNTS
Comments related to the Global Mentorship program

Comments related to the Cultural Events put together by the
International House
73
74
Appendix L: Timeline
Fall Semester Timeline for First Wave Evaluation for the Chicago Center
2016
Evaluation Tasks
Jan.
April
Aug.
Sept.
Oct.
Nov.
Finalize evaluation plan with Chicago
Wk of
Center
Jan. 4
Wk of
Jan. 11
Administer pilot of pre-test
Review findings from pilot test and
Wk of
update survey
Jan. 11
Wk of
Apr 4
Conduct pilot of focus group
Wk of
update focus group protocol
Apr 4
Administer pilot test of pre-test
Wk of
adjustments & new post-test questions
Apr 25
Wk of
update survey
Apr 25
Wk of
Aug 22
Administer pre-test survey
Wk of
Aug 29
Enter pre-test survey data into SPSS
First 2
Weeks
Enter pre-test survey data into SPSS
Run analysis of quantitative data via
SPSS, organize charts and tables
x
Talk to UNIV 110 about the Focus
Wk of
Group
Oct. 24
Send emails to invite focus group
Wk of
participants
Oct. 24
Send out reminder emails (weekly)
Conduct two focus groups
Listen to recording and generate notes
(18 hours)
Code and analyze qualitative data
Administer post-test survey (Final day of
course)
Enter post-test survey data into SPSS
Run analysis of quantitative data vis
SPSS, organize charts and tables
Prepare final report
Share final report with Provost and
other key stakeholders
Dec.
2017
Jan.
x
Wk of
Nov. 7
Nov. 725
Nov. 28- Dec. 130
14
Wk of
Dec. 5
Wk of
Dec. 12
Wk of
Dec. 19
Jan. 213
Wk of
Jan. 17
75
Appendix M: Budget
Activity
Survey
Administration
Item
Paper (Pre- & post-tests)
Cost per Item
Total
Quantity Cost
$0.12
50
$6.00
$0.00
Audio Recording Device

Consent Forms
Loyola Bookstore Gift Cards
Pizza
Soda (2 liters)
Plates & Cups
$0.00
(from LUC)
$0.00
$0.02
$15.00
$17.00
$2.00
$1.50
1
16
16
6
8
16
$0.00
$0.38
$240.00
$102.00
$16.00
$24.00
SPSS
(from LUC)
$0.00
$0.00
$0.00
$0.00
1
1
$0.00
$0.00
$10.00
$20.00
Focus Group
Room Reservation
Statistical Analysis
Qualitative Analysis
Transcribing
Coding
Reporting
Spiral Bound Reports
TOTAL
$408.38

Appendix N: Presentation
76
77
78
79
80
81

Hatland Conlon Evaluation Plan For Chicago Center 1

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Hatland Conlon Evaluation Plan For Chicago Center 1

Uploaded by

Copyright:

Available Formats

Running head: EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

Evaluation Plan for the Chicago Center Program

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

Validity and Reliability ............................................................................................................. 34

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

Evaluation Plan for the Chicago Center Program

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

Appendix A: UNIV 110 Syllabus

FALL TERM 2015

Time: Fridays 4:15p.m.-6:30p.m.

Office Hours: by appointment

Gaining factual knowledge (terminology, classifications, methods, trends).

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

EVALUATION PLAN FOR THE CHICAGO CENTER PROGRAM

Sept 5 : Field Trip: Chicago History Museum

Sept 11 : Chicago Neighborhoods / Introduction to Neighborhood Project (F1 due)

Sept 18 : Chicagos Cultural Scene: Literature, Art, and Architecture

Sept 26 : Field Trip: Art Institute of Chicago

Oct 2 : Schools and the City: Education in Chicago

Oct 9 : Green Chicago: Urban Space and Environment in Chicago

Oct 17 : Field Trip: LUREC

Oct 23 : Chicago Eats!!! Food and Culture in the Windy City

Oct 30 : Midterms Presentations (Chicago Neighborhood Projects due)

Nov 6 : Field Trip: Taste Bud Tour

Nov 13 : Big City Bosses: Politics in Chicago*