Running head: FINAL PLAN

Final Evaluation Plan: Intercultural and Diversity Centers Advocacy in the Community
Haniyyah Bashir
University at Buffalo
ELP 539: Topics in Program Evaluation and Assessment
Fall 2015

FINAL PLAN

Intervention
For the needs of students who are interested in starting, continuing, and increasing their activism
specifically within the campus and Buffalo community at large without a program to do so, the
intervention is the IDC Advocacy in the Community program.
Statistical Method
Comparison Groups: Differences in Differences that attempts to isolate the treatment effect by
comparing the change over time in the treatment group to the change over time for the control
group. Pre-test and post-test using the Likert Scale.
Control Group
The Control Group are the students who
registered on UBLinked and did not attend the
program.

Treatment Group
The Treatment Group are the students who
registered on UBLinked who attended the
program.

Outcome Measure 1

How Outcome Variable Operationalized

Other Potential Causes of Outcome

Control For
Students who have
the Student
Association cultural
club membership,
Students who are
diversity advocates,
students who are
more passionate and
more engaged.

Cannot Control For

Students who do not
have the Student
Association cultural
club membership,
Students who are not
diversity advocates,
students who are not
passionate and
disengaged.

(Optional)
Outcome Measure 2

How Outcome Variable Operationalized

Outcome Measure 3

How Outcome Variable Operationalized

Outcome Measure 4

How Outcome Variable Operationalized

FINAL PLAN

Final Evaluation Plan

It is the University at Buffalos Intercultural and Diversity Center's mission to support all
students in their personal development, create a sense of belonging, celebrate diversity, and
discuss issues that broaden a student's perspective. In this paper, I will discuss the functions and
intended student outcomes of the campus workshop series: Advocacy in the Community,
describe how the workshop functions on a month-to-month basis, analyze the significance of the
outcomes of the program, and explore possible methods that might be used to ensure the effects
of the program are isolated from other influences.
As a Student Life office, the Intercultural and Diversity Center values advocacy as a
means for the students to learn about themselves as individuals and the rich diversity of our
surrounding community. The challenge is to apply what is learned in the classroom to daily life
through these workshops, and developing on and off-campus partnerships with like-minded
community organizations. Through the Life and Learning workshop: Advocacy in the
Community program, based on the themed month, our goal is to introduce students to social
issues, create dialogue with community organizations in the Buffalo community, and explore the
students role in becoming a change agent. In contact with the organization, the goal is their
participation in leading the discussion that the students will learn the needs of the designated
population within the Buffalo community, have the ability to define "advocate," and begin to
understand/explore their personal role towards becoming an advocate. The goal is for the
students to know how to get involved with community organizations through volunteer and/or
internship opportunities so they can become change agents.

FINAL PLAN

During the fall and spring semester, the Intercultural and Diversity Center picks certain
themes from months to focus on for Advocacy in the Community. From September 15 th to
October 15th, it is National Hispanic Heritage Month that honors the culture, heritage, and
contributions of Hispanic Americans. October is Disability Employment Awareness Month;
therefore, there is awareness of the Americans with Disabilities Act (ADA) that prohibits
discrimination against people with disabilities in many aspects of life. In addition to disability
awareness, the month of October is a celebration of LGBT History month through the
exploration of the LGBT Rights Movement. During the month of November, Native Americans
are recognized for significant contributions that were made to the establishment and growth of
the United States. February is National Black History Month that celebrates and honors
achievements of people of the African Diaspora. Following February, March is Womens History
Month that highlights and draws attention to the accomplishments of women. April is Celebrate
Appreciate Diversity Month, which is dedicated to acknowledge the diversity by celebrating
differences and similarities of others. Last but not least, Asian-American and Pacific Islander
Heritage Month is in May and the purpose is to bring to light the Japanese immigration,
completion of the transcontinental railroad, and the highlight the Chinese immigrant workers. By
the end of each program of each themed month, the students should learn about the issues that
the population face and ways to become advocates for those issues.
In order to execute the Advocacy in the Community program, the intern identifies and
contacts potential speakers to invite, describes the program goals, provides time/date, allows
technology availability, and provides talking points. From that point, there is advertising around
the student union via slides on the green monster and flyers on bulletin boards, flyers posted on
social media, and a huge 20 by 30 poster right outside the office. During the day of the program,

FINAL PLAN

the room is set-up in a U shape for open discussion, the computer screen is set up (if needed),
and there is a swipe machine available to track students. With the swipe systems, you are able to
track who are coming to the events and what the demographics are. At the end of the event, the
speaker receives a thank you note and gift from the office for their time and participation.
Overall, the major program goals were accomplished with the phenomenal speakers from
Hispanic Heritage Council of WNY (Casimiro Rodriguez), Pride WNY (Matthew Crehan
Higgins), Community Services for the Developmentally Disabled (Janice Hamner-Baines and
Denise), and Native American Community Services (Michael Martin and Marcus Alexander).
They each had their connection to UB students, whether it was through class or club
participation. Students were interested in finding out more information and connecting with the
speakers after the workshop. For the last event, the intern chose to collaborate with the student
organization, First Nations SA, was very insightful and purposeful for the November workshop.
Flyers placed around the student union and in student club mailboxes were a great way to
publicize. In addition, flyers were sent to related courses/departments/professors and other
offices of interest.
The challenges of the program included failing to meet program attendance goals of 1015 individuals and poor social media publicity for the program. It was difficult to advertise for an
event when there was not access granted for the social media outlets. This program does not
currently have a budget to use to provide food/light snacks. In reflecting on those challenges, I
recommend collaborate with student clubs and organizations to select a time that their general
body meetings are held and work together to bring in a guest speaker into our space to increase
attendance. It might be useful to have a speaker from an on-campus student organization and off-

FINAL PLAN

campus Buffalo community organization for the themed workshop. Because of the time of the
workshop, there should be food/light snacks provided. As a take away item for the students who
attend, the office should provide notepads/pens and/or information cards with organization
contact and IDC office contact for the workshop. Most importantly, there should be a method for
the attendees to evaluate the workshop.
During this academic year, I have the opportunity to intern in the Intercultural and
Diversity Center. I am responsible for the Advocacy in the Community program, as well as other
program initiatives. Im examining this specific workshop series to figure out the effectiveness of
Life and Learning Workshop Series: Advocacy in the Community. University at Buffalos
students advocate for themselves on a daily basis and by attending the program suited to learn
about certain populations the hope is that they are encouraged to make a difference in the
community around them. In reiteration, the outcomes of the program are for the students to
understand the issues of the designated population within the Buffalo community, have the
ability to define "advocate," and begin to understand/explore their personal role towards
becoming an advocate.
In order to measure the listed above program outcomes of the participants, one needs
examine the advocate efficacy, their enthusiasm about the topic, and their ability to form their
own definition and meaning of advocacy. To evaluate the effectiveness of this program, I intend
to use the Multiple Time Series method, also known as Differences in Differences, that attempts
to isolate the treatment effect by comparing the change over time in the treatment group to the
change over time for the control group. Even though it assumes that the two groups are close to

FINAL PLAN

identical, it will be feasible to pick out the changes that are due to treatment from the changes
that would have happened anyway.
For this office program evaluation, one would need to utilize one of the campus
resources, UBLinked, to track participants. UBLinked is the main website for all of campus
activities and involvement that students used to discover, connect with, and acquire more
information about different clubs, organizations, and offices on campus. Clubs, organizations,
and offices post all their events and students have the ability to RSVP. With this RSVP feature,
offices and organizations know how many students to expect. The Control Group are the students
who registered on UBLinked and did not attend the program. The Treatment Group are the
students who registered on UBLinked who attended the program.
To ensure that these projected outcomes were caused by the program, the pre- and postsurveys for this program experience would be applicable for gathering quantitative and
qualitative data to give to the control group of the students who attend and participate. The
quantitative data will be able to provide reflective feedback on the experience. Once all students
register on UBLinked, each person will answer open-ended questions with geographic
information, give their definitions of advocacy, state the issues they are aware of in the
community, and whether or not they consider themselves advocates. After the Advocacy in the
Community program, the control group and treatment group will both complete the same survey
to see if their answers changed. Some of the challenges and/or factors that could contribute to
changes in the data include the difference in quality of the presenters from the different
organizations, how engaging the presentation is, the attention span of the students, and whether
or not the students decide to take notes. Once the data is collected, one needs to subtract the

FINAL PLAN

difference to see how much of an impact the treatment, Advocacy in the Community program,
has on the participants.
Evaluability Assessment
Activism in higher education has been happening campus wide and nationwide for
decades since inception. Inspired by the University at Buffalo White only, Black only signage
contro versy and National BlackOut Solidarity Movement, I am examining the University at
Buffalo Intercultural and Diversity Centers Advocacy in the Community program. This program
caters to the needs of students who are interested in starting, continuing, and increasing their
activism specifically within the campus and Buffalo community at large. The literature review is
viewing college student activism with a focus on social justice, civic engagement, and
multicultural competency as it affects college campus programs. Subsequently, based on the
discussion of current literatures, I will evaluate what still needs to be discovered and
incorporated to improve the Advocacy in the Community program through the development of a
logic model that links the activities of the program with the short-term outputs, mid-range
outcomes, and long-range impacts.
Current Issues, Trends, and Theories with College Student Activism.
Research shows there is a relationship between activism, social justice, civic/community
engagement, and multicultural competency in the higher education environment. (Ballard, 2014;
Bowman & Park, 2015; Denson & Bowman, 2013; Einfield & Collins, 2008; Ginwright &
James, 2002; Laird, 2005; Sax, 2004; Woolworth, 2014; Yee, 2008)
Youth advocacy is clearly defined as a form of social practice focused on the positive
development of young people through the use of political and social processes that seek to

FINAL PLAN

influence public policy, resource allocation, support services, and media perspectives around
systems of care. (Woolworth, 2014) In addition, youth advocacy has grown out of a number of
efforts connected to delinquency prevention, foster care reform, and adolescent mental health
that advance an assets-based or prevention-oriented approach to youth development. Informed by
the evolution of developmental systems theories, youth advocacy activities are initiated within
and by neighborhood youth organizations and community coalitions, professionally managed
advocacy groups, and social networks organized along the lines of youth empowerment, rights,
and voice. (Woolworth, 2014; Yee, 2008).
Researchers are still looking into educating about youth advocacy as it is beneficial for all
individuals involved in the process. Youth organizing and advocacy programs are proving to be
particularly transformative for young people from low-income families and traditionally
marginalized communities, as such programs encourage more empowering connections to selfidentity, cultures, and civic life. By reflecting on both the history and the current state of the
field, foundations can develop strategies to strengthen an emerging and robust body of work that
is ripe for advancement. (Woolworth, 2014; Yee, 2008) UB Advocacy in the Community
program is an impactful way for students to make those connections and reflect on the current
state of the partnered organizations.
Einfeld and Collins discuss how youth in a long-term service-learning program described
their understanding of and commitment to social justice, multicultural competence, and civic
engagement. Using the AmeriCorps service-learning program as a focus, people that participated
in the study increased their awareness of inequality, but only some adopted a commitment to
social justice. Furthermore, multicultural skills are developed while interacting with their clients,
such as empathy, patience, attachment, reciprocity, trust, and respect. (Bowman, 2015; Einfeld,

FINAL PLAN

10

A., & Collins, D., 2008, Laird, 2005; Sax, 2004) Participating in service programs provides a
commitment to continued civic engagement. There is still a need for additional improvements for
students to increase their competency at the college level. UB attempts to address the learning
aspect by categorizing the Advocacy in the Community program as a Life and Learning
Workshop Series and encourages students to go as part of their freshman year requirement.
Civic commitment is influential for college students and there are ways that colleges can
best prepare students for a life as a caring and involved citizen. (Sax, 2004) Many aspects of the
college experience serve to influence students commitment to social activism (Bowman, 2015;
Denson, 2013; Einfield, 2008; Ginwright, 2002; Sax, 2004; Woolworth, 2014; Yee, 2008), sense
of empowerment, and community involvement (Ballward, 2014; Denson, 2013; Einfield, 2008;
Sax, 2004; Yee, 2008). There is an effect of performing volunteer work during the college years,
which enhances students commitment to social activism and involvement in the community
after college. Clearly, forming a habit of volunteerism is critical to the long-term development of
citizenship. (Einfeld & Collins, 2008; Sax, 2004)
One of the many factors that influence citizenship development is socializing with
students from different racial or ethnic groups, which influenced both the sense of empowerment
as well as students involvement in their community after college. Furthermore, students
likelihood of community involvement was also enhanced by attending racial or cultural
awareness workshops. (Sax, 2004). It is interesting to consider the dynamics of the relationship
between involvement and civic responsibility. On the one hand, citizenship development is
enhanced by the nature of specific activities, such as attending religious services, socializing
across racial and ethnic lines, and discussing political and social issues. On the other hand,
students who become involved in such activities are likely forming a habit of involvement; it is

FINAL PLAN

11

this habit that carries over into the students life in the years after college. Overall, the key is to
provide a variety of opportunities for student involvement, particularly in ways that expose
students to diverse people and issues. (Bowman, 2015; Einfield, 2008, Laird, 2005; Sax, 2004)
Higher education researchers and practitioners have emphasized the educational benefits
of fostering meaningful interracial interaction on college campuses. The Bowman & Park study
focuses on two extremes of interracial contactvery close interracial friendship and the simple
frequency of cross-racial interaction with multiple groups. Multiple theoretical frameworks
suggest that these two types of interpersonal diversity experiences may have differential effects.
Regardless of students race/ethnicity, cross-racial interaction is consistently associated with
desired student outcomes, whereas close interracial friendship is often unrelated to these same
outcomes. (Bowman & Park, 2015) Future studies should examine and compare other types of
interracial contact and draw upon additional data to better compare the benefits of different types
of social networks, cross-racial interaction, and interracial friendship.
Bring the light the traumatic effects of racism on campuses, institutions must work to
support a healthy campus racial climate that maximizes learning and development for all
students since a racially diverse student body on its own will not produce positive outcomes, it
needs to be accompanied by intentional policies that support and nurture that diversity. From
diversifying the faculty to ensuring that the curriculum addresses issues of equity and diversity in
a meaningful fashion, institutions must act to maximize opportunities for students to engage with
diversity. (Bowman & Park, 2015) Ways that institutions work on addressing those issues are
through campus programming and trainings that ensure that students are understanding the needs
of different individuals and supporting them as well as learning how to be an ally and an
advocate.

FINAL PLAN

12

Similar to blacks prior to 1954, todays young people face intense economic isolation,
lack political power, and are subjected to pervasive social stigma. (Ginwright & James, 2002)
Urban youth of color, who have largely been ignored in mainstream youth development
literature. In response, youth throughout the country are mobilizing to demand a voice in public
policy and are transforming institutions to be more accountable to their communities. By placing
the focus on marginalized youth, it is visible the deeply rooted social issues they confront and
explore how they creatively respond through organizing, political education, and identity
development. (Bowman, 2015; Einfield, 2008; Ginwright & James, 2002; Sax, 2004) More
research needs to be done on youth organizing frameworks and synthesize key principles that
might contribute to a general theory or model of social justice and youth development. Finally,
through recent examples of youth political action like the Ferguson demonstrations and
BlackLivesMatter protests, we explicate the conditions for successfully engaging youth in their
political development and empowerment and examine the individual, community, and
institutional impacts of youth participation in political organizing.
There is growing research showing that students' views, attitudes, and university diversity
experiences promote preparation for a global society. However, there is little research available
outside of American higher education contexts. Denson & Bowman study collected at one
Australian university to examine whether students' views and attitudes towards diversity, and
their university diversity experiences, stimulate the development of key attributes needed to
function effectively in a global society, namely positive intergroup attitudes and civic
engagement. High-quality engagement with curricular diversity activities (institutionally
structured opportunities for students to engage with diversity) and with diverse peers (positive
diversity interactions) are associated with improved intergroup attitudes and civic engagement

FINAL PLAN

13

outcomes. While, poor quality engagement with diverse peers (negative diversity interactions)
are negatively associated with gains in these outcomes. (Denson & Bowman, 2013) These
findings are consistent regardless of students' pre-university experience with diversity and their
openness to diversity. More research needs to be done worldwide about diversity experiences and
ways that college students build their multicultural competency.
While there is a growing body of research that suggests that diversity and diversity
related programs and courses have positive educational effects for all students and researchers
are continuing to build a strong base of research is critical to better understand the diversity and
diversity related programs on our campuses. (Laird, 2005) A deeper understanding of the
educational effects of diversity will better equip instructors and practitioners to adapt to any
resulting changes and improve the education of college students. Researchers explain that
experiences with diversity, particularly interaction with diverse peers and curricular exposure to
diversity, provide the challenge that is necessary for the development of a healthy sense of self
and more complex cognitive structures.
Experiences with diversity are important influences on the development of student
learning and democratic outcomes, including students intellectual engagement and motivation as
well as citizenship engagement. Lairds study investigated the relationship between experiences
with diversity and the ways students view themselves. Two outcomes investigated were
academic self-confidence and social agency and included the disposition to think critically, an
outcome that captures students attitudes about and motivation for using thinking to solve
problems. Furthermore, the current study goes one step beyond examining only the direct effects
of diversity experiences such as taking diversity-related courses and interacting with diverse
peers.

FINAL PLAN

14

There has been investigations of how diversity experiences work together to influence
student outcomes. Students with more experiences with diversity, particularly enrollment in
diversity courses and positive interactions with diverse peers, are more likely to score higher on
academic self-confidence, social agency, and critical thinking disposition. (Laird, 2005)
Diversity is a critical component of educating college students as it fosters development of
certain aspects of self. More research needs to be done on the continued development and
offering of curricular opportunities for students to engage multiple aspects of diversity and
contexts within which students from different racial and ethnic backgrounds can interact
respectfully and productively.

LogicModel/TheoryofChange

Conside
Program
needs to
meet
students
needs

r:
Mission
Values

Staff,
Communit
y partners,
Technolog
y

Goals
Resource

Materials

Advertise,

College

Learning

Reach out

students,

Awareness

Behavior

to

Communit

Knowledg

Practice

Social

Communit

y partners

Attitudes

Action

y partners,
Conduct
workshop

Skills
Opinions
Aspiration
s

Social
change,
Civic duty

FINAL PLAN

15

Methods
This report was designed to ascertain the purpose of University at Buffalo students
having Advocacy in the Community program workshop series in the Intercultural and Diversity
Center. The Likert Scale will be the instrument we use in this assessment. The data will be
collected next semester at the beginning and end of each workshop. In each case, the Likert Scale
will be sent to all students who register electronically on UBLinked to attend and they respond
online. Later, the scale will be sent again to both groups of students who participating and who
choose not to. Im hoping for a high response rate for the study to be 100 percent. The
Universitys Intercultural and Diversity Center Director should approved this study.
Measurement: Likert Scale
Likert scaling is a bipolar scaling method, measuring either positive or negative response
to a statement. Using this unidimensional scaling method is useful because it is one-dimensional
in nature. For this pre-test and post-test, I choose to use an even-point scale, where the middle
option of "neither high nor low" is not available. (Trochim, 2006) As in all scaling methods, the
first step is to define what needs to be measured. I created the set of potential scale items
including confidence, understanding, happiness, intelligence, engagement, motivation,
appreciation, and likelihood for them to be rated on a 1-to-5 High-Low response scale.
Generally, the next step is to have a group of judges rate the items to help compute the
correlations between all pairs of items. In addition, I would consult with a colleague and the
Director of Intercultural and Diversity Center. In making judgements about which items to retain
for the final scale, it is best to throw out items that have a low correlation with the total
(summed) score across all items.

FINAL PLAN

16

For each item, get the average rating for the top quarter of judges and the bottom quarter.
Then, do a t-test of the differences between the mean value for the item for the top and bottom
quarter judges. (Trochim, 2006) Each respondent is asked to rate each item on some response
scale. This scale is a forced-choice response scale with an odd number of responses and no
middle neutral or undecided choice. In this situation, the respondent is forced to decide whether
they lean more towards the high or low end of the scale for each item. The final score for the
respondent on the scale is the sum of their ratings for all of the items.

Sample: Advocacy in the Community Scale

Please rate the following items on a scale of 1-5 in the following categories, 5=highest and
1=lowest
1. Confidence in ability to define advocacy:
1
2
3
2. Understanding of your role as an advocate:
1
2
3
3. Intelligence level of social issues in the Buffalo community:
1
2
3
4. Motivation to participate with community organizations:
1
2
3
5. Appreciation for culture and heritage of discussed population:
1
2
3
6. Likelihood of participating in this event again:
1
2
3

Survey Design
For the design of the survey, I used quasi-experimental design specifically the
nonequivalent groups design. In its simplest form, it requires a pretest and posttest for a treated
and comparison group. It is structured like a pretest-posttest randomized experiment for the
purposes of the intact groups that is similar as the treatment and control groups. In education, we

FINAL PLAN

17

might pick two comparable classrooms or schools. I tried to select groups that are as similar as
possible so I can fairly compare the treated one with the comparison one. For the showing of the
results, the chosen form is on a bivariate distribution in the simple pre-post, two group study. The
treated case, students who registered on UBLinked who attended, can be indicated with As while
the comparison cases, students who registered on UBLinked who did not attend, are indicated
with Bs. By looking at the graph, you should be able to see the treatment effect. The program
group (A) should consistently score better on the posttest than the comparison group (B) do. If
positive scores on the posttest are "better" then we can conclude that the program improved
things. It is important to keep in mind when looking at the internal validity is that the groups
differed before the program. Hence, the students do not have the same background in terms of
education and life experiences. In some cases, the bivariate distribution can show that the groups
may have had an initial advantage and that the positive results may be due in whole or in part to
this initial difference.
Possible Outcomes
Possible Outcome #1
Let's take a look at several different possible outcomes from a NEGD to see how they might be
interpreted. The important point here is that each of these outcomes has a different storyline.
Some are more susceptible to treats to internal validity than others. Before you read through each
of the descriptions, take a good look at the graph and try to figure out how you would explain the
results. If you were a critic, what kinds of problems would you be looking for? Then, read the
synopsis and see if it agrees with my perception.
Sometimes it's useful to look at the means for the two groups. The figure shows these means with
the pre-post means of the program group joined with a blue line and the pre-post means of the

FINAL PLAN

18

comparison group joined with a green one. This first outcome shows the situation in the two
bivariate plots above.
Here, we can see much
more clearly both the
original pretest
difference of five
points, and the larger
fifteen point posttest
difference.
How might we
interpret these results? To begin, you need to recall that with the NEGD we are usually most
concerned about selection threats. Which selection threats might be operating here? The key to
understanding this outcome is that the comparison group did not change between the pretest and
the posttest. Therefore, it would be hard to argue that that the outcome is due to a selectionmaturation threat. Why? Remember that a selection-maturation threat means that the groups are
maturing at different rates and that this creates the illusion of a program effect when there is not
one. But because the comparison group didn't mature (i.e., change) at all, it's hard to argue that it
was differential maturation that produced the outcome. What could have produced the outcome?
A selection-history threat certainly seems plausible. Perhaps some event occurred (other than the
program) that the program group reacted to and the comparison group didn't. Or, maybe a local
event occurred for the program group but not for the comparison group. Notice how much more
likely it is that outcome pattern #1 is caused by such a history threat than by a maturation
difference. What about the possibility of selection-regression? This one actually works a lot like

FINAL PLAN

19

the selection-maturation threat If the jump in the program group is due to regression to the mean,
it would have to be because the program group was below the overall population pretest average
and, consequently, regressed upwards on the posttest. But if that's true, it should be even more
the case for the comparison group who started with an even lower pretest average. The fact that
they don't appear to regress at all helps rule out the possibility the outcome #1 is the result of
regression to the mean.
Possible Outcome #2
Our second hypothetical outcome presents a very different picture. Here, both the program and
comparison groups gain from pre to post, with the program group gaining at a slightly faster rate.
This is almost the definition of a selection-maturation threat. The fact that the two groups
differed to begin with suggests that they may already be maturing at different rates. And the
posttest scores don't do anything to help rule that possibility out. This outcome might also arise
from a selection-history threat. If the two groups, because of their initial differences, react
differently to some historical event, we might obtain the outcome pattern shown. Both selectiontesting and selection-instrumentation are also possibilities, depending on the nature of the
measures used. This pattern could indicate a selection-mortality problem if there are more lowscoring program cases that drop out between testings. What about selection-regression? It doesn't
seem likely, for much the same reasoning as for outcome #1. If there was an upwards regression
to the mean from pre to post, we would expect that regression to be greater for the comparison
group because they have the lower pretest score.

FINAL PLAN

20

Possible Outcome #3
This third possible
outcome cries out
"selectionregression!" Or, at
least it would if it
could cry out. The
regression scenario is
that the program group was selected so that they were extremely high (relative to the population)
on the pretest. The fact that they scored lower, approaching the comparison group on the posttest,
may simply be due to their regressing toward the population mean. We might observe an
outcome like this when we study the effects of giving a scholarship or an award for academic
performance. We give the award because students did well (in this case, on the pretest). When we
observe their posttest performance, relative to an "average" group of students, they appear to
perform a more poorly. Pure regression! Notice how this outcome doesn't suggest a selectionmaturation threat. What
kind of maturation
process would have to
occur for the highly
advantaged program
group to decline while a
comparison group
evidences no change?

FINAL PLAN

21

Possible Outcome #4
Our fourth possible outcome also suggests a selection-regression threat. Here, the program group
is disadvantaged to begin with. The fact that they appear to pull closer to the program group on
the posttest may be due to regression. This outcome pattern may be suspected in studies of
compensatory programs -- programs designed to help address some problem or deficiency. For
instance, compensatory education programs are designed to help children who are doing poorly
in some subject. They are likely to have lower pretest performance than more average
comparison children.
Consequently, they are
likely to regress to the
mean in much the
pattern shown in
outcome #4.
Possible Outcome #5
This last hypothetical
outcome is sometimes
referred to as a 'cross-over" pattern. Here, the comparison group doesn't appear to change from
pre to post. But the program group does, starting out lower than the comparison group and
ending up above them. This is the clearest pattern of evidence for the effectiveness of the
program of all five of the hypothetical outcomes. It's hard to come up with a threat to internal
validity that would be plausible here. Certainly, there is no evidence for selection maturation here
unless you postulate that the two groups are involved in maturational processes that just tend to
start and stop and just coincidentally you caught the program group maturing while the

FINAL PLAN

22

comparison group had gone dormant. But, if that was the case, why did the program group
actually cross over the comparison group? Why didn't they approach the comparison group and
stop maturing? How likely is this outcome as a description of normal maturation? Not very.
Similarly, this isn't a selection-regression result. Regression might explain why a low scoring
program group
approaches the
comparison group
posttest score (as in
outcome #4), but it
doesn't explain why
they cross over.
Although this fifth
outcome is the
strongest evidence for a program effect, you can't very well construct your study expecting to
find this kind of pattern. It would be a little bit like saying "let's give our program to the toughest
cases and see if we can improve them so much that they not only become like 'average' cases, but
actually outperform them." That's an awfully big expectation to saddle any program with.
Typically, you wouldn't want to subject your program to that kind of expectation. But if you
happen to find that kind of result, you really have a program effect that has beat the odds.
http://www.socialresearchmethods.net/kb/quasnegd.php
Data Analysis
Before analyzing the data, I would have to make sure the data values are consistent. At
the level that the data is measured, I can describe the characteristics of these variables in terms of

FINAL PLAN

23

their central tendency (that is, the most common, middle, or average value) and their dispersion
(their spread or variation relative to the central tendency). The traditional ways to measure
central tendency include the mode (the most frequently occurring values or value), the median
(the value in the middle when the observations are ranked from smallest to largest), and the mean
(the arithmetic average calculated by adding up all the data values and dividing that sum by the
number of observations). Dispersion is measured most commonly but the range (the maximum
value minus the minimum value), the variance (the sum of the squared distances of each
observation from their mean divided by the number of degrees of freedom, which is the number
of independently varying differences and which equals the sample size minus one for a single
variable), the standard deviation (the positive square root of variance), and the coefficient of
variation (the standard deviation divided by the mean, multiplied by 100, which permits more
balanced comparisons of variables with substantially different units of measurement).
The statistical model that we would intuitively expect could be used in this situation
would have a pretest variable, posttest variable, and a dummy variable variable that describes
which group the person is in. These three variables would be the input for the statistical analysis.
We would be interested in estimating the difference between the groups on the posttest after
adjusting for differences on the pretest. This is essentially the Analysis of Covariance
(ANCOVA) model as described in connection with randomized experiments (see the discussion
of Analysis of Covariance and how we adjust for pretest differences)
Repeated measures is a special type of MANOVA in which the same thing is measured at
different times or there are multiple measures of the same trait. The simplest form of repeated
measures is a pre-test-post-test comparison. Main effects and covariates may be added to these
purely within-subjects designs to produce between-subjects repeated measures MANOVA or

FINAL PLAN

24

MANCOVA models. An important condition that must be satisfied for repeated measures results
to be satisfied, at least without appropriate adjustments, is sphericity, or constant correlations
among values of the same variable measured repeatedly over time or among different measures
of the same trait.
Because this is a nonequivalent design, I made the groups nonequivalent on the pretest by
adding five points to each program group person's pretest score. Then, I added 15 points to each
program person's posttest score. When we take the initial 5-point advantage into account, we
should find a 10 point program effect. The bivariate plot shows the data from this simulation. I
then analyzed the data with the ANCOVA model. Remember that the way I set this up I should
observe approximately a 10-point program effect if the ANCOVA analysis works correctly. The
results are presented in the table. In this analysis, I put in three scores for each person: a pretest
score (X), a posttest score (Y) and either a 0 or 1 to indicate whether the person was in the
program (Z=1) or comparison (Z=0) group. The table shows the equation that the ANCOVA
model estimates. The equation has the three values I put in, (X, Y and Z) and the three
coefficients that the program estimates.

FINAL PLAN

25
References

Ballard,P.J.(2014).Whatmotivatesyouthcivicinvolvement?.JournalofAdolescentResearch,
0743558413520224.
Bowman,N.A.,&Park,J.J.(2015).NotAllDiversityInteractionsareCreatedEqual:
CrossRacial Interaction, Close Interracial Friendship, and College Student
Outcomes.ResearchinHigherEducation,121.
Denson,N.,&Bowman,N.(2013).Universitydiversityandpreparationforaglobal
society:theroleofdiversityinshapingintergroupattitudesandcivicoutcomes.Studies
inHigherEducation,38(4),555570.
Einfeld,A.,&Collins,D.(2008).Therelationships betweenservicelearning,socialjustice,
multicultural competence, and civic engagement.Journal of College Student
Development,49(2),95109.
Ginwright,S.,&James,T.(2002).Fromassetstoagentsofchange:Socialjustice,organizing,
andyouthdevelopment.Newdirectionsforyouthdevelopment,2002(96),2746.
Laird,T.F.N.(2005).Collegestudentsexperienceswithdiversityandtheireffectson
academic selfconfidence, social agency, and disposition toward critical
thinking.ResearchinHigherEducation,46(4),365387.
Sax,L.J.(2004).CitizenshipdevelopmentandtheAmericancollegestudent. NewDirections
forInstitutionalResearch,2004(122),6580.
Schuh,J.H.(2009).Assessmentmethodsforstudentaffairs.SanFrancisco,CA:JosseyBass.
Trochim,W.M.K.(2006,October20).Researchmethodsknowledgebase.Retrievedfrom
http://www.socialresearchmethods.net/kb/index.php

FINAL PLAN

26

Woolworth,S.(2014).YouthAdvocacy.TheEncyclopediaofTheoreticalCriminology.
Yee,S.M.(2008).DevelopingtheFieldofYouthOrganizingandAdvocacy:What
FoundationsCanDo.NewDirectionsforYouthDevelopment,(117),109124.