VanderWey, S.

Cooper, R.
Turner, K.
(April 10, 2015)

Evidence-Based Program Design

presented at
The Association of Experiential education
North west regional conference, Stanwood
WA

ELBOW UP!
What are you doing, or what would you like
to be doing, to evaluate your programs?

Scott Vanderwey

Director of Adventure Education

Washington State University Extension

Robby Cooper
Clinical Assistant Professor
Center for Transformational Learning & Leadership
Washington State University

Ken Turner
Challenge Course Manager
Seattle Parks and Recreation

Learning Objectives
1. Understand key elements of a
successful evaluation tool.
2. Gain new tools and techniques for
developing evidence-based programming.
3. Understand current trends in
educational research.
4. Create an implementation plan for new ideas.

A Land Grant
Research
University

ADVENTURE EDUCATION:
EXPERIENTIAL EDUCATION
METHODOLOGIES
Natural Resource Education
The Learning Communities
Model: Building Successful
Learning Communities (BSLC)
Camp Long Challenge Program

Recommend Sequence

low
mult course:
hou iple vis single t
it
rs +
o
lunc s (typic
ally
h)
4

Mt
C
with halle
trap belay nger H
ez e ed
ig
jum climb h Cou
p
ing rse
and

Enc
Cou hantme
rse
n
with ts High
zi p
line
s

Logic Model

Evaluation
Development

The beginning

We know it works

How can we show it?

The Development Process

Not just creating a survey
1. Goals
2. Timeline
3. Outcomes
4. Measures
5. Procedures
6. Training
7. Implementation
8. Analysis
9. Revision
10.Dissemination

Goals
Process
Am I running my program well?
Outcome
Is my program making an impact?

Outcomes
Adventure Education
Outcomes
Targeting Life Skills
Model

Program Outcomes

* = self-efficacy

Measures
Concerns
Reliability
Validity

Measures
Communication

(Klein, et al. 2006;

Adolescent Health)

Decision-making

(Klein, et al. 2006;

Adolescent Health)

Teamwork (Annett, et al., 2000; Ergonomics)

Self-efficacy (Sherer, et al., 1982;
Psychological Reports)

Procedures
What can we realistically and reasonably
accomplish that will provide a valid & reliable
measurement?
Time
Resources
Qualifications
administration
analysis

Setting
Ethics

Procedures
Notice to Parents sent out with
program registration materials
WSU EXTENSIONCHALLENGE COURSE EVALUATION
WSU EXTENSIONCHALLENGE COURSE EVALUATION
INSTRUCTIONSFOR FACILITATOR
INSTRUCTIONSFOR FACILITATOR
Facilitators,
Facilitators,

Facilitator administers surveys

using packet containing:
Facilitator Instructions
Recruitment Script
Processing Page
Surveys
Envelope

Thank you for beinga part of theprogramevaluation process. This survey is a

Thank you for beinga part of theprogramevaluation process. This survey is a
crucial part of our continuingefforts to improveour programs and servethepeople
crucial part of our continuingefforts to improveour programs and servethepeople
of Washington.
of Washington.
Instructionsfor completingdatacollection:
Instructions for completingdatacollection:
Open Interdepartmental Mail envelope. Insideyou will find thenoticeto parents, a
Open Interdepartmental Mail envelope. Insideyou will find thenoticeto parents, a
recruitment script, a processingpage, pretest surveys, posttest surveys and another
recruitment script, a processingpage, pretest surveys, posttest surveys and another
envelope.
envelope.
1. Noticeto Parents: this was sent to parents prior to theprogram. It is thereso
1. Noticeto Parents: this was sent to parents prior to theprogram. It is thereso
that you can seewhat parents werenotified of when enrollingtheir children in
that you can seewhat parents werenotified of when enrollingtheir children in
the program.
the program.
2. Recruitment Script: this is thescript you will useto introducethesurvey. It is
2. Recruitment Script: this is thescript you will useto introducethesurvey. It is
there to make theprocess as simpleas possiblefor you. Pleasemakesureto read
there to make theprocess as simpleas possiblefor you. Pleasemakesureto read
it to theprogram participants it contains information they need to know in
it to theprogram participants it contains information they need to know in
order to satisfy requirements of the Officeof Research Protections. Failureto
order to satisfy requirements of the Officeof Research Protections. Failureto
informparticipants could result in disciplinary action.
informparticipants could result in disciplinary action.
3. ProcessingPage: this is thesheet that you will fill out and includewith the
3. ProcessingPage: this is thesheet that you will fill out and includewith the
completed surveys. Pleaseread it carefully and fully completeeach itemon the
completed surveys. Pleaseread it carefully and fully completeeach itemon the
page. It is an important part of the evaluation process.
page. It is an important part of the evaluation process.
4. Pretest Surveys: thesearethesurveys you will ask the participants to complete
4. Pretest Surveys: thesearethesurveys you will ask the participants to complete
BEFORE theprogram. Surveys aremarked PRETESTin thefooter at the
BEFORE theprogram. Surveys aremarked PRETESTin thefooter at the
bottomof thepage.
bottomof thepage.
5. Posttest Surveys: theseare thesurveys that you will ask theparticipants to
5. Posttest Surveys: theseare thesurveys that you will ask theparticipants to
completeafter theprogramhas concluded. Surveys aremarked POSTTESTin
completeafter theprogramhas concluded. Surveys aremarked POSTTESTin
the footer at the bottomof thepage.
the footer at the bottomof thepage.
6. Envelope: Onceall surveys arecompleted, placetheProcessingPage, Pretest
6. Envelope: Onceall surveys arecompleted, placetheProcessingPage, Pretest
Surveys, Posttest Surveys in the envelopeand seal it. This envelopewill be
Surveys, Posttest Surveys in theenvelopeand seal it. This envelopewill be
placed back into theInterdepartmental Mail envelopeand sent to Robby Cooper
placed back into theInterdepartmental Mail envelopeand sent to Robby Cooper
at theWSU Pullman campus (mail code4852) for data entry and analysis.
at theWSU Pullman campus (mail code4852) for data entry and analysis.
If youhaveanyquestionsregardingthesurveyprocess,pleasecontact:
If youhaveanyquestionsregardingthesurvey process,pleasecontact:
Robby Cooper, PhD
Robby Cooper, PhD
Department of Human Development, WSU
Department of Human Development, WSU
Johnson Tower 601C
Johnson Tower 601C
robby.cooper@wsu.edu
robby.cooper@wsu.edu
Office: 509-335-2442
Office: 509-335-2442
Mobile: 814-571-7262
Mobile: 814-571-7262

Procedures
Pretest Posttest Design
12 16 item pretest
21 25 item posttest
(satisfaction, demographics)

WSU EXTENSIONCHALLENGE COURSE EVALUATION

WSU EXTENSIONCHALLENGE COURSE EVALUATION
INSTRUCTIONSFOR FACILITATOR
INSTRUCTIONSFOR FACILITATOR
Facilitators,
Facilitators,
Thank you for beinga part of theprogramevaluation process. This survey is a
Thank you for beinga part of theprogramevaluation process. This survey is a
crucial part of our continuingefforts to improveour programs and servethepeople
crucial part of our continuingefforts to improveour programs and servethepeople
of Washington.
of Washington.
Instructionsfor completingdatacollection:
Instructions for completingdatacollection:
Open Interdepartmental Mail envelope. Insideyou will find thenoticeto parents, a
Open Interdepartmental Mail envelope. Insideyou will find thenoticeto parents, a
recruitment script, a processingpage, pretest surveys, posttest surveys and another
recruitment script, a processingpage, pretest surveys, posttest surveys and another
envelope.
envelope.
1. Noticeto Parents: this was sent to parents prior to theprogram. It is thereso
1. Noticeto Parents: this was sent to parents prior to theprogram. It is thereso
that you can seewhat parents werenotified of when enrollingtheir children in
that you can seewhat parents werenotified of when enrollingtheir children in
the program.
the program.

Completed surveys are sealed

in an envelope and mailed to
WSU Pullman campus.
Data is entered and analyzed
by faculty and graduate
assistants.

2. Recruitment Script: this is thescript you will useto introducethesurvey. It is

2. Recruitment Script: this is thescript you will useto introducethesurvey. It is
there to make theprocess as simpleas possiblefor you. Pleasemakesureto read
there to make theprocess as simpleas possiblefor you. Pleasemakesureto read
it to theprogram participants it contains information they need to know in
it to theprogram participants it contains information they need to know in
order to satisfy requirements of the Officeof Research Protections. Failureto
order to satisfy requirements of the Officeof Research Protections. Failureto
informparticipants could result in disciplinary action.
informparticipants could result in disciplinary action.
3. ProcessingPage: this is thesheet that you will fill out and includewith the
3. ProcessingPage: this is thesheet that you will fill out and includewith the
completed surveys. Pleaseread it carefully and fully completeeach itemon the
completed surveys. Pleaseread it carefully and fully completeeach itemon the
page. It is an important part of the evaluation process.
page. It is an important part of the evaluation process.
4. Pretest Surveys: thesearethesurveys you will ask the participants to complete
4. Pretest Surveys: thesearethesurveys you will ask the participants to complete
BEFORE theprogram. Surveys aremarked PRETESTin thefooter at the
BEFORE theprogram. Surveys aremarked PRETESTin thefooter at the
bottomof thepage.
bottomof thepage.
5. Posttest Surveys: theseare thesurveys that you will ask theparticipants to
5. Posttest Surveys: theseare thesurveys that you will ask theparticipants to
completeafter theprogramhas concluded. Surveys aremarked POSTTESTin
completeafter theprogramhas concluded. Surveys aremarked POSTTESTin
the footer at the bottomof thepage.
the footer at the bottomof thepage.
6. Envelope: Onceall surveys arecompleted, placetheProcessingPage, Pretest
6. Envelope: Onceall surveys arecompleted, placetheProcessingPage, Pretest
Surveys, Posttest Surveys in the envelopeand seal it. This envelopewill be
Surveys, Posttest Surveys in theenvelopeand seal it. This envelopewill be
placed back into theInterdepartmental Mail envelopeand sent to Robby Cooper
placed back into theInterdepartmental Mail envelopeand sent to Robby Cooper
at theWSU Pullman campus (mail code4852) for data entry and analysis.
at theWSU Pullman campus (mail code4852) for data entry and analysis.
If youhaveanyquestionsregardingthesurveyprocess,pleasecontact:
If youhaveanyquestionsregardingthesurvey process,pleasecontact:
Robby Cooper, PhD
Robby Cooper, PhD
Department of Human Development, WSU
Department of Human Development, WSU
Johnson Tower 601C
Johnson Tower 601C
robby.cooper@wsu.edu
robby.cooper@wsu.edu
Office: 509-335-2442
Office: 509-335-2442
Mobile: 814-571-7262
Mobile: 814-571-7262

Training & Implementation

Clear Procedures for
Implementation Fidelity
Realistic Expectations
Frequent Check-ins and Revision

Analysis
Scale

Reliability
Reliability of Measures

(Chronbachs Alpha)*

Version 1

Communication

.30

Decision-making

.48

Teamwork

.34

Self-efficacy

.33

*Averaged pretest / posttest Alpha

Alpha > .70 considered sufficient

Version 2

Revision
Procedural
Fidelity
-training
-audience

Realistic
Expectations
-survey

Analysis
Scale

Reliability
Reliability of Measures

(Chronbachs Alpha)*

Version 1

Communication

.30

Decision-making

.48

Teamwork

.34

Self-efficacy

.33

*Averaged pretest / posttest Alpha

Alpha > .70 considered sufficient

Version 2

Analysis
Scale

Reliability
Reliability of Measures

(Chronbachs Alpha)*

Version 1

Version 2

Communication

.30

.69

Decision-making

.48

.79

Teamwork

.34

.68

Self-efficacy

.33

.61

*Averaged pretest / posttest Alpha

Alpha > .70 considered sufficient

Results

Camp Long, Seattle, WA

Sample: 462 participants
Satisfaction
84% of participants said they
enjoyed
their experience at Camp Long

Results
August 2014 October 2014
N

Mean
Score
Change

Sig.
(p)

Effect Size
(d)

Communication

399

.172

5.99

.000*

.22**

Decision-making

462

.075

2.92

.004*

.12

Self-efficacy

197

.152

4.52

.000*

.26**

Teamwork

462

.103

4.32

.000*

.15

*Statistical significance: p < .05

** Small effect size: d > .2

Dissemination

Presentations
Trainings
Publications

Qualitative Comments from

Survey
Word Cloud showing
qualitative comments
from what was the
best part of your day
post question.

Learning Objectives
1. Understand key elements of a
successful evaluation tool.
2. Gain new tools and techniques for
developing evidence-based programming.
3. Understand current trends in
educational research.
4. Create an implementation plan for new ideas.

Contact
Information

WSU Extension
(253) 445-4581
Scott VanderWey
vanderwey@wsu.edu
Robby Cooper
robby.cooper@wsu.edu
Ken Turner
keno.turner@seattle.gov