Article Critique

EDPS 612 Program Evaluation

Shelina Hassanali
June 25, 2014

Article Critique 1
The importance of early intervention programs for social-emotional development has been
discussed quite in depth in the literature. This critique will focus on one such article by Cooke et al
(2007) on the impact of the Second Step violence prevention program on the behaviour of elementary
students.
The authors provide a succinct yet clear description of the population with which their study
was undertaken. The sample size was 741 students in grades three to five, from six different schools,
and was representative of the larger population. The researchers used a multi-component methodology
of evaluation, including student self-report questionnaires, independent behavioural observations, and
review of discipline referrals. To provide the reader with a good level of background information, the
authors provided a strong literature review which explained the importance of social-emotional
programs as well as discussed previous research on the Second Step program and the inconsistencies
raised by the said research. Clear outcomes were provided for the study, including (1) high
implementation fidelity; (2) strong teacher and administrator buy-in and support; (3) high levels of
community involvement and support; and (4) the provision of intensive, ongoing training and technical
support (Cooke et al, 2007). The authors were sure to consider developmental aspects (i.e. not including
grade one and two students due to difficulty completing self-reports) and confounding variables (i.e. not
including grade five students as they were already receiving another violence prevention program).
Assessments used in this study were implemented in a timely fashion, both immediately before
the intervention and immediately after. The psychometrics of the various student self-report measures
were discussed as well, and for the most part, the psychometrics seem sound. However, one of the
measures had an internal consistency lower range of 0.47, indicating low reliability. While it is not
common to see studies as large as this one in which each student is observed individually at baseline
and again post-intervention, this study did observe each child for five minutes. While this is beneficial,
the question can be raised as to how much information one can actually observe in a five minute period.
The researchers explain that the intervention was put into place with these students, including
extensive teacher training, high intervention fidelity, and students receiving the full Second Step
curriculum. They confirmed that the results of the study support their hypothesis, and provided the data
to support this claim. There were, however, some data which conflicted their hypothesis, including the
fact that impulse control actually decreased during the course of the intervention. The study also failed
to demonstrate a statistically significant decrease in violent behaviour and this is an important
consideration due to the fact that Second Step is a violence prevention program. The results section is
detailed however, and discusses reasons for these findings.
One of the strengths of this study was the large sample size and inclusion of diverse populations
in the study. Another strength was that the researchers attempted to approach the evaluation from a
different angle than other studies, as they focused heavily on teacher training and buy-in, thus
evaluating the program based on how it is supposed to be implemented. The researchers also utilized
sound methodology and presented the research in a clear, easy-to-follow manner.
There are some limitations to this study. The biggest of these is the lack of a control group this
has major implications. While over 700 students received the Second Step intervention, there was no
control group with which to compare the intervention group. Therefore, as indicated by the authors,
there is no real way to be certain that it is the intervention which caused the change in some of the
behaviour which was studied. If there was a control group composed of similar students, a causal
relationship between the intervention and change in behaviour would be better supported. In the
current research, there is no way to know the impact of other factors such as student maturity over the
year, school culture, school curriculum. Other limitations also exist, including poor psychometric
properties of at least one of the measures used, as well as the extremely short time-frame in which each
student was observed. The current research presents a decent model of program evaluation for the
Second Step program. The researchers discovered some support for the usefulness of the program, but
like previous evaluations of the program, the results are somewhat conflicting. Future evaluation of this
program could benefit from having a control group, utilizing self-report measures which all have high
psychometric properties and improved student observation procedures. Future evaluation of the
Second Step program should also incorporate some strengths of this particular study, including high
teacher involvement/training and a large sample size, as well as the multi-modal method of evaluation.

Article Critique 2
It is often the case that educational settings look for promising new programs to implement with
their students in hopes of creating better outcomes. This critique will focus on a program evaluation
conducted by the creator of a relatively recent remedial reading program called Discover Reading.
The program being evaluated in this article was created in 1994 by Stephen Truch. The article
opens with a short literature review which includes some quite dated research; the article was written in
2004, however some of the research presented in the area is from the 1980s. It would have perhaps
been beneficial to provide a more in-depth literature review with recent citations. At the beginning of
the article, it is claimed that the Discover Reading program might be the most comprehensive
commercial program yet developed (Truch, 2004, p.10). This is quite the claim, and leads the discerning
reader to examine the research with a critical eye.
The author explains the methodology in quite some detail, which is helpful. The present study
had a sample size of 155 clients at the authors clinic, most of whom were between the ages of 6-16,
with nine clients being over the age of 17. While the size of the sample is good, data was collected over
a two year time period which some may argue is a bit long. The author states that no attempt was made
to diagnose the participants as dyslexic or learning disabled, nor was any attempt made to determine
severity of the clients presenting reading difficulty. The explanation for this is that the clinic is meant to
provide a service and not to conduct research; therefore, it would be difficult to collect and analyze
data. It can be argued, however, that determining severity of the presenting reading concerns is
imperative to any research in the area of remedial reading programs and therefore the research should
have been conducted in an environment which would allow for this analysis. The sample also included
students with pre-existing diagnoses of ADHD, Aspergers, FASD, and Autism. Again, the data was not
analyzed to reflect these differences and the differential impact the Discover Reading program had on
these clients. Without such analysis, there is no way to know of the usefulness of the program on
students with diverse needs.
The pre and post-test measures used in this study are discussed in detail, and a wide range of
measures were used. The author used the CREVT-2 and WISC-3 vocabulary subtests which is a positive
point as these are well established assessment measures. Also, during the course of the 2 years, newer
versions of the above tools were available, but the author used the older versions for consistency.
However, the author also utilized many informal tools for visual memory, auditory segmenting and
auditory blending which he created himself. The use of informal, self-created tools is problematic
because there is no information available regarding the psychometric properties of these tools and
therefore it is not possible to know how valid and reliable these measures are. Additionally, sound
scientific research should be easily replicable by others to glean similar results. In this case, it would be
difficult to replicate the results because of lack of access to many of the tools used.
The actual intervention is discussed in this evaluation as well as the procedure. Participants in
the study attended the clinic to receive the program on an intensive individual basis, with students
attending for four hours on Saturdays and two hours on two weekdays, for a total of eight hours per
week. It is important to consider that while this is feasible in a clinical setting, it would be virtually
impossible for every student in a classroom to receive eight hours per week of individual interventions.
Clinicians who were providing the intervention were trained in the program and are reported to have
diligently followed the procedures. Again, in a school setting, one must consider the impact of
intervention fidelity; in other words, how diligently and consistently staff would implement the program
and how this may impact results. Finally, the author states that no control group was utilized in this
study. Therefore, there is no way to compare the results of the group who received the intervention to
those who did not and consequently, no way to determine causality of the intervention to the results.
This evaluation includes tables which show the results of the intervention, including pre and
post-test data and statistical information for each of the aspects of reading analyzed. The author states
that significant gains were made, and that these are attributable to the intervention. It is also stated
that many studentsdo not immediately show large gains in fluency, but re-assessments done
months and in some cases years later on the same student, usually show much larger fluency gains. Its
as if time is needed for some of the process to percolate and then to finally surface as fluent reading.
(p.26). This statement is extremely problematic because student gains in reading fluency months and
even years after the intervention is completed cannot simply be attributed to the Discover Reading
intervention without considering other factors such as the effects of further practice and learning with
other programs, student maturity over the years, and many other confounding factors.
Strengths of this study include a decent sample size, detailed explanation of various aspects of
reading and of the intervention itself, the use of subtests from established assessment tools and the
consistency with which these tools were used over the course of the study. Unfortunately, the
limitations seem to outweigh the strengths. Future evaluations of this program should keep these
limitations in mind. For example, research and data collection should occur over a shorter period of
time. Also, the research should include measures which assess the severity of the reading issues pre-
test, and should also include analysis of the data as it pertains to clients with a multitude of pre-existing
diagnoses. Formal, established assessment tools with sound reliability and validity should be utilized in
the future instead of informal, self-made tools with no psychometric data. A control group should be
utilized, and the program should be evaluated in an educational setting to determine effectiveness
outside of the clinic. Special consideration should be given to intervention fidelity and the realistic
implementation of the program in schools, where individual interventions for eight hours per week will
likely be problematic. The impact of confounding variables should also be considered in more detail in
future research. Finally, careful examination of all of these factors must be considered prior to selecting
Discover Reading or any remedial reading program for use in an educational setting.
References
Cooke, M.B., Ford, J., Levine, J., Burke, C., Newell, L., Lapidus, G. (2007). The effects of city-wide
implementation of "second step" on elementary school students' prosocial and aggressive behaviors.
Journal of Primary Intervention, 28(2), 93-115.
Truch, S. (2004). Remedial Outcomes with Different Reading Measures. Poster presentation,
International Dyslexia Conference. Retrieved from
http://www.readingfoundation.com/uploads/documents/reading-outcomes.pdf