RUNNING HEAD: On the Development of Reflective Lessons FROM FORMAL EMBEDDED ASSESSMENTS TO REFLECTIVE LESSONS: THE DEVELOPMENT OF FORMATIVE

ASSESSMENT SUITES

Carlos Ayala Sonoma State University RichardJ.Shavelson StanfordUniversity

Paul Brandon University of Hawaii YueYin UniversityofHawaii

Erin M. Furtak Max Planck Institute for Human Development

Maria Araceli Ruiz-Primo University of Colorado at Denver and Health Sciences Center

Donald Young University of Hawaii

Miki Tomita

On the Development of Reflective Lessons Stanford University and University of Hawaii

On the Development of Reflective Lessons Abstract The idea that formative assessments embedded in a curriculum could help guide teachers toward better instructional practices that lead to student learning has taken center stage in science assessment research (Atkins & Coffey, 2003; Black & Wiliam, 1998). In order to embed formative assessments in a curriculum, curriculum developers and assessment specialists need to collaborate to create these assessment tasks. This paper describes the development of the formal embedded formative assessment suites and implementation plans that were designed for the Romance Study using a Science Achievement Framework. It describes the fundamental shift away from summative assessment scripts to reflective lessons scripts. Samples of the assessment tasks and implementation plans are described along with the rationale for why these tasks were selected and where these tasks were placed in the curriculum. Finally we conclude about how to successfully embed formative assessments in new or existing curriculum and how to help teacher use these assessments. For example, we point out the critical importance of collaboration, of professional development aimed at enabling teachers to reconceptualize the role of assessments in their teaching, of linking formative assessments to overall goals, and of providing a learning trajectory as reference for teachers to locate students ideas developmentally and provide feedback accordingly.

On the Development of Reflective Lessons Introduction While some empirical evidence suggests that the use of curriculum-embedded (formative) assessment leads to increased learning (Bell & Cowie, 2001; Black & Wiliam, 1998; 2004; Shepherd, 2000), how these formative assessments are designed and used by curriculum developers and then eventually implemented by teachers is poorly understood. Moreover, assessment specialists and curriculum developers rarely collaborate on assessment development, let alone on embedding formative assessments in existing curricula, on preparing end-unit summative assessments, or in thinking about how teachers might use these assessments. The Romance project, described by Shavelson et al. (this issue), attempted to fill this knowledge gap. Here we describe how we went about building, refining and embedding formative assessments into an inquiry science curriculum. Most noteworthy is the use of an interdisciplinary Assessment Development Team that laid out a blueprint for formative assessment definition and development. We report the fruits of this collaboration, and how they matured over time. We then describe how we trained teachers to use formative assessments in their inquiry science teaching. Finally, we draw conclusions about these activities with the goal of informing collaborative assessment and curriculum developers with knowledge of how to successfully embed formal formative assessments in a new or existing curriculum and of the challenges of helping teachers use this emerging technology. Background

On the Development of Reflective Lessons We define the activities that we developed in this project as embedded formal formative assessments. Formal because we crafted assessment tasks which would be available for teachers to use at critical times in curriculum implementation; this contrasts with on-the-fly and planned-for formative assessment that capitalize on informal ongoing clinical observations or create teachable moments for enhancing students understanding (see Shavelson et al., this issue). They are embedded assessments because they are inserted into a curriculum to be used at a particular time as opposed to the end of a unit. They are formative assessments because they are developed to give a snap shot to students and teachers about what students know and are able to do at a particular time such that this information could be used to make teaching decisionswhat does the teacher or student do next (see Shavelson et al., this issue, for details). We chose a curriculum for the project, Foundational Approaches in Science Teaching (FAST), because the Stanford Educational Assessment Laboratory (SEAL) had collaborated previously with Curriculum Research & Development Group at the University of Hawaii (CRDG), the curriculum developers; because previous studies have supported FASTs efficacy (Pauls, Young, & Lapitkova, 1999; Tamir & Yamamoto, 1977; Young, 1983;1993); and because various national organizations have found it to be an exemplary program (U. S. Department of Educations Expert Panel on Mathematics and Science Education, 2001; National Staff Development Council, 1999; for more information, See Shavelson et al., this issue) The content for this project focused on the conception of why things sink and float that was built up from an understanding of mass, volume, density and relative

On the Development of Reflective Lessons density (see Table 1). This curriculum develops students science understandings incrementally in a manner that parallels how science knowledge was developed in the Western world (cf. King & Brownell, 1966), and as such, students understandings of why things sink and float are developed via explanations of sinking and floating phenomena sequentially beginning with the concepts of mass and volume and moving to the concepts of density and relative density. --------------------------Insert Table 1 Here --------------------------Assessment Development A science achievement framework guided the Assessment Development Team in the conceptualization, development, and evaluation of embedded and end-of-unit assessments. We begin by describing the framework and then turn attention to the Assessment Development Team, which turned out to be a very important, collaborative infrastructure for the project. Science Achievement Framework The projects assessment development was guided by conceptual framework for science achievement (see Shavelson et al., this issue). From the framework, we

conceived and developed embedded and end-of-unit assessments. That is, we conceived of science achievement as, in significant part, the acquisition of and reasoning with knowledge. More specifically we conceived of science achievement as comprised of (at

On the Development of Reflective Lessons least) four different but overlapping types of knowledge and reasoning: declarative, procedural, schematic and strategic (see Figure 1 in Shavelson et al., this issue). Declarative knowledge and reasoning is knowing thatfor example, knowing that force is a push or pull and light is a form of energy and reasoning with this knowledge. Declarative knowledge includes scientific definitions and facts, mostly in the form of terms, statements, descriptions, or data. For instance, a statement like,

combining two or more materials together forms a mixture is the scientific definition of mixture. A scientific fact would be, for example, the density of water is 1 gram per milliliter at 4 degrees centigrade and at one atmosphere of pressure. Procedural knowledge and reasoning is knowing how. For example, knowing how to design and reason about a study that manipulates one relevant variable and controls others, or how to measure the density of an object or how to graph the relation between the angle of an incline plane and the force needed to move an object up it. Procedural knowledge includes if-then production rules or a sequence of steps, in the form of actions or steps that can be carried out to achieve a certain goal leading to task completion. In a broad sense, students statements, such as, If I have a mixture like gravel and water, I will use a screen to separate them, or After I pour the mixture on the screen, the water goes through the screen and the gravel is left on it, reflect their procedural knowledge of separating mixtures. Schematic knowledge and reasoning is knowing why. For example, knowing why Delaware has a change of seasons or knowing why we see different phases of the moon, and reasoning with this knowledge. To know why is to have a scientifically

On the Development of Reflective Lessons justifiable theory or model or conception that explains the physical world. 1 Schematic knowledge includes principles, schemes, and mental models. Schematic

knowledge can be used to interpret problems, to troubleshoot systems, to explain what happened, and to predict the effect that changes in some concepts will have on other concepts (De Kleer & Brown, 1983; Gentner & Stevens, 1983). For instance, the

schematic knowledge students may develop when they learn about mixtures and solutions includes: (a) The method used to separate mixtures (screening or filtering) is determined by the size of the mixtures particles; (b) evaporation is needed to separate a dissolved material like salt from a liquid; or more generally, (c) evaporation can be used to separate a material from a liquid because the liquid is changed into the gas state. And strategic knowledge and reasoning is knowing when, where and how to use and reason with certain types of knowledge in a new situations. Strategic knowledge includes domain-specific conditional knowledge and strategies such as planning and problem-solving as well as monitoring progress toward a goal. People use strategic knowledge to recognize the situations where some procedures can be carried out, to examine the features of tasks in order to decide what schematic knowledge can be applied, to set task goals, or to control and monitor cognitive processing. Our research has linked certain types of assessment to this science achievement framework. Briefly put, to measure the structure of declarative knowledge, multiplechoice, short-answer and concept maps provide valid evidence (Ruiz-Primo & Shavelson, 1996a; Shavelson & Ruiz-Primo, 1999). To measure procedural knowledge, performance assessments are appropriate (e.g., Ruiz-Primo & Shavelson, 1996b; Shavelson, Baxter, &
1

A theory or model often called a mental model (Gentner & Stevens, 1983).

On the Development of Reflective Lessons Pine, 1992). To measure schematic knowledge, multiple-choice and short-answer items, and performance assessments are appropriate. Strategic knowledge is difficult to measure directly but is essential, especially with novel assessment tasks. The CRDG participants confirmed that the knowledge framework fit well with FAST, except for schematic knowledge, which was not addressed in the unit that was to be the focus of the study (see Brandon, et al., this issue). Assessment Development Team The Assessment Development Team guided the conceptualization, development and evaluation of the embedded and end-of-unit assessments. The Team consisted of Stanford assessment specialists and researchers and CRDG curriculum developers, FAST trainers and teachers, and researchers. (The collaboration of the Assessment Development Team towards the implementation of this study is more fully covered in Shavelson et al. and Brandon et al., this issue.) Once formed, the Team went through four major tasks to establish rapport, develop a common language, identify assessment targets, and begin assessment development. First, the team reviewed the project background, formative and summative assessments, the science achievement framework for assessment development and corresponding assessment methods, the project goals and each members responsibilities. Second, the team reviewed the curriculum for the study, FAST Physical Science (PS) investigations 1 to 14. To this end, the curriculum developers provided abbreviated hands-on demonstrations of PS 1-14 while other Team members participated as students. The team discussed what FAST teachers typically used as assessments and student

On the Development of Reflective Lessons corresponding responses. In addition, a SEAL member presented storyboards showing the types of knowledge that were addressed in each of the PS 1-14 investigations. The groups discussions about the lessons helped the team members understand the science achievement framework and helped clarify the purpose of the investigations. Thirdly, the Assessment Development Team developed a goal statement for the entire unit: Students individually and in groups, will be able to explain why things sink and float using relationships between mass, volume and density; develop, carry out, report and defend scientific investigations of buoyancy and develop, carry out, report. The ADT then discussed possible assessments, the knowledge types, target difficulty levels, and the match of the proposed assessments with the curriculum. Many assessment items discussed here ended up in the studys final assessment suites. We refer to the collection of assessment items and prompts given at a particular point in the curriculum implementation as an assessment suite (e.g., the multiple-choice, graphing and lab performance assessment suite at FAST investigation 4). What and where to embedded assessment? SEAL researchers developed a comprehensive assessment blue print for the PS 1-14 investigations using the science achievement framework described above. Using the comprehensive blueprint, the Assessment Development Team identified the most important end-of-unit concepts taught in the 14 investigations to be used in a posttest assessment suite, and identified the points (natural joints) in the instructional sequence in which the formative assessments were to be embedded. The team came up with three criteria to identify the natural joints: (1) a subgoal of the end-of-unit goal is achieved, that is, there is a body of knowledge and

10

On the Development of Reflective Lessons skills sufficiently comprehensive to be assessed; (2) teachers need to know about student understanding before they proceed with further instruction; and (3) feedback to students is critical to help them improve their understanding and skills of the material already taught (Shavelson, SEAL & CRDG, 2005, p. 6). Four embedded-assessment natural joints were identified in the 14 investigation sequence (Figure 1). In this curriculum the natural joints were clearly identifiable because students move from learning about sinking and floating phenomena using mass and then move on to another set of investigations that are focused on the next concept, volume. For example, a natural joint in the sequence of physical science investigations occurred between investigations 4 and 5 where students move away from examining why things sink and float using mass holding volume constant and begin using volume holding mass constant (Table 1). Deciding what to assess in the embedded and in the end-of-the-unit assessments was not as straight forward as finding the natural joints. In curriculum that typically takes eight to ten weeks to implement, there are many important concepts and procedures that students learn. Since these important topics are clearly identifiable to the curriculum developers all of these topics became equally important to them as assessment targets. However, it would impossible for the embedded assessments or the end of unit assessments to cover all of the topics and so decisions had to be made as to how to select the most important ones. For example, in the identification of the nine concepts to be used in the concept map, a SEAL assessment developer collected and then presented a large list of terms to the curriculum developers and asked them to identify the most important terms that could be used to in a concept map assessment. The curriculum

11

On the Development of Reflective Lessons developers selected all the terms because all the terms were important. In the case of the terms to be used in the concept map, the Assessment Development Team narrowed down the concept list by focusing on the overall goal of the unit and then asked, which terms are the most important for explaining why things sink and float? This list narrowing still generated a large number of concepts that was then whittled down in pilot testing using empirical information. Additionally, a FAST teacher and trainer on the team expressed concern about the length of time the assessments would add to the unit. One team member felt that student progress might be slowed by the embedded assessments. In response to this concern, a SEAL team member noted that the assessments should help teachers decrease their teaching time, because they will know the students deficiencies and as such the teacher would more efficiently deliver information to the students. The tensions between time spent on instruction vs. assessment, and depth vs. coverage of the assessments remained a concern throughout the project. Once the important concepts had been selected and the joints identified, the assessment specialists of the Assessment Development Team began an iterative process of designing and refining assessments based on the science achievement framework, piloting of assessments and validation with the rest of the Team. The first complete iteration of the assessment tasks identified four locations in the curriculum (excluding the pre-post tests) where formative assessments were to be placed (Figure 1). We reduced the list again because of the amount of time it would take to complete the assessments. The final selection of the assessments attended to the main goal of the unit.

12

On the Development of Reflective Lessons ----------------------Insert Figure 1 ----------------------Assessment refinement. The Assessment Development Team reviewed the first complete version of the assessments and began a process of refining them to match the purpose of the project and intent of the curriculum. We exclude a detailed discussion here of the pre and post-tests because we are focused on the embedded assessments (see Yin et al., this volume). The Team reviewed the assessments by discussing and/or carrying out the assessments. As assessment development and refinement continued, the assessments went through multiple iterations. Once all the embedded assessments were reviewed and discussed, the Team decided to make several important modifications to the project. First the team decided that the project should only focus on the 12 physical science lessons in FAST. Second the team had to make hard decisions about what not to include in the embedded assessments. For example, the first version of the assessments contained performance tasks for students. These tasks included using a balance, using metric rulers, measuring volume, etc. A series of passport assessments was created to assess these performances. The passport was a student record book where information is stored about whether or not a student passed a performance goal (e.g., Can Mary mass the rock? If so then give her a stamp in her Passport book). The Assessment Development Team dropped these passports for several reasons. The information that was collected although important for investigations was indirectly related to the units goal: explaining Why

13

On the Development of Reflective Lessons things sink and float. Moreover, the information about student performance could be verified in the conduct of hands-on investigations. Finally, the amount of teacher time and effort to carry out these assessments outweighed the value of the information collected. This focus on explaining Why things sink and float was significant because it foreshadowed larger changes to come. The Assessment Development Team also dropped the concept map from the summative tests and moved the concept maps after investigation 6 and investigation 11. The purpose for this was to reduce the amount of time in the pre- and posttests, to reduce the amount of time at each major embedded assessment joint, and to ameliorate an expected pre-post testing effect with the concept maps. From Embedded Assessments to Reflective Lessons Pilot Study. The Assessment Development Team carried out a pilot study with the embedded assessments. Three teachers were trained in the use of the embedded assessments and asked to use these with their classes during the projects second year. The pilot study focused on the teachers implementation of the embedded assessments, student performance on these assessments, and professional development. We trained the pilot study teachers by reviewing the curriculum, carrying out the embedded assessments with the teachers as students, and then having the teachers use the embedded assessment with students who were involved in the FAST investigations in summer school. The pilot studys findings suggested that the embedded assessments should be 1) short in duration and tightly focused on the key outcomes of the unitexplaining why things sink and

14

On the Development of Reflective Lessons float based on relative density; 2) allow for immediate feedback to teacher and students; 3) provide opportunities for students to test their why-things-sink-and-float explanations with evidence from the assessment event and to take a stand on what they believe; and 4) set the stage for the next set of investigations The studys findings also suggested that teachers treated the embedded assessments just like any other test they might give. For example, the teachers would review the material covered in the unit before the embedded assessments even though the purpose of the formative assessments was to do just that. The teachers treated the embedded assessments as external to the unit of instruction and did not use the assessments to inform their teaching. The teachers who participated in this pilot study had already been trained in FAST and knew that the assessments were added to the curriculum after the fact. This may have lead to the assessment externalization. However, in later studies we found the assessment externalization in non-FAST teachers. Furthermore, pilot study teachers often would provide feedback to students weeks after the assessment thus missing the teachable moments provided for by the embedded assessmentsnone of which was intended by the project staff. It became clear that how these assessments were to be used in the classroom was very important and that teachers preconceived notions about assessments influence their implementation. These pilot findings led to significant changes in the embedded assessments. Overall, the teachers believed these embedded assessments to be summative assessments and would revert from formative assessment pedagogy to a summative assessment teaching script (Shavelson, 1986). A summative assessment teaching script can be

15

On the Development of Reflective Lessons conceptualized as a formalized teaching pattern consisting of a set of expectations of what events are necessary and about the temporal order of such events. A summative assessment teaching script might include studying for the tests, taking practice tests, or reviewing lecture notes with the students prior to actually giving the test. In order to avoid the usual summative assessment teaching scripts, we changed the name from embedded assessments to Reflective Lessons. The Reflective Lessons also evolved from assessment activities to learning activities intended to provide instructional information to both the student and the teacher by: 1) building on what students already know; 2) attending to student conceptions and misconceptions; 3) making student conceptions and misconceptions public and observable; 4) priming students for future learning, and 5) reflecting on material covered. Other pilot study evidence collected suggested that teachers needed increased structure in order to use the Reflective Lessons and, as such, detailed information about how to use the teachable moments should be provided with the Reflective Lessons. For example, teachers were able to elicit student conceptions about why things sink and float, but they did not necessarily use these conceptions to further student learning. This shift from assessment activities to learning activities represents a fundamental change for teachers in the way to look at the formative assessment practices. Finally, pilot-study evidence suggested the need to reduce the number of Reflective Lessons. If left untouched, the Reflective Lessons would have taken a teacher about 15 lesson periods to complete (see Furtak et al., this issue). The Assessment Development Team removed some assessments because they believed that regular

16

On the Development of Reflective Lessons classroom practices allowed for formative assessment of students declarative and procedural knowledge and because they believed that too much time was taken away from the regular science curriculum. The tension between time spent on assessments and time spent on instruction as if the assessment were not instructional continued even though the Team began to move towards believing that these assessment were instructional. Consequently, the Team set the goals of adding no more than two lesson periods at the natural joints and one period each for the concept maps. Further, the Team set a goal of simplifying the structure of the assessments by identifying two reflective lesson scripts. These Reflective Lessons and scripts were used in the final Romance experiment.

17

On the Development of Reflective Lessons Reflective Lessons The Reflective Lessons were composed of a carefully designed sequence of investigations (prompts) that enabled teachers to step back at key points during an instruction unit (natural conceptual joints) to check student understanding, and to reflect on the next steps that they must take to move forward (Figure 2). Furthermore, these assessments were carefully designed to match the content and structure of the existing FAST investigations. The SEAL team developed drafts of the assessments and, using talk-aloud methods, refined them in tryout-refine-tryout cycles conducted with students from a local school near Stanford and the University of Hawaii CRDG Laboratory School. Reflective Lesson Goals In order to promote student understanding of why things sink and float via relative density concepts, the Reflective Lessons were designed to elicit and make public student sinking-and-floating conceptions, encourage communication and argumentation based on evidence from the investigations or assessment tasks, push on students why things sink and float conceptions, help students track their sinking and floating conceptions and help students reflect about these concepts (e.g., Duschl, 2003). Elicit and make students conceptions public. The Reflective Lessons were intended to make students thinking about different physical phenomena explicit (e.g., Asking students, Why things sink and float?). In order to achieve this purpose, Reflective Lessons included a set of activities/investigations that bring forth students conceptions of density and why things sink and float (e.g., having students predict

18

On the Development of Reflective Lessons whether a small-sized high density plastic block will sink or float and explain why). Moreover, they included different suggested teaching strategies, such as student group work, small and/or large group discussions, sharing activities, and questions and prompts that help teachers reveal students conceptions and thinking and make these conceptions public. Encourage communication and argumentation of ideas using evidence. We designed the Reflective Lessons to provide opportunities for students to discuss and debate what they know and understand. By having students take a stand and make their why-things-sink-and-float conceptions public, students could be provided with the opportunity to see competing student conceptions, and hear and evaluate the supporting evidence provided for the different competing views. That is, Reflective Lessons were concerned with what data counts as evidence for students (Uncle Joes tales of the sea vs. results of an investigation), how this evidence is used to support their predictions, decisions and explanations, and how these explanations could be generalized to other similar situations, that is, the universality of the scientific principles. For example, can the notion that more bbs means more sinking be applied in all sinking and floating events? The use of evidence for explanations in the Reflective Lessons was an extension of the pedagogical practices already found in FAST. Furthermore, the Reflective Lessons provided teachers with scripts to help students support their why-things-sink-and-float explanations based on scientifically sound evidence and help test the universality of student claims. That is, the Reflective Lessons provided for examples and models for setting up the lessons, collecting student

19

On the Development of Reflective Lessons conceptions, establishing discussion events, and providing questions to push students evidence-based why things sink and float explanations: How do you know that? What evidence do you have to support your explanations? In what cases does your explanation apply and what cases does it not apply? Such discussion allows students to think about how evidence should be used to generate and justify explanations as well as think about the universality of their explanations. Push on students why-things-sink-and-float conceptions. The Reflective Lessons were also designed to push on students ideas about why things sink and float. In the Reflective Lessons, students are presented with problems that seem to be buoyancy anomalies, but rather the problems provide instances where everyday knowledge about sinking and floating events cannot be easily applied. For example, the Reflective Lessons asked students to predict whether a large-sized low-density plastic block (polypropylene) or a small-sized lightweight high-density plastic block (PVC) would sink or float. Students focused on the size and weight of the two blocks (decisions based on everyday life) rather than on the density of the plastic (decisions based on scientific evidence). Help students track why-things-sink-and-float conceptions. The Reflective Lessons were also designed in part to track student why-things-sink-and-float conceptions. That is, in the Reflective Lesson suites there were assessment items that were used each time and that allow for teacher and students to track student understanding. With the goal of tracking students conceptual development, each Reflective Lesson included the Why Things Sink and Float prompt and the same concept

20

On the Development of Reflective Lessons terms were used in both Reflective-Lesson concept map administrations. By comparing earlier with later assessments, student progress could be observed. Reflect with students about their conceptions. The Reflective Lessons were intended to help teachers pinpoint students conceptual development at key instructional points, and help guide the development of students understanding by identifying where students were going wrong. Through making student conceptions public and then through discussions and debate based on evidence, teachers and students find the problems they are having as they move towards the unit goals. Furthermore, the Reflective Lessons provided teachers with strategies for progress that can help students achieve these goals. Connect to the curriculum. The Reflective Lesson goals were derived directly from the FAST curriculum. The curriculum developers stated the importance of these types of learning activities (i.e., encourage argumentation about evidence, push on student buoyancy conceptions, reflect on students buoyancy conceptions) in their instructional materials about group work, class discussion, looking for universality of student claims. What the reflective lessons did, is to make the whole reflective discussion explicit in terms of what to do with students, when to do it with students and how to do it with students. Types of Reflective Lessons In order to accomplish these goals, Reflective Lessons provided specific prompts that have proved useful for eliciting students conceptions, encouraging communication and argumentation based on evidence, and helping teachers and students reflect about

21

On the Development of Reflective Lessons their learning and instruction. These prompts vary according to where the Reflective Lessons are embedded within the unit. In order to simplify the Reflective Lessons, two types were identified. Type I Reflective Lesson The Type I Reflective Lessons were designed to expose students conceptions of why things sink and float, to encourage students to bring evidence to support their conceptions, and to raise questions about the universality of their conceptions when applied to new situations. The prompts and questions focus students on the use of evidence to support their conceptions of why things sink and float. These lessons were embedded at three joints in the unit, after investigations 4, 7 and 10 (Figure 2). Each Type I Reflective Lesson has four prompts for students to: (1) interpret and evaluate a graph, (2) predict-observe-explain an event related to sinking and floating, (3) answer a short question, and (4) predict-observe an event related to sinking and floating. Graph Prompt. This prompt asks students to use knowledge of investigations, resulting data and explanations they have collected in different FAST investigations as evidence to support their responses and conclusions (Figure 3). Students are familiar with the data representation used because the scatter plots are like those created by the students in their FAST investigations and reveal the important relationship between two variables that were just studied. This prompt requires students to interpret, evaluate and complete graphs that focus on the variables involved in sinking and floating specific to the FAST investigations that they have just completed. The Assessment Development

22

On the Development of Reflective Lessons Team chose the graphing prompts because of the importance of interpreting graphs in the curriculum in helping students use data to draw conclusions and support explanations. Graphs like these are used throughout the 12 physical science investigations in the unit. --------------------------Insert Figure 3 --------------------------Predict-Observe-Explain Prompt. This prompt get directly at schematic knowledge. It asks students to predict the outcome of a sinking-and-floating event and to justify their prediction, then observe the event, and then reconcile their predictions and observations (White & Gunstone, 1992). Students use what they have learned in the unit to help them make predictions and to explain and justify their ideas. In order to make students thinking explicit, they must write their predictions, observations, explanations and reconciliations. Students are expected to use evidence in their explanations and reconciliations. Careful administration is important for this prompt type. For example an important administration consideration is when and how to collect student predictions and evidence based explanations without affecting student reconciliationsto make ideas public and anonymous. The activities that were used in this assessment came from the literature on students understanding about density, developmental psychology literature of dimensionality (relating two variables together) in student learning, and from the repertoire of FAST and non-Fast science teachers associated with the project. These activities focus on the variables that the students have just completed in their FAST investigations.

23

On the Development of Reflective Lessons ------------------Insert Figure 4 ------------------Short-Answer Prompt. This prompt was a single question that asks students to explain why things sink and float with supporting examples and evidence (Figure 5). The same prompt is used as is same in all three Type I Reflective Lesson after investigations at 4, 7 and 10. This prompt is a direct result of the ADT work in clarifying the goal of the instructional unit. Furthermore, this prompt would eventually serve as a vehicle to track student understanding across FAST 1 implementations thus providing a link between the embedded formal formative assessments and the summative assessments (i.e., the pre and post tests). ----------------------Insert Figure 5 ----------------------Predict-Observe Prompt. A slight variation on the Predict-Observe-Explain prompt, this prompt asks students to predict and observe an event (Figure 6). These prompts are based on the FAST 1 challenge questions that are provided at the end of each natural joint investigation. These FAST challenge questions were restructured using a modified POE model and student worksheets were developed for these POs. Students are not asked to reconcile predictions and observations. POs act as a launching point for the

24

On the Development of Reflective Lessons next instructional activity of the unit in which the explanation will emerge. POs can be thought of as a springboard that motivates students to start thinking about the next investigation might be about. -----------------Insert Figure 6 ------------------Type I Reflective Lesson Implementation. The Team allowed for three class sessions for Reflective Lesson Type I implementation, although the Team expected implementation might take only two or two-and-a-half sessions (see Furtak et al., this issue). Each session was planned to take about 45 minutes or so to complete. The implementation is as follows (Figure 7): -----------------Insert Figure 7 ------------------The important components of the reflective lessons are the discussions that occur after the students have completed a few of the Reflective Lesson prompts. While the actual Reflective Lessons are important, they in themselves represent only part of what needs to happen to promote the goals of the Reflective Lessons. The discussions are key because in the discussions, the teacher makes student conceptions and reasoning public and where the relevance and importance of supporting ones explanations with evidence and pushing for universality are brought to the surface. The teachers manual for the Reflective Lessons suggests that the discussion during the first leg focuses on the

25

On the Development of Reflective Lessons information from the graph and the Predict-Observe-Explain prompt as well as what has happened in class. In the subsequent short answer question Why do things sink and float?, the discussion focuses on students conceptions and the universality of the ideas and evidence. Students are asked to extend what they know beyond the context of the Graph or Predict- Observe-Explain prompt, although both might be used as evidence. If students rely primarily on the graph or Predict- Observe-Explain prompt as evidence, the teacher is expected to push them beyond these toward universality and encourage students to use information learned in class. The last Reflective Lesson prompt, the Predict- Observe prompt, is performed in two parts. First students make a prediction based on recent class investigations, Reflective Lessons and discussions, and then they observe a demonstration that may challenge their ideas. The two parts may be carried out on the same day, or divided as shown in Figure 6 above so that part one is carried out at the end of class session two, while part two is carried out at the beginning of class session 3. Many teachers carried out the Predict-Observe prompt on one day and left the explanation and recap for the next day as an introduction to the next investigation. Type II Reflective Lesson This type of reflective lesson focuses on checking students progress in their conceptual understanding of the key concepts across the twelve investigations. Only one kind of prompt is used in this type of reflective lesson, a Concept Map. Concept maps provide evidence on how students see the relationships between ideas/concepts. The concept maps were based on the terms that make up the content of

26

On the Development of Reflective Lessons the entire series of twelve investigations. Concept maps make evident students understanding and the evolving sophistication of their thinking as their investigations progress. The concept map Reflective Lessons were inserted after Investigations 6 and 11. These Reflective Lessons are implemented in one session (Figure 8). -----------------Insert Figure 8 ------------------The first box in the implementation model reflects that students need to be trained to draw concept maps. This training takes about 30 minutes, but it needs to be done carefully and teachers need to make sure that students know how to construct concept maps in accordance with as set of rules. The second time students construct the maps they need only to be reminded of the rules and do not need the entire training. Based on the pilot study observations, the most important aspect of student interaction occurs when the students construct a group concept map. While a class concept map may be desirable, we found in our pilot study that only a handful of students participated in constructing a whole class concept map and consequently, the whole class concept map is not desirable. The sharing of main ideas benefits from looking for those terms that students have not been able to place into the concept maps, looking at the terms most often used in the group maps, and having students describe how their group concept maps were constructed. For example, students are not expected to know how to

27

On the Development of Reflective Lessons relate one of the concept map terms (density) with the other concept terms in their whythings-sink-and-float concept map until investigation 9. FAST Buoyancy Learning Trajectory While the curriculum was reviewed for natural joints, assessment blueprints were drawn and the Reflective Lessons refined, a learning trajectory for student understanding of why things sink and float using a relative density explanation was developed to guide teachers. That is, the 12 FAST investigations build a students understand of why things sink and float through a series of models. This model is evident in the curriculum and was made explicit through the development of the Reflective Lessons and the studys pre- and posttests. -----------------Insert Figure 9 ------------------That is, the students in the first three investigations work with alternative conceptions of sinking and floating events (size, number of bbs, greater or lesser amounts of something) and then in investigation 4 students focus in on mass as an important variable for sinking-and-floating events. In Investigations 5 and 6 students learn that volume, especially displaced volume, plays an important role in explaining sinking and floating events. In Investigations 4 through 6, students hold mass and volume variables independent of each other. In Investigation 7-9 students now combine both volume and mass together to explain sinking-and-floating events. In Investigation 10 (and the in part in 9), students learn about the density of an object, while in Investigation 11 students

28

On the Development of Reflective Lessons learn that liquids have density too. Finally in Investigation 12, students learn that sinking and floating events can be explained by comparing the density of an object and the density of medium in which it is placedrelative density. The learning trajectory is useful when a teacher is trying to understand where a student is in her understanding of why things sink and float. Many student responses can readily be placed in the model and a teacher armed with this knowledge, might fashion a discussion question that gets at what is important for a student to know to move forward. For example, if during the Reflective Lesson after investigation 4 a student speaks about how more sand makes the straw sink more a teacher might ask this student What do you think the bbs and sand have in common that affects why things sink and float? Knowing where a students response is in the FAST learning trajectory may help a teacher think about what a student needs to know in order to increase their understanding. Furthermore, providing teachers with this learning trajectory allowed teachers to identify where students might be in their conceptions about sinking and floating thus simplifying teachers categorization of student conceptions into five levels. Conclusions In order to create efficacious embedded formal formative assessments or Reflective Lessons in new or existing curriculum, several considerations may be drawn from this project. First, collaboration of the assessment specialists along with curriculum developers is vital. In this assessment development, the line between assessment specialists and curriculum developers was very thin and often blurred because these Reflective Lessons appeared seamless to students and teachers in terms of instructional

29

On the Development of Reflective Lessons approaches and in content. This helps ameliorate the summative assessment teaching script and student testing scripts as well as helps teachers and students make use of the information that they gather from these reflective lessons. If the assessments do not look like the other lessons it is hard to make the information connect to what the teachers and students are already and will be doing. Second, professional development should be provided to teachers to reconceptualize the value of assessment, especially formative assessment. Assessment specialists or curriculum developers cannot expect teachers to effectively use reflective lessons without training in their use of the strength of teachers summative assessment scripts which may defeat the purpose of the reflective lessons. Third, the reflective lessons must be linked to the overall goal of the curriculum and not only to the material that the students have just covered. If the reflective lessons are only linked to the material just covered, instructional time may be misused, focusing on concepts or procedures that might not be vital to the overall goal of the instructional unit (i.e., focusing on students abilities to mass an object in a why-things-sink-and-float unit). This may raise questions about the need for some lessons in established curricula. Fourth, in the development of these reflective lessons, a useful tool was the development of the why things sink and float learning trajectory. This trajectory albeit useful in the reflective lessons proved helpful beyond those lessons because teachers were expected to use the trajectory as a vehicle to track student understanding throughout unit implementation.

30

On the Development of Reflective Lessons Fifth, we lack knowledge of how teachers use reflective-lesson information (but see Furtak et al., this issue). When we consider formative assessments, assessment specialists and curriculum developers must consider all five important assessment pedagogies: knowing the content, developing the assessments, implementing the assessments, collecting information from the assessments, and most importantly, using the information gained (see Ayala, 2005). Finally, this project raises questions about the quantity and frequency of formative assessments in a curriculum. Having some embedded formal formative assessments or reflective lessons in a curriculum is useful because it reminds teachers to reflect back on what has been learned and hopefully guide future lessons toward unit goals. The more efficacious use of these assessments may reside in the application of formative assessment principles to all instructional activities (including reflective lessons). These formative assessments should be coherently focused on the overall goal of the unit rather than each minuscule instructional or assessment target.

31

On the Development of Reflective Lessons References Atkin, J. M. & Coffey, J (Eds.) (2003). Everyday Assessment in the science classroom. Arlington, VA: NSTA Press. Ayala, C, (2005, April). Assessment Pedagogies, Paper presented at the annual meeting of the American Educational Research Association, Montreal Canada. de Jong, T., & Ferguson-Hesser, M. G. M. (1996). Types and qualities of knowledge. Educational Psychologist, 31, 105-113. Bell, B., & Cowie, B. (2001). The characteristics of formative assessment in science education. Science Education, 85, 536553. Black, P.J., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in education, 5(1), 7-73. Black, P., & Wiliam, D. (2004). Classroom assessment is not (necessarily) formative assessment (and vice-versa). In M. Wilson (Ed.), Towards a coherence between classroom assessment and accountability. Chicago: University of Chicago Press. deKleer, J. & Brown, J.S. (1981). Mental models of physical mechanisms and their acquisition. In J.R. Anderson (ed.), Cognitive skills and their acquistion. Hillsdale, NJ: Erlbaum. Duschl, R. A. (2003). Assessment of inquiry. In J. M. Atkin & J. Coffey (Eds.), Everyday assessment in the science classroom (pp. 41-59). Arlington, VA: NSTA Press. Gentner, D. & Stevens, A. (1983). Mental models. Hillsdale, NJ: Erlbaum.

32

On the Development of Reflective Lessons

King, A. R. and J.A. Brownell, (1966). The curriculum and the disciplines of knowledge; a theory of curriculum practice. Wiley: New York. Li, M., & Shavelson, R.J. (2001). Examining the links between science achievement and assessment. A Paper presented at the American Educational Research Association Annual Meeting, Seattle, WA. Li, M., Ruiz-Primo, M.A., & Shavelson, R.J. (2006). Towards a science achievement framework: The case of TIMSS 1999. In S. Howie & T. Plomp (Eds.), Contexts of learning mathematics and science: Lessons learned from TIMSS. Routledge. National Research Council. (2001). Classroom assessment and the national science education standards. Washington, D.C.: National Academy Press. Pauls, J., Young, D. B., & Lapitkova, V. (1999). Laboratory for learning. The science teacher, 66, 2729. Pottenger, F. and Young, D. (1992). The Local Environment: FAST 1 Foundational Approaches in Science Teaching, University of Hawaii Manoa: Curriculum Research and Development Group. Ruiz-Primo, M. A., & Furtak, E. M. (2004). Informal assessment of students' understanding of scientific inquiry. Paper presented at the American Educational Research Association Annual Conference, San Diego, CA. London:

33

On the Development of Reflective Lessons Ruiz-Primo, M.A., & Shavelson, R.J. (1996a). Problems and issues in the use of concept maps in science assessment. Journal of Research in Science Teaching, 33(6), 569-600. Ruiz-Primo, A., & Shavelson, R. (1996b). Rhetoric and Reality in Science Performance Assessments: An Update. Journal of research in science teaching, 33(10), 10451063. Shavelson, R, Stanford Educational Assessment Laboratory (SEAL) and Curriculum Research & Development Group (CRDG). (2005). Embedding assessments in the FAST curriculum: The romance between curriculum and assessment. Final Report. Shavelson, R. J., & Ruiz-Primo, M. A. (1999). On the assessment of science achievement. (English version). Unterrichts wissenschaft, 27(2), 102-127. Shavelson, R. J., Baxter, G., & Pine, J. (1991). Performance assessment in science. Applied measurement In education, 4(4), 347-362. Shavelson, R.J. (1986). Toma de decision interactiva: Algunas reflexiones sobre los procesos cognoscitivos de los profesores. (Translation: Interactive decision making: Some thoughts on teacher cognition.) In L.M.V. Angulo (Ed.), Pensamientos de los profesores y toma de decisiones Universidad de Sevilla, Servicio de Publicaciones, Seville, Spain.

34

On the Development of Reflective Lessons Shepherd, L.A. (2000). The role of classroom assessment in teaching and learning. Santa Barbara, CA: Center for Research on Education, Diversity & Excellence, University of California, Santa Barbara. Tamir, P., & Yamamoto, K. The effects of junior high FAST program on student achievement and preferences in high school biology. Studies in Educational Evaluation, 1977,3(1), 717 White, R. and R. Gunstone. 1992. Probing understanding. Falmer Press: London, England. Young, D. B. (1982). Local science program makes good: The evaluation of FAST. Human Sciences, Technology, and Education, 1, 2328. Young, D. B. (1993). Science achievement and thinking skills. Pacific-Asian Education, 5, 3549.

35

On the Development of Reflective Lessons Figure Captions Figure 1: Assessment suite timeline. Figure 2. Final reflective lesson suites and placement. Figure 3. Reflective Lesson graph after investigation 7. Figure 4. Type I Reflective Lesson Predict-Observe-Explain prompt after investigation 7 Figure 5. Type I Reflective-lesson short-answer after Investigation 7. Figure 6. Type I Reflective Lesson Predict-Observe after investigation 7. Figure 7. Type I Reflective Lesson Suite implementation by session (adapted from SEALs Teacher guide to the reflective lessons, 2003) Figure 8. Implementation of the Type II Reflective Lesson in one session. Figure 9. FAST Buoyancy Learning Trajectory.

36

On the Development of Reflective Lessons

Figure 1:

37

On the Development of Reflective Lessons

Figure 2.

38

On the Development of Reflective Lessons

Figure 3

39

On the Development of Reflective Lessons

Figure 4.

40

On the Development of Reflective Lessons

Figure 5.

Figure 6.

41

On the Development of Reflective Lessons

FirstSession
Graph Interpretation Piece of Evidence #1
PredictObserveExplain DISCUSSION

(POE)

Piece of Evidence # 2

SecondSession

ShortAnswer WhyThingsSinkandFloat

DISCUSSION

Predict (P)

ThirdSession

Observe (O)

Next Investigation

Figure 7.

Concept Map Training

Individual Map

Group Map

Sharing Main Ideas from Group Maps

42
Figure 8.

On the Development of Reflective Lessons

Figure 9.

43

On the Development of Reflective Lessons

Table 1. First twelve FAST 1 investigations, student tasks, and learning goals (Adapted from Ruiz-Primo & Furtak, 2004) Lesson 1 Investigation Liquids and Vials Sinking Straws Student Tasks Observing vials of different liquids sinking and floating in different liquids (a buoyancy anomaly) Adding BBs to a straw and measuring the depth of sinking Graphing number of BBs versus depth of sinking Learning Goals Make scientific observations and test predictions Predict the number of BBs needed to sink a straw to particular depth Represent BB data in line graphs; more BBs more sinking Conclude that more mass more sinking Discover the relationship between the amount of ballast, the carton size and the depth of sinking Calculate the displaced volume of different cartons Graph mass vs. displace volume of floating and sinking objects Discover how a Cartesian Diver works Find the density of Cartesian divers of different masses and volumes Find the density of floating and sinking objects; density graph. Discover that different liquids have different densities Understand relative density

2 3

4 5

Graphing Sinking Straws Data Mass and the Finding the relationship between total mass Sinking Straws and depth of sinking of straws Sinking Cartons Measuring the depth of sinking of different sizes and of equal mass Volume and Sinking Cartons Floating and Sinking Objects Introduction to Cartesian Divers Density and the Cartesian Diver Density of Objects Density of Liquids Buoyancy of Liquids Finding the submerged and total volume of cartons Finding the mass and displaced volume of different objects Experimenting with Cartesian Divers

6 7 8

10 11 12

Investigating the relationship between the divers mass and volume at different sinking and floating positions Finding the relationship between mass and total volume and sinking and floating; Determining the density of liquids Experiment with different objects in liquids of different densities

44