JAMAL BERNHARD, MELISSA MELLISSINOS LERNHARDT,
AND ROSE MIRANDADECKER 
Evaluating


Instructional Materials  
IN 1996, WE WERE ASKED TO EVALUATE A new set of mathematics instructional materials for the middle school classroom. The instructional unit included a wide assortment of materials, so we decided to begin with some explicit criteria on which to base our evaluation. We turned to several sources, including the National Council of Teachers of Mathematics (NCTM) and local school districts, seeking evaluation guidelines that were in the spirit of the NCTM's Standards documents (1989,1991,1995). Every set of evaluation criteria that we examined seemed incomplete to use as the set of criteria, because each one missed elements that we thought should be considered by an indepth evaluation. The elements most commonly ignored include these: • Assessment within
the materials: The need for more authentic assessment practices has received
a great deal of attention, yet the JAMAL BERNHARD, jbernhard@cnnse.sdsu. edu, is interested in undergraduate mathematics education and preservice elementary and middle school teacher Preparation. MELISSA MELLISSINOS LERNHARDT, mmelliss@crmse.sdsu.edu, teaches mathematics to Preservice elementary and middle school teachers and is interested in how students make sense of data. She and Bernhard both teach at San Diego State University, San Diego, CA 92182. ROSE MIRANDADECKER, rmdecker@sdeoe.kl2.ca.us, teaches mathematics to high school students in San Diego. She is interested in secondary and postsecondary education and educational leadership. 
criteria that we examined did not seem to reflect this concern. • The use of technology: Although we found criteria for evaluating an individual piece of technology, none of the criteria focused on the role that technology plays within a particular instructional unit. • The practicality of the unit: An evaluation should consider whether the materials can be implemented effectively by teachers working in a typical school environment of long hours, crowded classrooms, inadequate funding, and the like, yet no set of criteria that we examined addressed this issue. After searching carefully for a comprehensive set of criteria, we decided to develop our own set of criteria covering the elements that need to be considered in mathematicscurriculum evaluation. We integrated ideas and suggestions from several resources, including the NCTM's Standards documents (1989, 1991, 1995), a variety of books and articles about mathematics assessment (Romberg 1995; Romberg and Wilson 1995; Swan 1993), and our own experiences as mathematics teachers and mathematicseducation researchers. Our different educational and vocational backgrounds often led to different ideas and interpretations, so we discussed and clarified the meanings that many of these documents held for us. During the development of our criteria, we paid special attention to language, trying to make each criterion specific enough to be concrete and meaningful yet broad enough to be applicable to a wide variety of classroom contexts. 
page #2 
We divided our criteria into four areas of evaluationcontent, technology and instructional tools, assessment, and teacher supportkeeping in mind that these areas overlap to some degree. The content criteria cover the mathematical subject matter, its organization and structure, and the activities in which students engage. The technology and instructionaltools criteria focus on how these tools operate, their flexibility and appropriateness for the given classroom, the learning opportunities that they afford students, and the practical issues related 
to their implementation. The assessment criteria deal with such issues as integrating assessment into the unit, using assessment as a learning tool, and determining the authenticity of assessment practices. Finally, the teachersupport criteria focus on the degree of support that the materials provide for teachers and the empowerment of teachers to make curricular and classroom decisions. We did not develop a rating scale to quantify our criteria because important decisionmaking information might become lost in the numbers. Instead, we 
photograph by Robert Queer III; ALL RIGHTS RESERVED
page #3 
chose a qualitative framework. Although this approach is likely to take more time than simply rating each criterion numerically, a qualitative approach encourages evaluators to think more deeply about the potential strengths and weaknesses of the instructional materials; it is thus worth the extra effort. We follow our evaluation criteria with some brief notes on how they might be used most effectively. For consistency, all the criteria have been phrased as yesno questions, but they certainly are not intended to be answered as such. Rather, each criterion is meant to be a guiding question for thinking about the strengths and weaknesses of the materials; to this end, one might imagine the phrase "To what extent . . ." at the start of each criterion. Also, these criteria are not meant to be of equal importance. Depending on the unit and the context in which they will be used, some criteria may be less pertinent, whereas others may take on added importance. The presence or absence of technology is one such example. Criteria for Evaluating Content The mathematics • Covers appropriate content. Do the materials appropriately cover and emphasize topics listed in the NCTM's content standards? Are they at an appropriate level for the targeted age group? Do they contain accurate and current information? • Challenges all students. Does the mathematical content challenge and expand the knowledge of students from diverse backgrounds and at different levels of understanding? • Connects subject areas. Do the materials embed mathematics content in realworld contexts and connect mathematics to other subject areas? Are connections made among mathematics topics? Is the content in line with the general educational goals of the classroom? The organization and structure • Develops concepts clearly. Are mathematical ideas clearly introduced and reinforced with examples and multiple representations, such as diagrams, graphs, and tables? Is vocabulary clearly developed; explained; and reinforced with examples, a glossary, and an index? • Incorporates technology andlor other instructional tools effectively. Do the materials employ appropriate instructional tools, such as calculators or manipulatives, that can engage students in mathematical reasoning and promote understanding? 
• Aligns with school context. Are the organization and structure of the materials conducive to the particular school environment, for example, special equipment needed, teacherpreparation time, and length of unit? • Allows flexibility. Can teachers tailor the materi als to their particular classroom environment? Student work • Develops reasoning skills. Do ample opportunities exist for students to develop their reasoning skills, for example, deductive, inductive, and spatial? • Develops communication skills. Do the materials promote classroom discourse in which students can listen to, respond to, and question the teacher and one another? Are students encouraged to communicate mathematically, both orally and in writing, and to use a variety of tools, such as diagrams, concrete materials, and technology, within the context of classroom discourse? • Develops problemsolving skills. Do a variety of robust problem situations encourage students to explore mathematics? • Grounds activity in meaninglul situations. Is student work situated in contexts that are meaningful and interesting to students from all backgrounds? • Promotes equity. Do the materials accurately and positively portray students of diverse backgrounds, including race, ethnicity, sex, health, age, religion, and social class? The evaluator should consider such issues as familiarity of content to all students, language used, inclusion of multicultural origins of mathematics, and multiple ways of knowing and doing mathematics. • Promotes positive values. Are students encouraged to respect and value one another's ideas, ways of thinking, and mathematical dispositions? Do the materials promote humanitarian issues, such as respect for the environment and the rights and property of others? Criteria for Evaluating Technology and
Technology • Runs smoothly. Is the technology free of programming bugs or other major problems? • Sets up quickly. Is the technology intuitive, well designed, and easy to use so that preparation time in the classroom is reasonable?

page #4 
• Stimulates interest Does the technology motivate students by using such features as quality graphics and sound? • Promotes equity. Does the technology reflect individual and cultural diversity? Does it approach knowing and doing mathematics from multiple perspectives? • Reflects technology in society. Does the technology help students understand the role and importance of technology in the real world? • Develops communication skills. Does the technology promote mathematical discussion within the classroom by representing students' mathematical ideas in a way that can be interpreted and discussed by others? Does it lend itself to working in pairs or small groups? • Incorporates assessment. Does the technology give frequent~ and qualitative feedback to the teacher and students? Does it give students the opportunity to evaluate their work and reflect on what they have learned? Other instructional tools • Uses instructional tools appropriately. Are the instructional tools appropriate for the mathematics being learned, that is, are they the best learning tools for the task; and are they used in a way that can help students construct appropriate mathematical knowledge? • Allows flexibility. Can the instructional tools be modified or controlled by the teacher to be integrated coherently into the rest of the curriculum and to suit the particular classroom environment? • Contains adequate documentation. For more complex instructional tools, are complete and clear instructions provided for both teachers and students? • Coincides with school context. Can the instructional tools be implemented realistically in the particular school environment? For example, does every student need a computer? Criteria for Evaluating Assessment Mathematics and learning • Involves appropriate activities. Do the assessments engage students in realistic and worthwhile mathematical tasks? Do they promote the use of important and correct mathematics? • Yields a variety of information. Do the assessments allow for inferences about what students understandtheir mathematical knowledge, their thinking processes, and their dispositions? 
• Integrates with instruction. Are the assessments an integral part of instruction? Do they build on students' understanding, interests, and experiences? • Encourages learning. Do the assessments offer opportunities for students to apply mathematical knowledge to new situations and to evaluate, reflect on, and improve their own work and the work of others? Equity • Promotes equity. Do the assessments encourage and acknowledge multiple ways of demonstrating knowledge and abilities? • Allows flexibility. Are the scoring guides flexible enough to accommodate student responses that are reasonable but unanticipated? Assessment practices
• Lends coherence. Do assessments fit together in a cohesive way that gives an overall representation of students' understanding? Do the assessments match the learning objectives of the instructional materials? • Engages students. Do the assessments present opportunities for students to interact with one another and be active decisionmakers? Do they encourage selfassessment? • Coincides with school context. Can the assessment tasks be done within available school time? Can teachers evaluate students' responses to tasks within a reasonable amount of time? Criteria for Evaluating Teacher Support Teacher decisionmaking • Empowers teachers to improve classroom discourse. Do the materials encourage teachers to make decisions about classroom discourse, such as when to provide information to students; when to clarify an issue; when to let students struggle with a problem; and when to have students

page #5 
justify their ideas, both orally and in writing? Do the materials prompt teachers to monitor students' participation in classroom discussions and to encourage each student to participate actively in class? Empowers teachers to make curricular decisions. Can teachers make decisions about the content covered, the work students do, and the types of assessment used? Do the materials encourage teachers to make decisions about altering, adding, or deleting materials so that they can pursue issues raised by students during discussions and so that the curriculum can reflect students' mathematical interests? Teacher support • Provides adequate instructional materials. Are the instructional materials clear and helpful enough so that teachers can plan and prepare for lessons in a reasonable amount of time? Do they help teachers acquire the knowledge needed to pose tasks that will engage students in important and interesting mathematical reasoning, especially with regard to the effective use of technology or manipulatives? • Supplies adequate support materials. Do the materials include such support items as extrapractice assignments, challenging assignments for advanced students, assessments, extra examples, solutions, multicultural and multilingual material, warmup exercises, software with documentation, pacing charts, lesson planners, and a list of resource materials? • Makes suggestions. Do the materials offer suggestions on how to engage students in deep mathematical thinking through the use of group learning, technology, marripulatives, and other important instructional strategies? Notes for Use TAKING THE TIME TO ADEQUATELY ASSESS INstructional materials on the basis of all of these criteria can be a daunting task. One possibility is to team up with others by having different people evaluate different aspects of the curriculum. The three of us, for example, came from quite different backgrounds: one was teaching in the local school district, one had worked for several years in curriculum and assessment, and one had gained expertise in educational technology; so it was natural for us to split the workload by having each of us focus on the criteria with which we were most familiar. Each of us still offered input to other aspects of the 
evaluation, but division of labor made the task of evaluating our unit much more manageable. Because our evaluation was part of a course assignment, our report took the form of a paper. This format may not be the best way to report such an evaluation when deciding whether a unit is appropriate for an actual classroom. One possibility is to organize the evaluation by subsectionsfor instance, "The Mathematics," 'ne Organization and Structure," and "Student Work"rather than by the individual criterion. Evaluating each criterion individually, then creating a long list may simply be too overwhelming. Instead, evaluators might read through the criteria for each subsection and make a list of the unit's pros and cons with respect to that subsection. In this way, the evaluator uses the specific criteria in the evaluation but may be more likely to recognize broader issues as they arise. Conclusion THESE CRITERIA REFLECT BOTH THE VISION FOR school mathematics expressed in the NCTM's Standards documents (1989, 1991, 1995) and other important considerations in implementing instructional materials in today's classrooms. Although we designed this set of criteria for use at the middle school level, it can be adapted to any level of schooling. We encourage educators to use these criteria as a starting point from which to develop their own set of criteria that reflects more closely the context of their own educational environment. References National Council of Teachers of Mathematics (NCTM). Curriculum and Evaluation Standards for School Mathematics. Reston, Va.: NCTM, 1989. . Professional Standards for Teaching Mathematics. Reston, Va.: NCTM, 1991. . Assessment Standards for School Mathematics. Reston, Va.: NCTM, 1995. Romberg, Thomas A., ed. Reform in School Mathematics and Authentic Assessment. Albany, N.Y.: State University of New York Press, 1995. Romberg, Thomas A., and Linda D. Wilson. "Issues Related to the Development of an Authentic Assessment System for School Mathematics." In Reform in School Mathematics and Authentic Assessment, edited by Thomas A. Romberg, 118. Albany, N.Y.: State University of New York Press, 1995. Swan, M. "Improving the Design and Balance of Mathematical Assessment." In Investigations into Assessment in Mathematics Education, edited by M. Niss, 195216. Boston, Mass.: Kluwer, 1993.6L 