Overview of 2005 Mathematics Assessment

• How are constructed-response questions scored?
• How are results reported?
• How is mathematics assessed?
• How is the NAEP mathematics assessment administered?
• How long does the NAEP assessment take?
• Sources
• What are the achievement level descriptions?
• What is assessed?
• Who was assessed?

Who was assessed?

• The NAEP 2005 assessment was administered to a stratified random sample of fourth-, eighth-, and twelfth-graders at the national level and to a stratified random sample of fourth- and eighth-graders at the state level.
• Both public and nonpublic school students were assessed at the national level.
• At the state or jurisdiction level, only the results for public school students are reported.
• Fifty-two jurisdictions participated, including the 50 states, the District of Columbia, and the Department of Defense Schools (Domestic and Overseas).
• National (public and nonpublic) and state (public only) samples include the following:
  • Grade 4
  • Approximately 172,000 students
  • Approximately 9,500 schools
  • Grade 8
  • Approximately 161,600 students
  • Approximately 7,200 schools
• See NAEP Sampling: Questions and Answers for additional information about sampling and Participation in NAEP 2005 by Public Schools and Students for state-specific information about the number of participating schools and students.

  Accommodated/Non-accommodated Samples

  • Prior to 1996, NAEP procedures for assessing mathematics did not permit the use of accommodations for special needs students who could not participate without them.
  • In order to make the NAEP assessment more inclusive, beginning in 1996 for mathematics (for national samples and in 2000 for state samples), new procedures were implemented to allow certain accommodations for students with disabilities and English language learners. A split-sample design was used so that both administrative procedures-with accommodations and without accommodations-could be used during the same assessment, but with different samples of students.
  • The split-sample design allows the reporting of trends across all the assessment years, as well as permitting an examination of the impact of permitting accommodations on overall results.
  • Beginning with the NAEP 2002 reading and writing assessment, NAEP used only one set of procedures-permitting the use of accommodations. This policy was continued for the 2003 and 2005 mathematics assessments.

  NAEP Mathematics Assessment
  Year	Grades	State	Sample	Accommodations Allowed
  1990	8	Yes	Entire sample	No
  1992	4, 8	Yes	Entire sample	No
  1996	4, 8	Yes	Entire sample (state)	No (state samples)
  			Split sample (national)	No for random half Yes for random half
  2000	4, 8	Yes	Split sample	No for random half Yes for random half
  2003	4, 8	Yes	Entire sample	Yes
  2005	4, 8	Yes	Entire sample	Yes

  Interpreting comparisons between accommodated and non-accommodated samples

  • Although data were collected in 1996 with accommodations permitted, results for accommodated samples are only reported for 2000, 2003, and 2005. (There was an accommodated sample in 1996 for national and results are reported. There wasn't an accommodated sample for states.)
  • Caution should be used in interpreting comparisons between accommodated and non-accommodated samples, e.g., between 1992 or 1996 results, and 2003 or 2005 results.
  • When accommodations were not permitted, students with disabilities and English language learners were not included in the assessment unless local school staff determined that they could be assessed meaningfully without accommodations. Therefore, in later assessments where accommodations were allowed, some students took the assessment that would have been excluded from previous assessments.

What is assessed?

• The NAEP Frameworks specify what is assessed and how it is to be assessed.
• The Mathematics Framework for NAEP was revised in 1996 and again in 2005. The new framework reflects current curricular emphases and objectives, while continuing a connection to previous frameworks. The connection allows the trend line at grades 4 and 8 that started with the 1990 assessment to be maintained.
• The Mathematics Framework for NAEP describes the content and format of the 2005 assessments. Revisions to the framework maintain the short-term trend, and results are comparable across all assessment years.
• The NAEP Mathematics Framework describes content in five strands [2] and the percentage of questions that should be devoted to each.
   

  Target percentage distribution of questions
  Content Strand	Grade 4	Grade 8
  Number properties and operations	40	20
  Measurement	20	15
  Geometry	15	20
  Data analysis and probability	10	15
  Algebra	15	30

  
   
• The Mathematics Framework for NAEP 2005 also describes three levels of mathematical complexity that include aspects of knowing and doing mathematics.
• Low Complexity
• Relies heavily on the recall and recognition of previously learned concepts and principles (recall, recognize, compute, perform)
• Moderate Complexity
• Involve more flexibility of thinking and choice among alternatives; require a response that goes beyond the habitual.
• High Complexity
• Require student to think in abstract and sophisticated ways; involves planning, analysis, judgment, and creative thought.
• Approximately half of the score on the assessment is based on items of moderate complexity, with the remainder of the score based equally on items of low and high complexity.

How is mathematics assessed?

• Mathematics is assessed using three types of questions:
  • Multiple-choice questions
    • make up 50 percent of the assessment, and
    • four choices are presented
  • Short, constructed-response questions
    • ".require students to give either a numerical result or the correct name or classification for a group of mathematical objects, draw an example of a given concept, or perhaps write a brief explanation for a given result." [3]
  • Extended, constructed-response questions
    • demand more than a numerical or short verbal response; and
    • require students to carefully consider a problem, plan an approach, solve the problem, and interpret their solution in terms of the original problem.
• In 2005, the distribution of types of questions across all blocks was as follows [4]:
   

  Type of Question	Grade 4	Grade 8
  Multiple-choice	111	122
  Short, constructed response	52	49
  Extended, constructed response	7	7
  Total	170	178

  
   
• Calculators were provided for about one-third of the assessment (one-third of the blocks).
• Not all items in a calculator available assessment block require the use of a calculator.
• NAEP provides the calculators:
  • Grade 4-four-function calculator
  • Grade 8-scientific calculator
  • Items in non-calculator blocks require students to demonstrate computation or estimation skills without a calculator.
• Some items use manipulatives, e.g., rulers, protractors, spinners, and geometric shapes. The manipulatives are provided by NAEP.

How are constructed-response questions scored?

• Unique scoring guides are developed for each constructed-response question.
• Scoring guides describe the specific criteria for assigning a score level for student responses.
• Extended, constructed-response questions had four- and five-level scoring guides.
• Many short, constructed-response questions had three-level guides that allowed for partial credit, while others were rated as either acceptable or unacceptable.
• Scoring process:
  • Expert scorers are extensively trained to apply the scoring criteria consistently and fairly.
  • Scoring is monitored to ensure the scoring standards are being adhered to reliably.
  • Monitoring measures the consistency of scoring to the same items administered in different assessments-therefore, ensuring consistency of the application of scoring standards across assessment years.
  • Over 4,435,831 mathematics constructed responses were scored for the 2005 assessment.

How is the NAEP mathematics assessment administered?

• Each student took two, 25-minute sets of mathematics questions (also referred to as blocks), one set of general background questions, and one set of background questions related to mathematics.
• In order to provide a comprehensive assessment of mathematics and to minimize the burden on any individual student, NAEP uses matrix sampling. Each student takes a subset of the total set of questions, i.e., two blocks out of ten total blocks per grade level.
• Because each block is spiraled with other blocks and is administered to a representative sample of students, the results can be combined to produce average group and subgroup results based on the entire assessment.

How long does the NAEP assessment take?

• No more than about 1 hour per student to actually take the assessment-about 50 minutes on mathematics questions, and a few more minutes on background questions.

How are results reported?

• Scale scores-indicate how much students know and can do
• Average scale scores and percentiles
• Range is 0-500
• Achievement levels-what students should know and be able to do.
• Subscales
• Results are analyzed and summarized by subscales that correspond to the five content strands: number properties and operations; measurement; geometry; data analysis and probability; and algebra.
• Separate subscale results are combined to create a single overall score for mathematics.

What are the achievement level descriptions?

Policy definitions of NAEP Achievement Levels:

• Basic: This level denotes partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade.
• Proficient: This level represents solid academic performance for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter.
• Advanced: This level signifies superior performance.

NAEP Achievement Level descriptions for mathematics-set separately by grade

Grade 4

• Basic: Fourth-grade students performing at the Basic level should show some evidence of understanding the mathematical concepts and procedures in the five NAEP content strands.

  Fourth-graders performing at the Basic level should be able to estimate and use basic facts to perform simple computations with whole numbers, show some understanding of fractions and decimals, and solve some simple real-world problems in all NAEP content areas. Students at this level should be able to use-although not always accurately-four-function calculators, rulers, and geometric shapes. Their written responses are often minimal and presented without supporting information.

• Proficient: Fourth-grade students performing at the Proficient level should consistently apply integrated procedural knowledge and conceptual understanding to problem solving in the five NAEP content strands.

  Fourth-graders performing at the Proficient level should be able to use whole numbers to estimate, compute, and determine whether results are reasonable. They should have a conceptual understanding of fractions and decimals; be able to solve real-world problems in all NAEP content areas; and use four-function calculators, rulers, and geometric shapes appropriately. Students performing at the Proficient level should employ problem-solving strategies such as identifying and using appropriate information. Their written solutions should be organized and presented both with supporting information and with explanations of how they were achieved.

• Advanced: Fourth-grade students performing at the Advanced level should apply integrated procedural knowledge and conceptual understanding to complex and non-routine real-world problem solving in the five NAEP content strands.

  Fourth-graders performing at the Advanced level should be able to solve complex non-routine real-word problems in all NAEP content strands. They should display mastery in the use of four-function calculators, rulers, and geometric shapes. The students are expected to draw logical conclusions and justify answers and solution processes by explaining why, as well as how, they were achieved. They should go beyond the obvious in their interpretations and be able to communicate their thoughts clearly and concisely.

Grade 8

• Basic: Eighth-grade students performing at the Basic level should exhibit evidence of conceptual and procedural understanding in the five NAEP content strands. This level of performance signifies an understanding of arithmetic operations-including estimation-on whole numbers, decimals, fractions, and percents.

  Eighth-graders performing at the Basic level should complete problems correctly with the help of structural prompts such as diagrams, charts, and graphs. They should be able to solve problems in all NAEP content strands through the appropriate selection and use of strategies and technological tools-including calculators, computers, and geometric shapes. Students at this level also should be able to use fundamental algebraic and informal geometric concepts in problem solving.

  As they approach the proficient level, students at the Basic level should be able to determine which of the available data are necessary and sufficient for correct solutions and use them in problem solving. However, these eighth-graders show limited skill in communicating mathematically.

• Proficient: Eighth-grade students performing at the Proficient level should apply mathematical concepts and procedures consistently to complex problems in the five NAEP content strands.

  Eighth-graders performing at the Proficient level should be able to conjecture, defend their ideas, and give supporting examples. They should understand the connections among fractions, percents, decimals, and other mathematical topics such as algebra and functions. Students at this level are expected to have a thorough understanding of basic-level arithmetic operations-an understanding sufficient for problem solving in practical situations.

  Quantity and spatial relationships in problem solving and reasoning should be familiar to them, and they should be able to convey underlying reasoning skills beyond the level of arithmetic. They should be able to compare and contrast mathematical ideas and generate their own examples. These students should make inferences from data and graphs, apply properties of informal geometry, and accurately use the tools of technology. Students at this level should understand the process of gathering and organizing data and be able to calculate, evaluate, and communicate results within the domain of statistics and probability.

• Advanced: Eighth-grade students performing at the Advanced level should be able to reach beyond the recognition, identification, and application of mathematical rules to generalize and synthesize concepts and principles in the five NAEP content strands.

  Eighth-graders performing at the Advanced level should be able to probe examples and counterexamples to shape generalizations from which they can develop models.

  Eighth-graders performing at the Advanced level should use number sense and geometric awareness to consider the reasonableness of an answer. They are expected to use abstract thinking to create unique problem-solving techniques and explain the reasoning processes underlying their conclusions.

  Source: Appendix A, NAGB (2004). Mathematics Framework for the 2005 National Assessment of Educational Progress. Washington, DC: Author

Description of Mathematics Content Areas

These content areas apply to each of the three grades assessed by NAEP. The questions designed to test the various content areas at a particular grade level tend to reflect the expectations normally associated with instruction at that grade level.

Number Properties and Operations

This content area focuses on students' ability to represent numbers, order numbers, compute with numbers, make estimates appropriate to given situations, use ratios and proportional reasoning, and apply number properties and operations to solve real-world and mathematical problems. This content area also addresses number sense-comfort in dealing with numbers-and addresses students' understanding of what numbers tell us, equivalent ways to represent numbers, and the use of numbers to represent attributes of real-world objects and quantities. At grade 4 the focus is on whole numbers and fractions; at grade 8 the focus extends to include rational numbers; at grade 12 the focus extends to include real numbers.

Measurement

This content area focuses on students' understanding of measurement attributes such as capacity, weight/mass, time and temperature, as well as on the geometric attributes of length, area, and volume. Students may be asked to select appropriate units and tools for measuring, to measure length with a ruler at all three grades, to measure angles with a protractor at grades 8 and 12, and to solve application problems related to units of measurement. At grade 4, the focus is on time, temperature, capacity, length, weight, perimeter, and area. At grades 8 and 12, students are also expected to understand and demonstrate knowledge of volume and surface area. Knowledge of both customary and metric units is expected. Students may be asked to solve problems that require conversions between (with conversion factors given) or within systems of measurement.

Geometry

By grade 4, students are expected to be familiar with simple plane figures such as lines, circles, triangles, and rectangles, as well as with solid figures such as cubes, spheres, and cylinders. They are also expected to be able to recognize examples of parallel and perpendicular lines. As students move to middle school and beyond, understanding of two- and three-dimensional figures should deepen, with increased understanding of properties of these figures, especially parallelism, perpendicularity, angle relations in polygons, congruence, similarity, and the Pythagorean theorem. Students at all grades are expected to show knowledge of symmetry and transformations of shapes, and to identify images resulting from flips, rotations, or turns. Justifications and reasoning in both formal and informal settings are expected at grades 8 and 12.

Data Analysis and Probability

This content area focuses on students' skills in four areas: (1) data representation; (2) characteristics of data sets; (3) experiments and samples; and (4) probability. Data representation focuses on reading and interpreting data, solving problems based on data and, at the upper grades, evaluating the effectiveness of the presentation of data. At grade 4, students are expected to use standard statistical measures such as the median, range, or mode, and to compare sets of related data; at grades 8 and 12 they are also expected to show understanding of other statistical concepts, such as the impact of outliers and the line of best fit in a scatterplot. By grade 8, students are expected to have some knowledge of experiments and samples, such as being able to recognize possible sources of bias in sampling and to identify random versus non-random sampling, and by grade 12 they are also expected to make inferences from sample results. Students at all grades are expected to use statistics and statistical concepts to analyze and communicate interpretations of data. Students may be asked to solve problems that address appropriate methods of gathering data, the visual exploration of data, ways to represent data, or the development and evaluation of arguments based on the analysis of data. Probability is assessed informally at grade 4 and more formally at grades 8 and 12.

Algebra

This content area focuses on students' understanding of patterns, relations, and functions; algebraic representation; variables, expressions and operations; and equations and inequalities. At grade 4, students are expected to show knowledge of simple patterns and expressions; at grade 8 this knowledge extends to include linear equations; and at grade 12 it extends further to include quadratic and exponential equations and functions. Representational skills, such as students' ability to translate between different forms of representation (e.g., from a written description to an equation), the ability to graph and interpret points located on a coordinate system, and the ability to use algebraic properties to draw a conclusion are assessed in this area. Students' may be asked to express relationships algebraically as number sentences, equations, or inequalities; manipulate algebraic expressions, or to solve and interpret algebraic equations and inequalities that are grade-level appropriate. The use of algebraic tools to solve real-world problems is an important component of the algebra content area.

[1] Sources

Appendix A. Overview of Procedures Used for the NAEP 2005 Mathematics Assessment, State Report Generator.

NAGB (2004) Mathematics Framework for the 2005 National Assessment of Educational Progress. Washington, DC: Author

[2] See detailed content descriptions.

[3] NAGB (2004) Mathematics Framework for the 2005 National Assessment of Educational Progress. Washington, DC: Author

[4] Appendix A. Overview of Procedures Used for the NAEP 2005 Mathematics Assessment, State Report Generator.