Overview of 2005 Reading Assessment

• Who was assessed?
• What is assessed?
• How is mathematics assessed?
• How are constructed-response questions scored?
• How is the NAEP mathematics assessment administered?
• How long does the NAEP assessment take?
• How are results reported?
• What are the achievement level descriptions?
• Sources

Who was assessed?

• The NAEP 2005 assessment was administered to a stratified random sample of fourth-, eighth-, and twelfth-graders at the national level and to fourth- and eighth-graders at the state level.
• Both public and nonpublic school students were assessed at the national level.
• At the state or jurisdiction level, only the results for public school students are reported.
• Fifty-two jurisdictions participated, including the 50 states, the District of Columbia, and the Department of Defense Schools (Domestic and Overseas).
• National (public and nonpublic) and state (public only) samples include the following:
  • Grade 4
    • Approximately 165,700 students
    • Approximately 9,500 schools
  • Grade 8
    • Approximately 159,400 students
    • Approximately 7,200 schools
• See NAEP Sampling: Questions and Answers for additional information about sampling and Participation in NAEP 2005 by Public Schools and Students for state-specific information about the number of participating schools and students.

  Accommodated/Non-accommodated Samples

  • Prior to 1998, NAEP procedures for assessing reading did not permit the use of accommodations for special needs students who could not participate without them.
  • In order to make the NAEP assessment more inclusive, beginning in 1998, new procedures were implemented to allow certain accommodations for special needs students. A split-sample design was used so that both administrative procedures-with accommodations and without accommodations-could be used during the same assessment, but with different samples of students.
  • The split-sample design allows the reporting of trends across all the assessment years, as well as permitting an examination of the impact of permitting accommodations on overall results.
  • Beginning in 2002, NAEP used only one set of procedures-permitting the use of accommodations. This policy was continued for the 2003 and 2005 reading assessments.
     

    NAEP Reading Assessment
    Year	Grades	State	Sample	Accommodations Allowed
    1992	4	Yes	Entire sample	No
    1994	4, 8	Yes	Entire sample	No
    1998	4, 8	Yes	Split Sample	No for random half Yes for random half
    2000	4	No	Split sample	No for random half Yes for random half
    2002	4, 8	Yes	Entire sample	Yes
    2003	4, 8	Yes	Entire sample	Yes
    2003	4, 8	Yes	Entire sample	Yes

  Interpreting comparisons between accommodated and non-accommodated samples

  • Caution should be used in interpreting comparisons between accommodated and non-accommodated samples, e.g., between 1992 or 1994 results, and 2005 results.
  • When accommodations were not permitted, students with disabilities and English language learners were not included in the assessment unless local school staff determined that they could be assessed meaningfully without accommodations. Therefore, in later assessments where accommodations were allowed, some students took the assessment that would have been excluded from previous assessments.

What is assessed?

• The NAEP Frameworks specify what is assessed and how it is to be assessed.
• The Reading Framework for NAEP 2005 continues the NAEP reading trend begun in 1992.
• The framework sets forth a broad definition of "reading literacy" that includes developing a general understanding of written text, thinking about it, and using various texts for different purposes.
• There are three contexts for reading:
  • reading for literary experience,
  • reading for information, and
  • reading to perform a task (grades 8 and 12 only).
• Contexts provide guidance for passage selection.
• There are four aspects of reading-each represent different ways in which readers develop understanding of a text:
  • forming a general understanding,
  • developing interpretation,
  • making reader/text connections, and
  • examining content and structure.
• The framework specifies the target percentage of questions devoted to each reading context, and the target percentage of student time devoted to each aspect of reading [2].
   

  Contexts for Reading -Target Distribution of Questions
  Grade	Literary Experience	To Be Informed	To Perform a Task
  4	55%	45%	NA
  8	40%	40%	20%
  Aspects of Reading -Target Distribution of Student Time
  Grade	Forming a General Understanding/Developing Interpretation	Making Reader/Text Connections	Examining Content and Structure
  4	60%	15%	25%
  8	55%	15%	30%

How is reading assessed?

• Students read passages and are asked to answer comprehension questions.
• Passages are authentic and are not developed specifically for the NAEP assessment.
• Passages are drawn from those that students might encounter at school, in the library, or at home.
• Passages are not abridged, but are presented full-length and resemble how students would encounter the passages in the format of the original publication.
• Multiple-choice questions-four choices are presented.
• Constructed-response questions-students write their own response:
  • Short-a few sentences.
  • Extended-a paragraph or a full page.
• In 2005, the distribution of types of questions across all blocks was as follows
   

  Type of Question	Grade 4	Grade 8
  Multiple-choice	52	62
  Short, constructed response	39	65
  Extended, constructed response	8	15
  Total	99	142

How are constructed-response questions scored?

• Unique scoring guides are developed for each question.
• Scoring guides describe the specific criteria for assigning a score level for student responses.
• Scoring process:
  • Expert scorers are extensively trained to apply the scoring criteria consistently and fairly.
  • Scoring is monitored to ensure the scoring standards are being adhered to reliably.
  • Monitoring measures the consistency of scoring to the same items administered in different assessments-therefore, ensuring consistency of the application of scoring standards across assessment years.
  • Over 3,773,700 reading constructed responses were scored for the 2005 assessment.

How is the NAEP reading assessment administered?

• Each fourth-grade student took two 25-minute sets of questions. Each eighth-grade student took either two 25-minute sets or one 50-minute set. All students took one set of general background questions, and one set of background questions related to reading.
• Each block contained one passage corresponding to one of the contexts for reading and 9-12 multiple-choice and constructed-response questions.
• In order to provide a comprehensive assessment of reading and to minimize the burden on any individual student, NAEP uses matrix sampling. Each student takes a subset of the total set of questions.
• Because each block is spiraled with other blocks and is administered to a representative sample of students, the results can be combined to produce average group and subgroup results based on the entire assessment.

How long does the NAEP assessment take?

• No more than about 1 hour per student to actually take the assessment-about 50 minutes on reading questions, and a few more minutes on background questions.

How are results reported?

• Scale scores-indicate how much students know and can do
• Average scale scores and percentiles
• Range is 0-500
• Achievement levels-what students should know and be able to do.
• Subscales
• Results are analyzed and summarized by subscales that correspond to the three reading contexts: reading for literary experience, reading for information, and reading to perform a task (grades 8 and 12 only).
• Separate subscale results are combined to create a single overall score for reading comprehension.

What are the achievement level descriptions?

Policy definitions of NAEP Achievement Levels:

• Basic: This level denotes partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade.
• Proficient: This level represents solid academic performance for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter.
• Advanced: This level signifies superior performance.

NAEP Achievement Level descriptions for mathematics-set separately by grade

Grade 4

• Basic: Fourth-grade students performing at the Basic level should demonstrate an understanding of the overall meaning of what they read. When reading text appropriate for fourth-graders, they should be able to make relatively obvious connections between the text and their own experiences, and extend the ideas in the text by making simple inferences.

  For example, when reading literary text, they should be able to tell what the story is generally about-providing details to support their understanding-and be able to connect aspects of the stories to their own experiences. When reading informational text, Basic-level fourth-graders should be able to tell what the selection is generally about or identify the purpose for reading it, provide details to support their understanding, and connect ideas from the text to their background knowledge and experiences.

• Proficient: Fourth-grade students performing at the Proficient level should be able to demonstrate an overall understanding of the text, providing inferential as well as literal information. When reading text appropriate to fourth grade, they should be able to extend the ideas in the text by making inferences, drawing conclusions, and making connections to their own experiences. The connections between the text and what the student infers should be clear.

  For example, when reading literary text, Proficient-level fourth-graders should be able to summarize the story, draw conclusions about the characters or plot, and recognize relationships such as cause and effect. When reading informational text, Proficient-level students should be able to summarize the information and identify the author's intent or purpose. They should be able to draw reasonable conclusions from the text, recognize relationships such as cause and effect or similarities and differences, and identify the meaning of the selection's key concepts.

• Advanced: Fourth-grade students performing at the Advanced level should be able to generalize about topics in the reading selection and demonstrate an awareness of how authors compose and use literary devices. When reading text appropriate to fourth grade, they should be able to judge texts critically and, in general, give thorough answers that indicate careful thought.

  For example, when reading literary text, Advanced-level students should be able to make generalizations about the point of the story and extend its meaning by integrating personal experiences and other readings with ideas suggested by the text. They should be able to identify literary devices such as figurative language.

  When reading informational text, Advanced-level fourth-graders should be able to explain the author's intent by using supporting material from the text. They should be able to make critical judgments of the form and content of the text and explain their judgments clearly.

Grade 8

• Basic: Eighth-grade students performing at the Basic level should demonstrate a literal understanding of what they read and be able to make some interpretations. When reading text appropriate to eighth grade, they should be able to identify specific aspects of the text that reflect overall meaning, extend the ideas in the text by making simple inferences, recognize and relate interpretations and connections among ideas in the text to personal experience, and draw conclusions based on the text.

  For example, when reading literary text, Basic-level eighth-graders should be able to identify themes and make inferences and logical predictions about aspects such as plot and characters. When reading informational text, they should be able to identify the main idea and the author's purpose. They should make inferences and draw conclusions supported by information in the text. They should recognize the relationships among the facts, ideas, events, and concepts of the text (e.g., cause and effect, order). When reading practical text, they should be able to identify the main purpose and make predictions about the relatively obvious outcomes of procedures in the text.

• Proficient: Eighth-grade students performing at the Proficient level should be able to show an overall understanding of the text, including inferential as well as literal information. When reading text appropriate to eighth grade, they should be able to extend the ideas in the text by making clear inferences from it, by drawing conclusions, and by making connections to their own experiences-including other reading experiences. Proficient eighth- graders should be able to identify some of the devices authors use in composing text.

  For example, when reading literary text, students at the Proficient level should be able to give details and examples to support themes that they identify. They should be able to use implied as well as explicit information in articulating themes; to interpret the actions, behaviors, and motives of characters; and to identify the use of literary devices such as personification and foreshadowing. When reading informational text, they should be able to summarize the text using explicit and implied information and support conclusions with inferences based on the text. When reading practical text, Proficient-level students should be able to describe its purpose and support their views with examples and details. They should be able to judge the importance of certain steps and procedures.

• Advanced: Eighth-grade students performing at the Advanced level should be able to describe the more abstract themes and ideas of the overall text. When reading text appropriate to eighth grade, they should be able to analyze both meaning and form and support their analyses explicitly with examples from the text, and they should be able to extend text information by relating it to their experiences and to world events. At this level, student responses should be thorough, thoughtful, and extensive.

  For example, when reading literary text, Advanced-level eighth-graders should be able to make complex, abstract summaries and theme statements. They should be able to describe the interactions of various literary elements (i.e., setting, plot, characters, and theme) and explain how the use of literary devices affects both the meaning of the text and their response to the author's style. They should be able to critically analyze and evaluate the composition of the text. When reading informational text, they should be able to analyze the author's purpose and point of view. They should be able to use cultural and historical background information to develop perspectives on the text and be able to apply text information to broad issues and world situations. When reading practical text, Advanced-level students should be able to synthesize information that will guide their performance, apply text information to new situations, and critique the usefulness of the form and content.

  Source: National Assessment Governing Board. (2004) Reading Framework for the 2005 National Assessment of Educational Progress. Washington, DC

[1] Sources

Appendix A. Overview of Procedures Used for the NAEP 2005 Reading Assessment, State Report Generator. National Assessment Governing Board. (2004) Reading Framework for the 2005 National Assessment of Educational Progress. Washington, DC

[2] See Appendix A, Overview of Procedures Used for the NAEP 2005 Reading Assessment for the actual distribution of questions and time for the 2005 assessment.