Course Hub

EDST 0213A
Educational Testing

  • Syllabus
    EDST0213_13Sept2023.docx (38.21 KB)

     

    Understanding Educational Testing

    EDST 0213A

    Syllabus: 13 September 2023 (subject to minor revision)

     

    I.  Course/Instructor Information:

    Course Title: Understanding Educational Testing

    Semester/Year: Fall 2023

    Class meeting times: Tuesdays & Thursdays, 2:15 – 3:30

    Class location: Twilight 305

    Instructor: Steve Hoffman

    Email: hoffman@middlebury.edu

    Office Hours via Zoom:

    • Tuesdays: 3:45 – 5:15 PM
    • Thursdays: 3:45 – 5:15 PM
    • By Appointment

    II.   Course Description:

    Achievement testing is now a cornerstone of education policy. It is also complex and routinely misunderstood by educators, policymakers, and the media. In this course students will use statistical methods to explore and address testing issues that arise in both policy and practice. We will examine the uses and abuses of educational assessment. We will examine and interrogate trends and group differences in achievement. And we will broaden our understanding of essential concepts of measurement, such as reliability, validity, and bias, by analyzing both large and small datasets. Prior experience with the statistical package “R” is not required, as learning this package will be part of the course. 3 hrs. lect.

    III.  Learning Outcomes:

    In this course, we will:

    • critically examine the uses and abuses of educational assessment
    • examine and interrogate trends and group differences in achievement
    • broaden our understanding of essential concepts of measurement, such as reliability, validity, and bias
    • gain facility employing statistical methods to explore and address testing issues that arise in both policy and practice
    • use the statistical package “R” to analyze testing data sets
    • communicate our understandings and findings clearly to the general public

    IV.   Course Materials and Required Texts:

    Koretz, D. (2008). Measuring up: What educational testing really tells us. Harvard University Press. (ebook OK)

    Early on (on page 2), Professor Koretz noted that at the time he wrote this book there were many books on educational testing. Some were very pro-testing while others were anti-testing. Fifteen years later this is still true. Koretz intended to provide a balanced approach to explaining the field of educational testing to a wide audience, and this book is still required for courses on testing at other colleges and universities — both those for the “informed consumers of test scores” as well as for more technical courses. So, while this book is not new, and we will come across explanations that seem dated, it remains an excellent resource for us as we work through today’s issues around educational testing.

     

    Popham, W. (2016). The ABCs of Educational Testing, Corwin Press. (ebook OK)

    This book is an easier read than the Koretz text. Professor Popham insists at the start of this book that it’s about “the basics of educational testing” written so that readers can learn “a handful of foundational concepts and procedures linked to the testing that routinely goes on in our schools” (p. 1). We won’t follow the contents in the exact order he outlined, but we will use the whole text during the semester.

     

    Wickham, H., Çetinkaya-Rundel, M. & Grolemund, G. (2022). R for Data Science. https://r4ds.hadley.nz (free to use)

    This ebook is about how to “do” data analysis and data science using the “R” package, which you may have used in other courses at Middlebury. We will utilize only a portion of this book, though you may find it useful to bookmark this for work outside of this course. For those of you with no experience with R at all, don’t fret. I and many others here at Middlebury will help you learn the basic tools in R, allowing you to thrive in this course. Be sure to use the second edition (2e). This is a substantial rewrite from the first edition, and you’ll be learning some different coding techniques from the students in the Fall 2022 iteration of EDST 0213.

    V.   Course Structure:

    Lecture/discussion sessions are scheduled every Tuesday and Thursday from 2:15 – 3:30. Your active participation and engagement are essential to both your success and your colleagues’ success in this course. Come prepared to participate each Tuesday and Thursday. The general format is a mixture of pre-recorded lectures, in-class activities, lectures, in-class discussion, asynchronous Canvas discussion, small-group work, partner projects, and opportunities for students to lead learning.

     

    VI.   Grading Information:

    Course grades will be based on a typical 100-point system. A = 93 – 100; A- = 90 – 92; B+ = 87 – 89; etc. Late work will likely be penalized.

    • Attendance and participation – 20%
    • Quizzes – 20%
    • Discussion posts – 20%
    • Data-Analytic Memos – 20%
    • Final project – 20%

    Attendance and participation

    Attendance is required. Come to class on Tuesday and Thursday afternoons prepared and ready to participate. Twilight 305 is equipped for Zoom, and most sessions will be taped, as well. Students who must miss a class should notify Hoffman before the class meeting and arrange for alternate ways to contribute to the class community. (Pro tip: Don’t write “let me know if I missed anything.”)

    Participation grades are calculated based on attendance, punctuality, contributions during small-group activities, and thoughtful engagement in discussions.

    Note: During the two weeks before Thanksgiving Break, students will (in small groups) take turns leading class, providing appropriate classroom activities, and authoring assessments.

    Quizzes

    Short, low-stakes quizzes to check understanding of the course content will occur often throughout the term. Dates for scheduled quizzes are listed on the Course Calendar, but there will be additional low-stakes quizzes throughout the term. Sometimes quizzes will be administered during face-to-face class sessions. Sometimes quizzes will be administered asynchronously through Canvas. These are not meant to be high-pressure events.

    Discussion posts

    Students will write or record your reflections in response to Canvas discussion prompts about designated assigned readings. Sometimes these will be written posts of 400 words or fewer. Other times you will be asked to compose a short video (typically about one minute long). Organize your thinking about the ideas and arguments made by the authors. If you reference articles, papers, books, or other media beyond the course texts, cite them appropriately. Make an informed critique of ideas, rather than merely summarizing the readings. Finally, read or watch your colleagues’ posts, and write a brief, thoughtful response to at least three (3) of your classmates’ posts. Posts are tentatively due on September 20, October 4, October 18, November 1, November 15, and November 29.

    Data-Analytic Memos

    Designed to develop and extend your data-analytic skills and to help you learn to communicate your findings clearly to others, these are collaborative assignments to be completed with a partner. Please engage in a full, fair, and mutually agreeable collaboration with your partner. Do not simply divide up the work. Discuss and plan the analyses together, debate what you have found with each other, and collaborate on the writing of your memo. My objective is to provide a high-quality opportunity for you to learn, review, teach, and communicate the course material to others. DAMs are tentatively due September 27, October 11, October 25, November 8, and December 3.

    Final project

    Due December 18.

    Assigned after Thanksgiving Break, with an opportunity to draft an outline of your project in early December, this project will serve as our final exam for the course. (There will be no scheduled in-person test during finals week.)

    VII.  Relevant Policies:

    Academic Integrity: As an academic community devoted to the life of the mind, Middlebury requires that every student complete intellectual honesty in the preparation and submission of all academic work. Details of our Academic Honesty, Honor Code, and Related Disciplinary Policies are available in Middlebury’s handbook.

    Honor Code Pledge: The Honor Code pledge reads as follows: "I have neither given nor received unauthorized aid on this assignment." It is the responsibility of the student to write out in full, adhere to, and sign the Honor Code pledge on all examinations, research papers, and laboratory reports. Faculty members reserve the right to require the signed Honor Code pledge on other kinds of academic work.

    Disability access/accommodation: Students who have Letters of Accommodation in this class are encouraged to contact their professor as early in the semester as possible to ensure that such accommodations are implemented in a timely fashion.

    For those without Letters of Accommodation, assistance is available to eligible students through the Disability Resource Center (formerly called Student Accessibility Services). All discussions will remain confidential.

    Please contact one of the ADA Coordinators at ada@middlebury.edu for more information.

    VIII.   Expectations of Students:

    Attendance: Attendance is required. Come to class on Tuesday and Thursday afternoons prepared and ready to participate. Students who must miss a class should notify Professor Hoffman before the class meeting and arrange for alternate ways to contribute to the class community. (Pro tip: Don’t write “let me know if I missed anything.”)

    Participation: Your active participation during class sessions is expected. Participation grades are calculated based on attendance, punctuality, contributions during small-group activities, and thoughtful engagement in discussions.

    Collaboration: Collaborative assignments to be completed with one or more partners are required. Five Data-Analytic Memo assignments produced in partnership and a day leading class in November with a different group of colleagues require substantial work with your colleagues outside of the formal class sessions. Please engage in a full, fair, and mutually agreeable collaboration with your partners. Do not simply divide up the work.

    Late Work: If you need an extension, please ask before the assignment is due and collaborate with me to come up with a reasonable plan for turning in late work. I generally assume that if you ask, you need an extension for a valid reason.

    IX.   Relevant Campus Resources:

    Center for Teaching, Learning, and Research: The CTLR provides academic support for students in many specific content areas and in writing across the curriculum through both professional tutors and peer tutors. The Center is also the place where students can find assistance in time- management and study skills. These services are free to all students. For more information on how to get the help you need, go to the CTLR’s student resource page.

    Disability Resource Center: The DRC provides support for students with disabilities and facilitates the accommodations process by helping students understand the resources and options available and by helping faculty understand how to increase access and full participation in courses. The DRC can also provide referrals for students who would like to undergo diagnostic testing. Students who are on financial aid and have never undergone diagnostic testing can apply to the CTLR for support to cover the cost of off-campus testing. DRC services are free to all students.

    X.   Course Calendar:

    Dates for major assignments/projects (Quiz dates subject to change):

    • September 14: Syllabus Quiz
    • September 20: Discussion Post
    • September 21: Quiz on Measurement
    • September 27: DAM 1
    • October 3: Quiz on Performance Standards
    • October 4: Discussion Post
    • October 5: Quiz on Scales
    • October 11: DAM 2
    • October 18: Discussion Post
    • October 19: Quiz on Error
    • October 25: DAM 3
    • October 26: Quiz on Reliability and Validity
    • November 1: Discussion Post
    • November 2: Quiz on Adverse Impact & Bias
    • November 7: Quiz (Student-led Assessment)
    • November 8: DAM 4
    • November 9: Quiz (Student-led Assessment)
    • November 14: Quiz (Student-led Assessment)
    • November 15: Discussion Post
    • November 16: Quiz (Student-led Assessment)
    • November 29: Discussion Post
    • December 3: DAM 5
    • December 18: Final Project Due

    Daily Schedule of readings/assignments (subject to minor changes):

    September

    September 12: Introduction to the course

    • Measuring Up, Prologue & Chapter 1
    • The ABCs of Educational Testing, Preface & Chapter 1
    • R for Data Science 2e, Welcome, Preface to 2nd edition, & Chapter 1

    September 14: Fundamental Issues in Measurement

    • The ABCs of Educational Testing, Chapter 2
    • Measuring Up, Chapter 2
    • R for Data Science 2e, Chapter 2

    September 19: Measurement concepts

    • Measuring Up, Chapter 3
    • R for Data Science 2e, Chapters 3, 5, & 7
    • Thorndike, Chapter 2, pp. 23 – 28, (Canvas file). This supplemental measurement textbook (Measurement and Evaluation in Psychology and Education, by Thorndike & Thorndike-Christ) includes a play data set for us to work with.
    • Optional reading: Lindquist, E.F. (1951) (Canvas file)

    September 21: Standardized Testing in the USA

    • The ABCs of Educational Testing, Chapter 3
    • Standards for Educational and Psychological Testing 2014, pp. 1 – 4
    • Measuring Up, Chapter 4
    • Thorndike, Chapter 2, pp. 28 – 38, (Canvas file). Just skim the instructions on how to perform various statistical procedures in SPSS and Excel; we will work in R.
    • R for Data Science 2e, Chapter 29

    September 26: Trends in standardized achievement tests

    September 28: Achievement patterns and COVID

     

    OCTOBER

    October 3: Performance standards

    • Measuring Up, Chapter 8 (pages 179 – 200)
    • Pages 1 – 2 & Pages 57 - 68 New Meridian Technical Report 2020-2021 (formerly PARCC)
    • R for Data Science 2e, Chapter 4
    • Optional reading: Hansche L., et al. (1999). Handbook for the Development of Performance Standards: Meeting the Requirements of Title I. Washington, D.C.: Council of Chief State School Officers. Chapters 1 (pages 3 – 6), 3 (pages 11 – 16 only), and 10 (pages 87 – 103). (Canvas file)

    October 5: Scales

    • The ABCs of Educational Testing, Chapter 7
    • Measuring Up, Chapter 8 (pages 200 – 214)
    • Thorndike, pp. 72 – 95 (Canvas). Skim the instructions on how to perform various statistical procedures in Excel and SPSS.
    • R for Data Science 2e, Chapters 8, 9, & 21

    October 10: NAEP – The Nation’s Report Card

    October 12: Frameworks for NAEP

    • Mathematics Framework for the 2026 National Assessment of Educational Progress, Ch. 2, Mathematics Content, pp. 14 – 48. This chapter includes detailed tables of standards, but you don’t need to keep track of all of this. The goal is to get a sense of the breadth of the standards as a whole and the nature of the ‘chunks’ into which mathematics is broken into for the purpose of building the NAEP test. (Note: If you are curious, look at the archived copies of the math frameworks for 2017 or 2019. Most of the content of these frameworks remain in all versions.)
    • Reading Framework for the 2022 and 2024 NAEP. 2, Content and Design of NAEP in Reading, pp. 17 – 46. (The details of the standards are less important than the big picture, but it’s important for you to understand the content breakdown and the assessment issues involved.)

     

    FALL BREAK (October 13)

     

    October 17: International Comparisons

    October 19: Error

    • Measuring Up, Chapter 7

    October 24: Reliability

    October 26: Validity

    October 31: Score Inflation

    • Measuring Up, Chapter 10
    • Holcomb, R., Jennings, J. L., & Koretz (2013). The roots of score inflation: An examination of opportunities in two states’ tests. In G. Sunderman (Ed.), Charting Reform, Achieving Equity in a Diverse Nation. Information Age Publishing. (Canvas file)

     

    November

    November 2: Adverse impact and bias

    November 7: College Admissions (Student led, readings may be revised)

    November 9: More on College Admissions (Student led, readings may be revised)

    November 14: Cheating (Student led, readings may be revised)

    • Aviv, R. (2014). Wrong answer: In an era of high-stakes testing, a struggling school made a shocking choice. The New Yorker. July 21, 2014. (Canvas)
    • Merrow, J. (2013). Michelle Rhee’s reign of error. The Merrow Report.

    November 16: More on cheating (Student led, readings may be revised)

     

    THANKSGIVING BREAK

     

    November 28: Testing Special Populations

    • Measuring Up, Chapters 12 & 13

    November 30: Performance Assessment

     

    December

    December 5: Formative Assessment and assorted school-based topics

    • The ABCs of Educational Testing, Chapters 8, 9, & 10

    December 7: Course Review and Final Project workshop

     

    Additional Resources: