I Only Have One Question . . .
Posted June 25, 2015
Editor's Note
Have your students ever been confused about the difference between operant and classical conditioning? Have they ever gone off track in their understanding of dependent and independent variables? Imagine you could use a single multiple choice item in the middle of class to actually improve learning! Here, Rob McEntarffer uses a real life case study to clearly explain how a simple but powerful new technique can be easily applied in the classroom.
------------------------------------
By Rob McEntarffer
In a 2006 article Wylie and Ciafolo describe a technique called “single diagnostic items” that may be a great tool for teachers to use to gauge the impact of classroom demonstrations. Single diagnostic items focus on one important concept and “diagnose” student misconceptions about that concept. Imagine using a single item to determine what your students are learning! Wylie and Ciafolo define these items as “single, multiple choice questions connected to a specific content standard or objective. They have one or more answer choices that are incorrect but related to common student misconceptions regarding that standard or objective” (p. 4). The incorrect responses indicate a specific misconception about the concept, so that student responses identify specific misconceptions.
I wanted to see how single diagnostic items worked in a real classroom so I asked an instructor of an introductory psychology class at a local small liberal arts college for permission to work with one of her classes. She and I decided to focus on the topic of working memory. The text for the course did not cover this topic thoroughly and the instructor had not yet discussed this topic with the class.
My experience with single diagnostic items in the classroom
After introducing myself and explaining the goals of the research project, I asked the class to respond in writing to the prompt: “In a few sentences, please briefly describe working memory.” Then I conducted a working memory demonstration: Students closed their eyes and mentally counted the number of windows in their house. After they finished, they closed their eyes again to “count the number of words in the sentence I just said.” After they finished this task, students indicated whether they had to use their fingers to count when I asked them about the number of windows in their house (none of the students raised their hands). Then I asked how many used their fingers to count the number of words in the sentence (almost all the students raised their hands). Then I projected a single diagnostic item on the screen:
Why do most people use their fingers when they count the words in the sentence, but not when they count the windows?
A. Windows are visual, and visual things are easy to process.
B. Most people are visual learners.
C. The windows are in long term memory, but the words are in short term memory.
D. Familiarity - I'm more familiar with my windows than I am the words in that sentence, so that task is harder.
E. I can picture the windows but I can't picture the words, and that has something to do with it.
F. Working memory must process words and pictures differently.
Students then indicated their response to this item (using their cell phones and the website Poll Everywhere: http://www.polleverywhere.com/). We briefly discussed the diversity of their responses, shown here:
In our discussion the students pointed out that at least one student in the class chose each of the possible responses. We discussed the frequency of the different responses : most students chose answer C (“The windows are in long term memory, but the words are in short term memory”) or answer E (“I can picture the windows but I can't picture the words, and that has something to do with it”). We briefly discussed the diversity of responses and concluded that the data indicate that the class doesn’t yet have a common explanation for why the word counting task required almost everyone to count on their fingers and the windows counting task did not.
Then I explained the origin of the task: Baddeley and Hitch (1974) established that working memory is an active system made up of separate elements that deal with different kinds of information differently. To complete the “counting the windows” task, first working memory determines that the windows need to be pictured and then counted (“central executive” function). Then working memory activates the element that handles words and numbers in order to count the windows (“phonological loop”), and the element that can picture each window visually (visuo-spatial sketchpad”). When faced with the “count the number of words in the sentence I just said” task, the central executive encounters a problem. The phonological loop has to repeat the words in the sentence, but the visuo-spatial sketchpad can’t count, so most people have to use their fingers to complete the task.
After explaining the working memory research and terminology to the class, the students again wrote answers to the writing prompt “In a few sentences, please briefly describe working memory. “ They also again used their cell phones to vote on the correct answer to the diagnostic item:
The class discussed these data and agreed that the memory demonstration and explanation changed their conceptions and understandings about the nature of working memory. Almost everyone in the class agreed in the end that answer F “working memory must process words and pictures differently” was the most correct answer. We discussed the two previous most common answers (C and E) and the class was able to describe in what ways those responses were correct and incorrect.
Later I analyzed the students’ written responses to look for other evidence of changes in understanding of the working memory concept. I created a short rubric to use to score students’ pre and post writing responses:
Each student response was scored by me and a colleague who did not know which responses were “pre” and which were “post.” These scoring data also indicate changes in understanding the working memory concept.
Single diagnostic items like this one could be used to assess the effectiveness of the classroom demonstration about operational definitions. These “effectiveness data” could be used to make decisions about which demonstrations are most effective and which need to be modified. These same data could have multiple formative purposes: Teachers can regroup students into discussion groups based on their responses and ask groups to process the rationale behind their answers. Heterogeneous discussion groups might be useful, each student discussing their different answer with the goal of the group moving toward a consensus conclusion. Teachers could use the two most common answers and use other classroom demonstrations/activities to focus on those misconceptions directly. All these formative uses of the assessment data share a common characteristic: data from this one item are used to focus specifically on student misunderstandings about this important concept. This focus on the misconceptions these students demonstrate address student thinking actively and directly. The assessment data informs instruction by the teacher and metacognition by the students.
How to develop Single Diagnostic Items
Developing single-diagnostic items does require teachers to invest time in the item development process, but can save time in the classroom by efficiently providing valuable information about student misconceptions. One item-develop process is described below
1. Gather teachers who teach the same/similar content. Writing single-diagnostic items requires “deep” content knowledge, and is best done with a group of experienced teachers.
2. Choose a “big idea” to focus on. Single-diagnostic items take a while to write, so the group should spend its time focusing on an idea/concept/etc. that is a “big deal.” Some authors call these “hinge” or “threshold” concepts: ideas that students need to understand well in order to make progress in the discipline.
3. Ask the group to list misconceptions about the “big idea” (another way to phrase this task is to ask “How do students go wrong about this idea?) List all the misconceptions the group develops, then look at the list and collapse any similar ideas into appropriate categories.
4. Write a stem for the single-diagnostic item that will require students to use the “big idea.”
5. Write options for the single-diagnostic idea, one option per misconception and one possible correct answer. Note: multiple correct answers can be included, and the group should end up with one (and only one) option for each misconception. Ideally, if a student chooses an incorrect option, teachers should be confident the student did so because they are laboring under that specific misconception.
6. Test the item with real students. Participating teachers should use the item in class, and ask students who choose an incorrect response WHY they chose that response to test the relationships between incorrect options and misconceptions.
7. Revise based on feedback.
References:
Baddeley, A. D., & Hitch, G. (1974). Working memory. In G.H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (Vol. 8, pp. 47-89). New York, NY: Academic Press
Wylie, C., & Ciofalo, J. (2006). Using diagnostic classroom assessment: One question at a time. Teachers College Record, Jan. 10, 2006, 1-6.
[Rob McEntarffer taught Psychology, AP Psychology, and Philosophy for 13 years at Lincoln Southeast high school in Lincoln, NE, and was involved with the AP Psychology reading for many years. He became interested in educational measurement issues, and got his Masters degree in Educational Measurement (Qualitative and Quantitative Methods) from the University of Nebraska, Lincoln in 2003. Rob started his work as an Assessment/Evaluation specialist with Lincoln Public Schools in 2005, and works with the district on large scale and classroom assessment issues. Rob earned his Phd in Teaching, Learning, and Teacher education in 2013, focusing his research on how teachers make room for formative assessment processes in their classrooms. Rob lives with his wife, two kids, dog, and cat in Lincoln, NE and works for Lincoln Public Schools.]
