This is long so grab a beverage and have a read…
This past Thursday the Mississippi Department of Education (MDE) released a “testlet” for each of its tested areas (grades 3-8 and algebra one). [There are similar documents released for English/Reading, but I am not sure about their release date(s).] The testlets are meant to be practice tests for classroom teachers and/or administrators to use with the students before they take the end of year (EOY) course exams. As an algebra one teacher and assessment aficionado, I of course had my paws on the printed document before our administrators walked it down to our classrooms.
The first thing my administrator said, document in hand, was “We need to decide as a group how we are going to use this.” Because I am a crusty, seasoned, classroom teacher with 26 years of experience I replied, “Before we meet as a group, I need to check the quality of this document. I am not about to waste two days of instructional time giving my kids a junk pretest.” To be fair, my administrators are AWESOME and totally respected my need to comb through the document.
Fast forward a few hours and I was in a state of extreme testlet nausea. Many of the questions were so bad I just didn’t know what to do. I emailed several of the questions with my annotated concerns to both MDE and to my math colleagues nationwide. My math colleagues were saddened, appalled, and in one case, not surprised. MDE did acknowledge that on the first question THE GRAPH DIDN’T EVEN MATCH THE EQUATION and so none of the statements were valid, and in fact amended that question immediately.
It still wasn’t that great and so I fixed it.
Question 1 as it first appeared:
Question 1 after MDE fixed it, and as it still appears in the teacher resources:
(NOTE: The answer key says that answers A and E are valid. Don’t get me started. I already am.)
Question 1 after I fixed it:
It is honestly still flawed — will probably have to throw it out completely.
(As a fun exercise you could put the answers to my revised questions in the comment thread.)
Before I show you some more questions as they currently appear and how I fixed them, let me back up a few years. Last year Mississippi students and teachers were subjected to PARCC testing. It was horrible. Too much testing (Universal screener, benchmark, PARCC PBA, PARCC EOY) and the PBA and EOY were all on-line so our students were testing in small batches for weeks. [Our district does not have a one-to-one student to computer ratio.] Instructional time was interrupted, compromised, and some days nonexistent. We jumped for joy when the legislature (or was it governor?) threw out PARCC. But the celebration was naive and short-lived. Mississippi signed a 10-year, multi-million dollar contract with a testing company I had never heard of: Questar. To the best of my knowledge, Questar had never written a math assessment before Mississippi hired them.
At some point this past fall, Questar released its “item sampler” for algebra one. It was a small batch of on-line interactive questions with the print version available too. I was thrilled to see these questions, but found several mistakes in the answer key. I emailed MDE and they referred me to Questar. I emailed Questar and nothing changed on the answer key. Five months later and still, the answer key has not been changed. Makes me really confident that my students’ work will be scored correctly this spring! Ugh. In addition, I emailed Questar asking when we could expect a full-length sample test. The email I got in response was so rude and amounted to, “Your job is to teach the curriculum, not teach to the test.” I am sure the person emailing me was a twenty-five year old who has spent maybe 18 months in a classroom. He probably doesn’t even work there anymore.
I am sure many superintendents, curriculum specialists, and district test coordinators (DTCs) had the same question I did, “When are we going to see more practice items?” I guess releasing practice items was not part of Questar’s multi-million dollar contract. So, the under-funded, under-staffed MDE took it upon themselves to release testlets to its teachers. Unfortunately, the algebra one result was not good.
I spent the better part of Friday fixing the testlet. One of my administrators asked if he should get me a sub. I declined, saying that my students would be so busy on a cooperative activity that they would barely need me. I asked MDE if they wanted my fixed version but the reply I got was, “That won’t be necessary.” Instead, lots of teachers in the northern districts are using my fixed testlet anyway via their DTCs, who got the fixed version from our DTC with my permission.
What follows are several questions, as they appear in the MDE teacher resources, and then as they appear in my fixed version.
MDE Question 8:
(NOTE: Their answer key says F, T, T, F, T.)
How I fixed Question 8:
MDE Question 10:
(Note: The answer key says A and B are valid.)
How I fixed Question 10:
MDE Question 18:
(Note: Answer key says A, B, E)
How I fixed Question 18:
These items are but a small sample of the items I fixed. Admittedly, my fixes are imperfect, but the items are a whole lot better than what I first saw.
So now I need to turn my attention to the 8th grade testlet. Here we go…
MDE answer key says B and E. Nope. Not correct. B is the only statement that correctly describes this function.
Math, Teaching, and Teaching Math 
Being from the sort-of-mediocre state of Illinois, the land of nine months without a budget and therefore no authority to spend any money except where the courts have forced the state to pay, I’m surprised that PARCC hasn’t dropped Illinois out of fear that we won’t pay. That said, I wouldn’t want to be in your shoes either. I would like to hope that your concerns will be heard in time, but if the samples are this bad, it may not matter.
How much is Questar going to pay you for fixing their practice tests? What a sham. MDE should have a committee of teachers reviewing this stuff if they can’t do it themselves. Zero confidence that the operational assessments will be any better.
Remember these items came from our Department of Education not from our vendor. Our vendor is not releasing anything but a “item sampler” — kind of like a “fun size” candy bar when you need and entire dessert buffet to feed 100 people.
AMEN!
I was fixing MDE’s items, not Questar’s items.
The issues are with MDE ‘s practice testlet, not so much with Questar’s item sampler. MDE had to produce the testlet because Questar only gave us a few sample items.
Do we even know if MDE’s practice testlet will be like the Questar test questions at all?
No we don’t.
I think you did an admirable job trying to fix items. BUT…when you start with complete nonsense like some of these, the finished product after all your revisions won’t ever justify the amount of time you have to put in to fix the questions. I’ve been down this road many times with another vendor.
If your principal wants a practice test, it’s a better use of your time as a teacher to just write completely new items. I also think that would be more helpful for your students, although there’s something to be said for writing them in the same format as they’ll see on the standardized test when it comes out.
You are so right. I write tests and quizzes all the time so my students are constantly practicing — but it is good to have our students work items from another author, in the format that will be expected. Ugh.
Remember, these items did not come from our vendor — our vendor is not providing enough practice items so our Department of Education put these items together for us.
Here’s what I found online. Practice generator here: https://ms-practice.nextera.questarai.com/student/Webclient/PracticeTest/
And an Algebra 1 practice test also available via PDF: https://ms-practice.nextera.questarai.com/student/Webclient/wwwroot/Mississippi/documents/printedTest/MS1604_AlgI_Sampler.pdf
Yes. Those are the only items with which Questar has provided us. The answer to number 19 is wrong. The answers to several of the others are kind of confusing (B instead of b, e.g.).
Pingback: Mississippi Algebra I Teachers Unite! | Virge Cornelius — Jedi Master
Pingback: Testlet Update | Virge Cornelius — Jedi Master
I’m in Minnesota, and one year, an insistent father was finally allowed to look at his daughter’s test, which she had “failed”, and found several questions which were graded incorrectly. Lawsuits ensued, test vendor fired, but testing still goes on! When MN began BASIC TESTING (8th grade level), they did invite some real classroom teachers in to give comments on test questions. I’m not sure if they still do, but maybe this explains why Minnesota’s “standardized” test scores are higher than those of some other states. You should suggest the state legislators take the exams to see how well they are spending YOUR tax $$. It would be interesting if there ever WAS a truly standardized test in this country, other than the exceptionally well written ACT, SAT, and AP exams.
If we demanded the legislators take these tests, trust me, the embarrassment would be even greater and the testing, like you noted, would continue. Here is what I suspect happened (though this sequence of events has not been verified by MDE):
Supposedly teachers did write questions in the summer of 2015 for / with the vendor. In fact, my principal wanted me to go down to Jackson but I was going to be in New Hampshire at that time teaching summer school. MDE in its contract with the vendor (Questar) must have only asked for a small number of practice problems. In other words, the vendor was only contracted to produce an item sampler for each of the seven assessments as part of its 10-year, 110+ million dollar contract. So I think what must have happened is that MDE was under a lot of pressure from districts state-wide to produce more practice items for its teachers and students. A standard is written, and then how it is interpreted in a test question can vary widely. Teachers need to see items! Students need practice with items! (Look at AP — look how many items they have released!) So, MDE cobbled together these “testlets” from items written in summer 2015 that were not taken back to Minneapolis to Questar’s headquarters. Many of these items on the testlets were flawed. Since I only teach algebra I (and AP Calculus), I only looked at the algebra one testlet and tried to fix it. Districts with central office math specialists were pouring over all of the seven testlets, trying to fix them and give feedback to MDE. I might note that we as teachers and curriculum specialists were doing this on our own free time and for no compensation.
Here’s an even bigger issue that I haven’t even written about in detail yet: There is a mistake on Questar’s algebra I item sampler answer key. I have emailed both MDE and Questar about it (once in November when the sampler items were first released and again in March) and no one emails me back. Not Questar. Not MDE. So what will happen when the tests are live at the end of April and our children sit taking these tests? There is no doubt in my mind that there will be poor questions which make the answer choices invalid, and/or computer scoring where the answer key is wrong. What about if a student types in y = 3x – 5 instead of h = 3t – 5? Will that be wrong? We spend a lot of time in deliberation for AP Calculus meetings about whether or not to award these points each year, based on the stem of the problem. But a computer will be scoring our children’s math work. When the statisticians look at the results will those poor questions or wrong answer key items “thrown out” and not count towards the children’s scores? I have no idea.
Another major issue is that we are testing our students on computers. We do not interact with computers in the math classroom on a daily basis in our district. So will the students’ scores be clouded by the testing platform? In other words, are we measuring their math content knowledge or are we measuring their computer fluency?
I ask a lot of questions. I will continue to ask them until someone answers them.
Pingback: The End of March Madness | Virge Cornelius — Jedi Master
I too have problems with the entire thing…and also found many mistakes. How do you fix it when it is this bad!?! I have reworded many and have given my students questions similar to each question provided but this is one of my favorites….I gave one of the 8th grade “Maplet” questions to my 8th graders (I usually teach upper level math classes but we are short teachers) and EVERYONE of mine missed the question about Henry and the fish… or did they…..
Henry is fishing from a small boat. His fishing hook is 8 meters below the water, and a fish is swimming at the same depth as the hook, 15 meters away. How far away is Henry from the fish.
Clearly this is a Pythagorean Theorem Problem (not even a hard one…a triple!), however I have told my students to pay attention to detail. Almost everyone of my kids said that the fish is 17 meters from where the hook enters the water but there is not enough information to tell how far Henry is from that point….. The answer key says that the answer is 17 meters. Does this mean that my kids will miss it? The way is looks, there will not be a place for the students to explain on some of the questions…just numeric input.
Many of the questions are tricky and the answers just seem to make it worse. Are we testing what the kids know or are we seeing how tricky we can be. I am so glad I found this page because I also have emailed MDE with no reply and emailed Questar with no reply….VERY FRUSTRATED!!!!! You can teach the curriculum all day every day and not emphasize the part you need…
Let’s look at the “Reference Sheet”…at what point did we introduce base e to Algebra 1. In the curriculum example they gave the basic exponential equation P(1+r)^n but the reference sheet put the base e? Funny that the Algebra 1 reference sheet is almost identical to the 8th grade reference sheet except for the last few formulas. Oversite? Who knows.
VERY FRUSTRATED!
Are you in our Facebook Group?
I looked for it but nothing came up. Can you invite me? Michelle Wells Green
Yes. It should be appearing now.