Friday, May 29, 2009

Tests .... (TIMSS & the MA teacher licensing test)

Part I. TIMSS.

Lately I've been reading a lot about the Trends in International Mathematics and Science Study, which has helped us compare the performance of U.S. students in math to that of students in other countries. It was given to students in 4th and 8th grades in 1995, 1999, 2003, and 2007, and the U.S. performs well below a number of countries, including Japan. Until just now I hadn't really thought to question the validity of the study.

But when I went to the NCES
(National Center for Education Statistics) website, I saw Dare to Compare, where you can test yourself on the questions kids get. The very first question (I said I wanted 12th greade level) was on statistics, and required knowing the factoid that about 95% of all values in a normal distribution lie within two standard deviations of the mean. That's an arbitrary sort of thing, and not representative of important mathematical skills, in my opinion. [Added:] About 1/4 of the questions seem to be about probability and statistics.

Here's the question:

 1 A clock manufacturer has found that the amount of time their clocks gain or lose per week is normally distributed with a mean of 0 minutes and a standard deviation of 0.5 minute, as shown above. In a random sample of 1,500 of their clocks, which of the following is closest to the expected number of clocks that would gain or lose more than 1 minute per week?

The answers provided are 15, 30, 50, 70, and 90. Before teaching statistics, I'd have estimated the area in the tails by sight, and guessed something less than 10% (150), and picked 90, which is wrong.

One bad question isn't enough to invalidate a whole test, but it does make me question the judgment of the people who created it. (Of course, it could be that the website questions aren't as carefully screened as the real test questions...) [Added:] Of course, if all 12th graders are expected to have taken statistics, then there's not much problem. Is that the case these days?

Part II. MA Teacher Licensing Test

Massachusetts recently instituted a math test required for teachers to get licensed. Only 27% of the people taking this test passed it. [The score needed to pass is not available, as far as I can see.] Are the candidates unprepared mathematically? Most likely some of them are. Is the test reasonable? Well, you can get it here, and judge for yourself. [Note: A news article originally led me to the wrong test, what looks like a math subject test for the high school teachers. Thanks, Colleen for pointing out the right one. Both are on this page.]

The test has 45 multiple choice questions and an essay question, in which the test-taker is asked to evaluate a student's work. I took the test just now, and got a few wrong. (Achoo! My brain doesn't work today, with this cold.) About 2/3 of the questions should be pretty straightforward for someone well-prepared. Then there's 12 to 15 questions that will really take some thinking. Is that too much to ask in the stressful situation of a licensing test?

[Commentary on wrong test deleted.]

Here's a quote from a Boston Globe article:

"The failure rate isn't because we have placed some arcane, really hard, high-level math on the test," said Robert Bickerton, associate commissioner for the Elementary and Secondary Education Department, "What we've done is say you need to know this math deeply and you need to be able to answer your students when they ask why."

Hmm, I wonder if he's taken the test. I do want math teachers to be able to think deeply, but this test may be asking a lot. I guess I'd make it a bit shorter, and maybe put the more straightforward questions first. What do you think?

[Burt pointed out some great articles, but his links didn't work. I don't know how to do links in comments, so I'm adding them here: Patricia Kenschaft at 2005 AMS meeting, Racial Equity Requires Teaching Elementary School Teachers More Mathematics, and Deborah Ball et al, Knowing Mathematics for Teaching]

1. You need 240 points to pass the MA test. The scoring is pretty complicated, but that looks to me to be a bit under 75%.

2. Why would base 16, modular arithmetic, Mandelbrot sequences, vectors, trig derivitives, and matrices be on an elementary ed licensing test?

There is a real disconnect between the questions on this test and what I perceive to be the math content knowledge of some elementary teachers. I've interviewed hundreds of teachers over the past 14 years. Most were uncomfortable with topics beyond those you would find in a typical grade 7 math course. Some exhibited very limited knowledge of numbers and arithmetic.

Understanding math concepts at the elementary level was the subject of the book, Knowing and Teaching Elementary Math, by Liping Ma. The author interviewed elementary teachers about basic math concepts. She found there was a significant lack of understanding regarding place value and algorithms. While teachers could carry out the procedures, they could not explain the rationale behind the process.

Clearly, a new test is needed. I can't see any relationship between the questions on the test and what an elementary teacher needs to know in order to be an effective math instructor of young children.

3. Are you sure that was the test given to elementary ed teachers? Here's a link to a math practice test I found for K-5 teachers: http://www.mtel.nesinc.com/PDFs/MA_FLD003_SubtestII_PRACTICE_TEST.pdf

This test is much more on target with what an elementary school teacher needs to know. If this is similar to the actual test, then the problem is much more serious.

4. I found the TIMSS sample to be pretty low difficulty. I didn't think any were worded particularly badly. You are right, the normal curve question was weird.

Colleen, I've noticed many initiatives center around irrelevant band-aids, like "use manipulatives!" "work in coorperative groups!" when really harder work needs to be done for the teachers to gain understanding. They don't need college level math, they actually need to learn the elementary concepts, not just how to proficiently apply an algorithm.

5. I would be quite interested to determine which test the prospective teachers took. There is an extreme difference in the level of difficulty between the two tests.

NicoleJ

6. i don't know anything about statistics
but i know something about statistics *class*.

and typically a student knowing
anything at all about normal distributions
(beyond "symmetric" "bell-shaped",
and "mean equals median") will know
some version of "the empirical rule".
also known, and *better* known, as
the 68-95-99.7 rule...

and, for the pencil-and-paper crowd
anyway (and maybe some of the
all-you-need-is-"technology"
people too who knows), this
generally works out as a *good* thing.

a few baseline *actual numbers* to hang
the concept of a "z-score" on. in effect,
one is learning the values of
P( -z < x < z)
for the z-values 1, 2, and 3 here.

it is altogether fitting and proper that we do so.

7. an important mathematical *fact*,
if you will, rather than an important *skill*.

8. @vlorbik: I didn't happen to know that 'important mathematical fact' until after I began teaching stat, just sayin'. ;>

@Colleen, Kate, and Nicole: Well, I've got egg on my face! Broken link I'll fix, but I need to rewrite my whole post. I was pointed to the test by one of the newspaper articles about it, which must have pointed wrong, because your interpretation makes total sense, Colleen. I should have researched more carefully when I saw how hard it was.

@Kate: Sorry I didn't write down the details of the questions I thought were poorly worded. Seems they give a few different questions each time you do it. I do notice lots of stat-related questions, though the rest are easier to approach with logic. (My last version of it had 5 questions out of 20 on probability or stat.)

I'll go play with the right test now and re-write.

9. @Kate I would be delighted if the K-5 teachers I work with were proficient in math concepts through grade 8. I've seen my own teachers tell students that 5% = 0.5, that -3^2 = 9, and that 0.012 > 0.02 (because there's a 12 after the zero). Fortunately, they're not with us anymore. I would have figured a much higher failure rate based on the original test link.

@Sue I'm sorry you have to rewrite this post. Actually, once you see the real practice test, you will have a whole new topic to explore. 75% failing the first test almost makes sense. It was very difficult; even harder than the SAT IIC math subject test. But 75% failing the second test? Yikes, that's just frightening. I'll look forward to your thoughts on this.

10. I want to think about it more, but I think I'd blame the high failure rate more on stress than on abysmal skills.

Want me to list the questions I thought would take some serious thinking? They're not the same ones I missed,btw. ;> (Those were careless errors.)

And I see, upon re-reading, that I was referring to the (wrong) MA test, not the TIMSS, when I mentioned badly worded questions. Probably not important at this point.

11. So why do 72% of prospective elementary teachers fail a pretty reasonable test?

It asks them to *think* on a third of the questions. And most people have turned off their 'math brains', so that's just asking too much. (Unfortunately.)

I felt like I made significant progress with the people who took mathematical reasoning (our math for elementary teachers course) from me. I'd love to see if my former students would do any better than the average on this test.

We have a vicious circle in our culture. Most people dislike math. Elementary teachers aren't any different than others in this regard, and they infect the kids (who loved math before they went to school). One solution would be to use math specialists from the start. Have people who love math working/playing with the kids. (And make sure the classroom teachers play too, so they can overcome their fears.)

12. "... more on stress".
makes sense to me:
the famous "math phobia".
it takes *peace of mind*
to study mathematics
(this is part of what's

certainly for "thinking" problems.
it's natural that these should
cause much the *most* stress.

one *virtue*... and it *is* a virtue...
of "routine" problems is that of
freeing the mind from the
"things of the world" *without*
great effort. i never would've
believed it of myself but these days
i *like* to unwind with a little
brute-force calculation.

of course, the downside (and it is etc);
one either has or has not practiced
"routine" problems; one knows on sight
what to do... or *adds one more straw*
that is the testee's "peace of mind".

as for the z-score stuff; sure.
i never new it till i taught the course *either*.
but the trend in recent decades
has been to push statistics
i see it as a turf grab
by the enemies of clarity.
paranoid rant by me;
somebody else.

i'd rather *not* do that stuff...
not without first lingering lovingly
over the set theory and probabilty
and gazing deeply on the spectacular
Binomial Theorem... but it appears
to be *standard* material...

13. As Colleen mentioned, Liping Ma’s study showed how poorly prepared elementary school teacher are for teaching math. That is only one of so many studies showing that elementary school teachers in particular do not understand math concepts. Here is an article written by a mathematician who has visited elementary schools to teach math to teachers.
Deborah Ball and her colleagues have been studying the math knowledge required for teaching math and have been working on a test for teachers, which “significantly predicted the size of student gain scores.” Knowing Mathematics for Teaching is a little dated but is a good article.

14. Hi Burt. Those links didn't work. I don't know myself how to get links to show up nicely in comments, but I could add your links to my post. I'd enjoy reading what you tried to point to.

Email me at suevanhattum at hotmail if you want.

15. Thanks for the link, looks good!

(He didn't forget. The system wouldn't let him. I'll get his stuff posted as part of the blog soon.)

16. Oh. I thought you were giving me/us some other links. I see now they're the same ones. Anyway, I've added to the blog post. Thanks to both of you.

17. Great post at Bridging Differences, on the New York state tests, and how they're manipulated.