An article in the most recent issue of TESOL Quarterly examines the efficacy of teaching Japanese university students the metaphors behind some phrasal verbs in comparison to simply explaining the verbs though Japanese. The treatment group (n=59), was given something like the following for each of the focussed prepositions (up, down, into, out, off):
In contrast, the control group was merely given translations with no grouping. For example:
- call off = 中止する
- calm down = 落ち着く
Fill in the blanks with the words given below so that the sentence will make sense.
- When she heard the news, she burst ( ) tears.
- No one can figure ( ) how the fire started.
As you can see, the control group just beat out the experimental group in the studied words, but the experimental group did somewhat better in the new items. It turns out that this second difference is statistically significant. But what about the effect size? It's not published, but I calculate Cohen's d at .68, which is a moderate to large effect. If you add the two halves of the test together, though, you get a very modest 0.25 effect.
Here's how the author interprets the results:
"Thus the results of the experiment provide evidence for the claim that learning phrasal verbs can be greatly aided by increasing the awareness of orientational metaphors than (sic) by encouraging mere memorization." (emphasis added)What do you think? Is this spin or honest reporting?
> If you add the two halves of the test together, though, you get a very modest 0.25 effect.
ReplyDeleteI'm not sure it makes sense to simply combine the two halves of the test, giving them equal weighting: though each half contains only fifteen words, one half also represents only fifteen words, whereas the other is assumed to be representative of a much larger group.
Overall, I think the best way to summarize these results would be with two sentences, one summarizing one half and one summarizing the other, without trying to synthesize them into one big picture.
(That said, I think the research design may have sold the claim short. I'm guessing that if you gave the students another test a week later, you'd find that the experimental group's advantage would spread a bit into the instructed words: both groups would forget some instructed words, but the experimental group can cope with that somewhat. But on the other hand, they'd probably lose some advantage with the new items, as they'd have forgotten some of the metaphors. So who knows? It's too bad the researchers didn't test that.)
I'm also not sure about the merit of combining the two halves, but I put it out there for the sake of completeness.
ReplyDeleteIt does seem the author is overstating the robustness of the findings. Ran has a point that some followup could provide more support for the claim. However, it was not done; thus, the researcher should be a little more cautious in making the claim.
ReplyDelete