Psychology experiments and statistics (homework)

Get your science fix here: research, quackery, activism and all the rest
Post Reply
User avatar
Sciolus
Dorkwood
Posts: 1151
Joined: Mon Nov 11, 2019 6:42 pm

Psychology experiments and statistics (homework)

Post by Sciolus » Sat Jan 07, 2023 1:05 pm

A young person of my acquaintance has recently started a psychology degree, and I rashly offered to help her with the statistics module. I just want to check with you lot in case I'm talking out of my arse. Especially the bit where I slag off her lecturers.

A typical question goes something like: We did an experiment to see if intervention X improves psychological trait Y. We took some subjects and did test Z on them before and after the intervention. Here are the results. Should we use a z-test or a t-test? Is this one-tailed or two-tailed? Etc.

My answer to the one-tailed or two-tailed test question is something like:
A one-tailed test would address the question 'does this intervention improve performance on this task' and a two-tailed test would address the question 'does this intervention change, either improve or worsen, performance on this task'. The stated hypothesis is that the intervention improves performance (albeit performance on trait Y rather than test Z), so a one-tailed test is appropriate. However, both of these are reasonable questions to ask, and both should be investigated before implementing the intervention, but it is essential to state the intended analysis in the experimental protocol before carrying out the experiment, not after seeing the results; otherwise there is a danger of p-hacking.
Moreover, this assignment question is poorly structured, as it normalises poor experimental practice by inviting students to consider how they should conduct the analysis after the experiment has been carried out and after seeing the results. The question should be rewritten to follow good experimental practice by asking the questions about the experimental design and the appropriate analysis methods before presenting any results.

(15% of marks are for "demonstrating enhanced understanding", so they've specifically asked for this sort of thing.)

Is that a load of bollocks? I've seen plenty on this forum about bad psychology experiments but it's not really my area.

User avatar
Bird on a Fire
Princess POW
Posts: 10120
Joined: Fri Oct 11, 2019 5:05 pm
Location: Portugal

Re: Psychology experiments and statistics (homework)

Post by Bird on a Fire » Sat Jan 07, 2023 1:26 pm

A one-tailed test effectively doubles your statistical significance, because it is explicitly discounting the hypothesis that the treatment lowers performance. Normally, lower performance is a priori possible, so you'd use a two-tailed test.

The other key thing with those data would be to do a paired test, i.e. comparing each subject's change in score from zero, because the samples aren't independent. Exactly which test would depend on the nature of the data.

A reality of research is that you're often analysing data collected by somebody else, e.g working with a long-term dataset. It's ok to decide an analytical approach based on a hypothesis and the nature of the data (e.g. its distribution might change what test is appropriate anyway). It becomes p-hacking if you try lots of tests and subdivisions and stuff and then only report significant results, but as long as you're clear about the a priori hypotheses you're not hacking anything. As it's generally difficult to compute p-values in your head I don't think there's a danger from merely thinking about tests.
We have the right to a clean, healthy, sustainable environment.

User avatar
Sciolus
Dorkwood
Posts: 1151
Joined: Mon Nov 11, 2019 6:42 pm

Re: Psychology experiments and statistics (homework)

Post by Sciolus » Sun Jan 08, 2023 5:02 pm

Thanks Boaf. I will adapt my advice accordingly.

Post Reply