Psychology experiments and statistics (homework)
Posted: Sat Jan 07, 2023 1:05 pm
A young person of my acquaintance has recently started a psychology degree, and I rashly offered to help her with the statistics module. I just want to check with you lot in case I'm talking out of my arse. Especially the bit where I slag off her lecturers.
A typical question goes something like: We did an experiment to see if intervention X improves psychological trait Y. We took some subjects and did test Z on them before and after the intervention. Here are the results. Should we use a z-test or a t-test? Is this one-tailed or two-tailed? Etc.
My answer to the one-tailed or two-tailed test question is something like:
A one-tailed test would address the question 'does this intervention improve performance on this task' and a two-tailed test would address the question 'does this intervention change, either improve or worsen, performance on this task'. The stated hypothesis is that the intervention improves performance (albeit performance on trait Y rather than test Z), so a one-tailed test is appropriate. However, both of these are reasonable questions to ask, and both should be investigated before implementing the intervention, but it is essential to state the intended analysis in the experimental protocol before carrying out the experiment, not after seeing the results; otherwise there is a danger of p-hacking.
Moreover, this assignment question is poorly structured, as it normalises poor experimental practice by inviting students to consider how they should conduct the analysis after the experiment has been carried out and after seeing the results. The question should be rewritten to follow good experimental practice by asking the questions about the experimental design and the appropriate analysis methods before presenting any results.
(15% of marks are for "demonstrating enhanced understanding", so they've specifically asked for this sort of thing.)
Is that a load of bollocks? I've seen plenty on this forum about bad psychology experiments but it's not really my area.
A typical question goes something like: We did an experiment to see if intervention X improves psychological trait Y. We took some subjects and did test Z on them before and after the intervention. Here are the results. Should we use a z-test or a t-test? Is this one-tailed or two-tailed? Etc.
My answer to the one-tailed or two-tailed test question is something like:
A one-tailed test would address the question 'does this intervention improve performance on this task' and a two-tailed test would address the question 'does this intervention change, either improve or worsen, performance on this task'. The stated hypothesis is that the intervention improves performance (albeit performance on trait Y rather than test Z), so a one-tailed test is appropriate. However, both of these are reasonable questions to ask, and both should be investigated before implementing the intervention, but it is essential to state the intended analysis in the experimental protocol before carrying out the experiment, not after seeing the results; otherwise there is a danger of p-hacking.
Moreover, this assignment question is poorly structured, as it normalises poor experimental practice by inviting students to consider how they should conduct the analysis after the experiment has been carried out and after seeing the results. The question should be rewritten to follow good experimental practice by asking the questions about the experimental design and the appropriate analysis methods before presenting any results.
(15% of marks are for "demonstrating enhanced understanding", so they've specifically asked for this sort of thing.)
Is that a load of bollocks? I've seen plenty on this forum about bad psychology experiments but it's not really my area.