Do not reject the null hypothesis

Get your science fix here: research, quackery, activism and all the rest
Post Reply
User avatar
MavisEnderby
Fleury White
Posts: 10
Joined: Mon Nov 11, 2019 4:28 pm

Do not reject the null hypothesis

Post by MavisEnderby » Wed Nov 13, 2019 8:01 am

I read this on my way to work: It's do not reject, not accept. It got me thinking about the place of maths in STEM. Mathematicians are known for being quite pedantic, and I wonder if non mathematicians would look at this blog and think "accept, do not reject, they are the same thing, why does it matter?" I think it does matter; it took me a long time to understand hypothesis testing, and at least in part I think it was because of the fuzzy way that people talk about it. And even now, when I see stats in published research, I don't dig too deeply into it, and take it on trust that the authors and reviewers and press release writers will not overstate their case.

But does this sort of linguistic dance have an effect on public understanding of research? As a maths communicator and educator, what should I be doing to help the public to better understand and interpret stats, in order to help them make informed decisions? And should I be spending my time being pedantic about language?

Pianissimo
Bank Butt
Posts: 29
Joined: Tue Nov 12, 2019 3:56 pm

Re: Do not reject the null hypothesis

Post by Pianissimo » Wed Nov 13, 2019 8:15 am

I’m perfectly willing to believe that “accept” does not equal “do not reject”, but as someone who hasn’t thought about a null hypothesis in a while I’m struggling to remember why those things are not the same. If someone could enlighten me I’d be grateful.

That said I do think it’s imporant that STEM educators are pedantic about their language, because it avoids as much as possible the potential for misconceptions. I’m not sure what can be done about simplifying statistical results beyond giving them in terms of how likely they are to be wrong. Even that seemingly simple extra information is often missing to the detriment of public understanding.

User avatar
MavisEnderby
Fleury White
Posts: 10
Joined: Mon Nov 11, 2019 4:28 pm

Re: Do not reject the null hypothesis

Post by MavisEnderby » Wed Nov 13, 2019 8:28 am

The way I think about it is that all we're saying is that a particular statistical test has shown that the results we got in a particular experiment are quite likely to occur by chance rather than causal relationship, so we do not have sufficient evidence to reject the null hypothesis in favour of our alternative hypothesis. But then, some pretty shoddy statistics has happened in the past when people have kept collecting data until they get the statistical results they want, so I'm wary of saying "we don't accept the null hypothesis because we aren't saying it's true, we're just saying it's probably not false, and more data might overturn that..."

User avatar
El Pollo Diablo
Stummy Beige
Posts: 3323
Joined: Wed Sep 25, 2019 4:41 pm
Location: FBPE

Re: Do not reject the null hypothesis

Post by El Pollo Diablo » Wed Nov 13, 2019 8:35 am

Yes, I think that's fair. The null hypothesis is just a comparator. If you tested ten supposed psychics for their predictive powers and the tests showed they all performed as well as random chance except one, who was a bit better, then your special one might be psychic, but probably just got lucky. But we don't know.

"Do not reject" is, to my logical mind, about keeping possibilities open. "Accept" is about closing them down. The latter is for when you're very confident it is that, the former for when you're not confident it's something else
If truth is many-sided, mendacity is many-tongued

User avatar
Fishnut
After Pie
Posts: 2447
Joined: Mon Nov 11, 2019 1:15 pm
Location: UK

Re: Do not reject the null hypothesis

Post by Fishnut » Wed Nov 13, 2019 9:14 am

I've been doing some reading on the philosophy of science recently and I think the "do not reject" is because, as EPD says, new data can come in. It's the black swan problem - our hypothesis that all swans are white is confirmed as long as we are talking in a northern hemispheric context. As soon as we venture south of the equator with our sampling and discover black swans in Western Australia our hypothesis is no longer "not rejected". It's also one of the reasons why "the hypothesis works" isn't actually a great way of verifying whether something is scientific or not. Phlogiston theory worked, Ptolemy's epicycles worked. But they were wrong.
it's okay to say "I don't know"

User avatar
MavisEnderby
Fleury White
Posts: 10
Joined: Mon Nov 11, 2019 4:28 pm

Re: Do not reject the null hypothesis

Post by MavisEnderby » Wed Nov 13, 2019 12:42 pm

Yeah this is something I allude to in maths talks about the difference between knowledge in maths and in science; in maths, we prove stuff and it stays proved (within a particular set of axioms), in science we have the best explanation for now, but we might get better explanations in the future. This is also why I hate "scientists have found the mathematical formula that explains..." stories. It should say "scientists have found that our current data fits this model quite well", but that doesn't sell newspapers.

User avatar
dyqik
Princess POW
Posts: 7524
Joined: Wed Sep 25, 2019 4:19 pm
Location: Masshole
Contact:

Re: Do not reject the null hypothesis

Post by dyqik » Wed Nov 13, 2019 12:55 pm

On that difference, I wouldn't call epicycles wrong, as such - at least not as a description of the motions of planets, although they are wrong if you try to use them as a cause of the motions of the planets. They are a geometric model that works pretty well, and can be extended to work as well as you'd like. It's just not the simplest description of the behavior at a specific level of accuracy.

I think of epicycles as a kind of harmonic expansion of the actual motion, and so they're no more wrong than a Fourier transform of some other motion is.

minusnine
Gray Pubic
Posts: 18
Joined: Mon Nov 11, 2019 1:31 pm

Re: Do not reject the null hypothesis

Post by minusnine » Wed Nov 13, 2019 1:44 pm

In my mind this point is entangled with the question of what we mean by a 'negative result', usually because I'm thinking about both in the context of talking to my project students about experimental design and data interpretation rather than PUS per se. A negative result is not one where we find that something doesn't work, it's one where we can't tell whether something works or not, either because the data doesn't distinguish between the two outcomes or the experiment was badly designed.

Not rejecting the null hypothesis is then similar, in that if you can't reject the null hypothesis then you can't say whether the data supports the hypothesis or not - it doesn't mean that the hypothesis is disproved. Accepting the null hypothesis would then require some other evidence, strictly speaking. But in a PUS context, especially around topics where deeply held beliefs or motivations hold sway, that answer leaves open the response of "so you haven't proved it isn't working",or more insidiously, "it works for me".

User avatar
dyqik
Princess POW
Posts: 7524
Joined: Wed Sep 25, 2019 4:19 pm
Location: Masshole
Contact:

Re: Do not reject the null hypothesis

Post by dyqik » Wed Nov 13, 2019 2:00 pm

minusnine wrote:
Wed Nov 13, 2019 1:44 pm
In my mind this point is entangled with the question of what we mean by a 'negative result', usually because I'm thinking about both in the context of talking to my project students about experimental design and data interpretation rather than PUS per se. A negative result is not one where we find that something doesn't work, it's one where we can't tell whether something works or not, either because the data doesn't distinguish between the two outcomes or the experiment was badly designed.
I think I'd call this a null result rather than a negative result (and also maybe a failed experiment if nothing was learnt at all). To me, a negative result is one that excludes the hypothesis but not the null hypothesis, a positive result is one that excludes the null hypothesis but not the one being tested, if you are testing say a new treatment against an existing one, or a tweak to an astronomical instrument against its performance before (think of the optician with the "better or worse" game for that...)

However, I'm more likely to be involved in astronomical and cosmological observations, where often the aim is to make detection of a new source or place a new lower limit on the level of a particular kind of CMB anisotropy. There a failed experiment/null result is one which doesn't place a new lower limit on the level or emission or that doesn't detect the emission above background. A "negative" result is one that lowers the limit on the level of emission but doesn't make a detection, and the "positive" result is one that detects the emission and determines its level.

The other kinds of outputs here are either a Bayesian model selection exercise, where e.g. a cosmological model with and without dark energy is fitted to the data, or where the best fit values of cosmological parameters are extracted. A null result is one that doesn't move the confidence limits on the parameters, a positive or negative result can move those limits.

minusnine
Gray Pubic
Posts: 18
Joined: Mon Nov 11, 2019 1:31 pm

Re: Do not reject the null hypothesis

Post by minusnine » Wed Nov 13, 2019 4:29 pm

Although it sounds like we're working at length scales that are about 25 orders of magnitude apart, your description of your experiments is not that different in principle from what my students are doing (making simple nanoparticle assays and finding their limits of detection). Our aims are different, though. Because my aim is primarily educational rather than properly advancing the state of the art (I've got a cheap, old bench spectrometer and... not much else, and each student has about 8 weeks in the lab - we've not generated anything close to publishable) I'm not so concerned with the distinction you draw between null/failed and negative. Whether my students generate results that are negative in the sense you use or positive is less important than whether or not they are negative (in the sense I used it)/null (in your sense).
Which only goes to show, perhaps, that as we try to be more precise with our language we risk excluding nuances that are critical in one context but not in another.

User avatar
science_fox
Snowbonk
Posts: 512
Joined: Mon Nov 11, 2019 1:34 pm
Location: Manchester

Re: Do not reject the null hypothesis

Post by science_fox » Wed Nov 13, 2019 4:50 pm

Biology/analytical chemistry is somewhere between those two scales!

Generally a hypothesis is whether a modification/mechanism works as expected. This is confirmed by experiments producing results consistent with that expectation. There's often insufficient consideration given to the formal H0 process resulting in confusion between a) 'it's doing something just not what we expected' and b) 'it's not made any difference' the latter followed by 'did we do it properly?'

a) is a fairly clear case to not reject the null, ie our expectation of the mechanism is as yet insufficient.
b) is trickier? technically I suppose it should also be 'not rejected' as it would need positive evidence of an alternative mechanism.

Mostly though any hint of having found what was wanted is taken as 'proof' that the desired process has occurred.
I'm not afraid of catching Covid, I'm afraid of catching idiot.

Chris Preston
Snowbonk
Posts: 529
Joined: Tue Nov 12, 2019 8:05 am

Re: Do not reject the null hypothesis

Post by Chris Preston » Wed Nov 13, 2019 9:34 pm

It is more an epistemology thing. They way that null hypotheses are set up is to test for rejection. If the null hypothesis is rejected all well and good. If the test does not do so, then it is not a case for accepting the null hypothesis, because the test was not set up to do that. You just do not reject it.

It the real world, what you do is use the information gained to build a new null hypothesis to test. Once the hypothesis has not been rejected under enough different conditions, it becomes an accepted as a good description of the situation. It can always be rejected later if new information comes to light.
Here grows much rhubarb.

User avatar
dyqik
Princess POW
Posts: 7524
Joined: Wed Sep 25, 2019 4:19 pm
Location: Masshole
Contact:

Re: Do not reject the null hypothesis

Post by dyqik » Wed Nov 13, 2019 9:42 pm

minusnine wrote:
Wed Nov 13, 2019 4:29 pm
Although it sounds like we're working at length scales that are about 25 orders of magnitude apart.
My research basically covers at least 24 orders of magnitude in distance (Angstrom thickness of a graphene sheet or AlO barrier in superconducting tunnel junction to the Earth's diameter), more if you include cosmology as part of my research - Big Bang cosmology and inflation basically covers the Planck length to the Hubble distance, which is 61 orders of magnitude.

User avatar
shpalman
Princess POW
Posts: 8241
Joined: Mon Nov 11, 2019 12:53 pm
Location: One step beyond
Contact:

Re: Do not reject the null hypothesis

Post by shpalman » Wed Nov 13, 2019 10:21 pm

Chris Preston wrote:
Wed Nov 13, 2019 9:34 pm
It is more an epistemology thing. They way that null hypotheses are set up is to test for rejection. If the null hypothesis is rejected all well and good. If the test does not do so, then it is not a case for accepting the null hypothesis, because the test was not set up to do that. You just do not reject it.

It the real world, what you do is use the information gained to build a new null hypothesis to test. Once the hypothesis has not been rejected under enough different conditions, it becomes an accepted as a good description of the situation. It can always be rejected later if new information comes to light.
I'm not sure I understand the second part: the null hypothesis is only defined in terms of what you're trying to test, isn't it? It's the hypothesis that whatever you're trying to test hasn't actually done anything. A test whose results allow you to reject the null hypothesis corroborates the hypothesis under which that test was conceived, which you can then refine and test again some other way. If your test results can't reject the null hypothesis, well, you might at least know what your theory shouldn't be, but you still have to go and think of something else.

Have I got this completely the wrong way around? Can you give an example of how it's the null hypothesis which becomes the theory?

(Yes I know you can never prove a theory but not all falsifiable but not-yet-falsified theories are as useful as each other.)
having that swing is a necessary but not sufficient condition for it meaning a thing
@shpalman@mastodon.me.uk

KAJ
Fuzzable
Posts: 310
Joined: Thu Nov 14, 2019 5:05 pm
Location: UK

Re: Do not reject the null hypothesis

Post by KAJ » Thu Nov 14, 2019 5:34 pm

shpalman wrote:
Wed Nov 13, 2019 10:21 pm
Chris Preston wrote:
Wed Nov 13, 2019 9:34 pm
It is more an epistemology thing. They way that null hypotheses are set up is to test for rejection. If the null hypothesis is rejected all well and good. If the test does not do so, then it is not a case for accepting the null hypothesis, because the test was not set up to do that. You just do not reject it.

It the real world, what you do is use the information gained to build a new null hypothesis to test. Once the hypothesis has not been rejected under enough different conditions, it becomes an accepted as a good description of the situation. It can always be rejected later if new information comes to light.
I'm not sure I understand the second part: the null hypothesis is only defined in terms of what you're trying to test, isn't it? It's the hypothesis that whatever you're trying to test hasn't actually done anything. A test whose results allow you to reject the null hypothesis corroborates the hypothesis under which that test was conceived, which you can then refine and test again some other way. If your test results can't reject the null hypothesis, well, you might at least know what your theory shouldn't be, but you still have to go and think of something else.

Have I got this completely the wrong way around? Can you give an example of how it's the null hypothesis which becomes the theory?

(Yes I know you can never prove a theory but not all falsifiable but not-yet-falsified theories are as useful as each other.)
Hmmm. The null hypothesis is defined in terms of the data. It is [usually] developed as a contrast to the technical hypothesis = "what you're trying to test".
The important thing to remember is that the p-value is the probability of the observed data given the null hypothesis, not the probability of the null hypothesis given the observed data. These can be very different - the probability that a person is male given they are Prime Minister is very different from the probability that they are Prime Minister given they are male.
The null hypothesis must [almost always] be defined as a point value, in order to calculate the p-value. Inter alia that means you are [almost always] very confident that the null hypothesis is not [exactly] true.
These are among the reasons that null hypothesis significance testing [NHST] is frowned upon by many who have thought about it. Personally, I regard interval estimation as more useful, and NHST results are often better interpreted as interval estimates.

Failure to reject the null hypothesis is indeed evidence (NOT proof) that the null hypothesis is [at least] approximately true, Absence of Evidence IS Evidence of Absence link.

KAJ
Fuzzable
Posts: 310
Joined: Thu Nov 14, 2019 5:05 pm
Location: UK

Re: Do not reject the null hypothesis

Post by KAJ » Thu Nov 14, 2019 6:12 pm

Whoops. Correction after the edit window.

The null hypothesis is defined in terms of the population from which the data is derived.

KAJ
Fuzzable
Posts: 310
Joined: Thu Nov 14, 2019 5:05 pm
Location: UK

Re: Do not reject the null hypothesis

Post by KAJ » Fri Nov 15, 2019 9:51 pm

That the p-value [of a NHST] is the likelihood of the observed data [actually the test statistic] given the null hypothesis, not the probability of the null hypothesis given the observed data is central to understanding the issue in the OP - do not reject the null hypothesis as distinct from accepting it.

Accepting the null hypothesis means deciding it is more probable than the alternate hypothesis, but a NHST doesn't give probability estimates for either hypothesis.
  • A low p-value allows a conclusion. IF the data is a "proper" random sample so that it's real life likelihood is quite high (no "cherry picking") then a low NHST p-value means the probability of the null hypothesis is small - the data is not consistent with the null hypothesis. The null and alternate hypotheses are complementary - complete and mutually exclusive - so the probability of the alternate hypothesis must be high.
  • But a high p-value does not lead to the opposite conclusion. The data is consistent with the null hypothesis, but we do not know if it is consistent with the alternate hypothesis.
    And we cannot compare the likelihood of the data under the two hypotheses. The null hypothesis is point valued, allowing calculation of a data likelihood. The alternate hypothesis is the set of all other possible values, we cannot calculate a data likelihood under such a hypothesis.
So a non-significant NHST really tells us very little and gives little reason to change a pre-existing opinion - if (IF) we accepted a hypothesis before the NHST we can continue to accept it after.

We tend to conflate the statistical null hypothesis - point valued, low probability, used to calculate data likelihood, with the default technical hypothesis - which we are minded to accept in the absence of contrary evidence. A non-significant NHST, giving little reason to change, allows continued acceptance of the default technical hypothesis.

But often the null hypothesis is constructed to contrast a test technical hypothesis ("what you're trying to test") which is the default - we may be seeking support. The null hypothesis is defined to complement that test technical hypothesis and corresponds to an a-priori unlikely "null technical" hypothesis. A non-significant NHST, giving little reason to change, does not support accepting the null technical hypothesis - failing to reject the null hypothesis does not necessarily mean accepting it.

So "do not reject the null" is always a safe expression, "[continue to] accept the null" may be unsafe.

But the real problem is in the use and interpretation of NHSTs. We really want to compare the probabilities of technical hypotheses given the data. But we cannot calculate the probabilities of either hypothesis, we cannot even compare the likelihoods of the data under the two hypotheses.

The answer? Depends which statistician you ask. Bayesians might try to calculate hypothesis probabilities - but others have problems with priors. I like [confidence] interval estimation - but they have substantial problems as well. Really, I think the right answer is to avoid "recipes" and see what the data [analysis] adds to understanding - use judgement.

Going back to the OP:
As a maths communicator and educator, what should I be doing to help the public to better understand and interpret stats, in order to help them make informed decisions? And should I be spending my time being pedantic about language?
Tell them that statistics is as much a judgement based art as other STEM techniques, and that the ubiquity of NHSTs doesn't mean they are widely appropriate or easy to interpret. And, yes, spend some time being pedantic about language - when it matters.

Post Reply