next »

[st_alfonso]

Burying a result, that potentially may help many patients, that is 93% likely to not be due to to chance

1-p is not the probability that a statistical finding is not "due to chance." This is a common misinterpretation of p values.

[jackit]

Alright, St. A, I'll give you that.

Nonetheless, p = 0.07 does mean that it is unlikely, even quite unlikely, that the observations arise from a null difference.  As another forumite said, you need the confidence intervals to talk about the observation itself.

Much of what I am going on about here has to do with statistical power.   It really is too bad that the word 'significance' was hijacked to be associated with a level of p.  In practice, the 'significance' of a finding inevitably involves interpreting the power of the study.  Over-powered studies can detect clinically insignificant effects.  Likewise, it is unwise to dismiss big effects in underpowered studies -- despite the lack of so-called 'significance.'  In practice, the statistical significance of a finding should probably take into account both power the probability of declaring a false positive.  But I realize that the ship has left the dock...

[drgunn]

This discussion makes me glad, for once, that in my field statistical analysis is like the wild west.

What kind of science do you do, sciencephd?  In my area, molecular biology, the p<0.05 = magic idea definitely holds sway, among researchers, grant reviewers, and journal editors.  Of course, that didn't stop me from reporting the 0.06 and 0.07 findings I had that were in line with the rest of the results.  On my poster, I did in fact use *=p<0.05, **=p>0.1  When anyone asked, and they did, I just said those are indicated because, though they don't meet the conventional threshhold,  they're consistent with the rest of the results.

[drbeeper]

I do not think I would ever use the term "nearing significance", but in a paper I am currently writting I do use the term "not nearing signifigance (P=0.68)." I am not sure why I like one, but not the other.

[norvell]

I do not think I would ever use the term "nearing significance", but in a paper I am currently writting I do use the term "not nearing signifigance (P=0.68)." I am not sure why I like one, but not the other.

It it's good for the goose, it's good for the gander; I wouldn't use either.

[jackit]

And has anyone else out there used the term 'highly significant'?


Don't be shy.

[barred_owl]

And has anyone else out there used the term 'highly significant'?


Don't be shy.

Not me.  But I think Radar O'Reilly did once in an episode of M*A*S*H in which he was attempting to flirt with a rather intellectual nurse.  Really.  And his use of the expression had about the same impact (i.e., none) with the nurse as it might have if used in describing results, I think.

[sciencephd]

This is both significant and highly unique.

[tt_wannabe]

Not me.  But I think Radar O'Reilly did once in an episode of M*A*S*H in which he was attempting to flirt with a rather intellectual nurse.  Really.  And his use of the expression had about the same impact (i.e., none) with the nurse as it might have if used in describing results, I think.

Ahh, Bach.

[al_wallace]

At the risk of outing myself, I actually published a paper using the expression "desperately yearning toward significance (p = 0.0503)". I originally put it in as a joke and then forgot to take it out. It got through review and I noticed it at the proof stage and then said, "eh, what the heck" and let it go. 

[educator1]

I have never understood the desire to imbue a particular p value with near religious significance. It is merely the probability of error (Type 1) if one rejects the null hypothesis (assuming that the model and the data meet the assumptions - if not, we are merely estimating p itself).
That consequences of that risk certainly varies from application to application as between say, drug studies with high side effect consequences and deciding between two approaches to teaching statistics (you still have to pick one). The decision on what risk to accept should depend on the consequences implied by that risk. This is statistical significance.
The other discussion that seems to have been intertwined in this thread is the issue of practical significance. A very minor difference can be statistically significant. This gets into effect sizes and other measures of the practical use of the difference or pattern discovered. There can be no practical significance, however, unless we find the risk of rejecting the null to be acceptable.

[kedves]

At the risk of outing myself, I actually published a paper using the expression "desperately yearning toward significance (p = 0.0503)". I originally put it in as a joke and then forgot to take it out. It got through review and I noticed it at the proof stage and then said, "eh, what the heck" and let it go. 

I like this ever so much.

[temporaryname]

And has anyone else out there used the term 'highly significant'?

Don't be shy.

I think it (or something quite similar) is in my dissertation, and it's common in my subfield. Since I completed my PhD, though, I've learned real statistics (i.e., not the bare-bones, non-state-of-the-current-art stats I was taught in my graduate program), and so I don't do it any more. (Nor do I allow it in articles I review.)

(This whole thing is one reason I'm very much not a fan of graduate programs offering courses outside of their actual subject area while steering their students toward those and away from the real experts.)

[drbeeper]

I would (and think I have) used highly significant. I have been thinking about why I am okay with highly significant and not approaching significance, but dislike nearing significance. To me the term nearing significance implies knowing exactly where the magical line is, but the other two terms seem to imply that you are not really sure where the line is, but you know that wherever it is, you are no where near it.

Treating significance as a binary object (either it is or it isn't) seems a little strange to me. To my understanding, when we say something is significant we mean that the likelihood that the data came from the null hypothesis is low. Therefore, the "degree" of signficance is something like the distance the actual p-value is from the magical p-value. Maybe this is why I shoud start to also consider the effect size.

[drspouse]

I've used "approaching" (not "nearing") significance, as one is supposed to report the actual p value anyway (or at least the range, if you are reporting a lot in a table for example) then that's what matters.  So if you have * for <.05, ** for <.01 then it seems fine to report .054 as is.

Sometimes if you are going on to carry out further analyses with variables that are significantly correlated, it seems even rather dishonest to include those with p=.049 and exclude those with p=.051.

I've also used the very helpful "significant at the one-tailed level", which if you have predicted the direction is perfectly valid.