next »

[jackit]

in medicine, p < 0.05 is adopted as a convention, so that different authors don't move the goal posts.  The word significance is used to literally denote that 0.05 or less has been achieved.

Having said that, it is perfectly reasonable to report correlations and their p values if they hover slightly above 0.05.  'Trending towards significance' is a reasonable phrase for small samples.

[namazu]

Having said that, it is perfectly reasonable to report correlations and their p values if they hover slightly above 0.05.  'Trending towards significance' is a reasonable phrase for small samples.

I kind of dislike this construction because it seems to imbue the data with a will and desires that more properly belong to the investigator(s).  (This seems related to what DvF said above, I guess.)  That said, I'm sure I've worded things in similarly "aspirational" ways.

Also, to me "trend" implies something more specific and directional (over time or across ordered categories) than "p-value slightly above the significance threshold".

In general, I concur with the "God surely loves..." people, and prefer to report effect sizes and confidence intervals where possible. 

I'm also surprised no one's brought up Bayesian methods yet...but that's another can of worms.

[jackit]

I see your point.  But the 'trend' can be said to exist because, if enough new data is collected, and the new data have the same correlation, then significance will be reached.

[tee_bee]

I agree with the last three posters that such a p value can be of interest, but when people mention a p close to .1 it often seems to me to be a wistful thing, as if the result really is significant but that the statistics aren't behaving properly this time out of spite. - DvF

What p<0.10 means is that the probability of this outcome happening randomly is 10 times in 100. That's it. Any further attribution of meaning to this statistic is more an act of rhetoric than an act of science. It's all about what level of statistical "confidence" one is willing to tolerate in "science." Some fields may have developed a standard expectation of a certain confidence interval, like .05. But in some cases, interesting results are worth reporting, with some caveats, at other p-values, as many have noted. And, in some cases, like in educational or other public policy interventions, a nine-in-ten probability that the effect is significant is better information than anyone else may have, and is often worth acting upon in applied settings.

It is wrong to say that something "approaches significance" as one gets to .05, or .001, or .0001, or whatever, because all we can say is that something is significant at a particular level. My sociology stats prof in grad school gave us all an article from a psych journal illustrating the widely incorrect use of the term "more significant" to describe a model or variable where the p-value is less than some other model or variable.

in medicine, p < 0.05 is adopted as a convention, so that different authors don't move the goal posts.  The word significance is used to literally denote that 0.05 or less has been achieved.

Having said that, it is perfectly reasonable to report correlations and their p values if they hover slightly above 0.05.  'Trending towards significance' is a reasonable phrase for small samples.

It's really not a reasonable phrase for any sample size for the reasons I just cited. A p value is not more or less significant than any other p value. It's merely significant at that level. Then you're left with a rhetorical or evidentiary convention or argument, but not a statistical question.

[daniel_von_flanagan]

I agree with the last three posters that such a p value can be of interest, but when people mention a p close to .1 it often seems to me to be a wistful thing, as if the result really is significant but that the statistics aren't behaving properly this time out of spite. - DvF

What p<0.10 means is that the probability of this outcome happening randomly is 10 times in 100. That's it.

Strictly speaking, what it means is that if the underlying model is correct, and if the null hypothesis holds, then the probability of the observed outcome happening is at most 1 in 10.

Of course, as Namazu mentions, Bayesian methods can lead to p values with conceptually different interpretations. - DvF

[tee_bee]

I agree with the last three posters that such a p value can be of interest, but when people mention a p close to .1 it often seems to me to be a wistful thing, as if the result really is significant but that the statistics aren't behaving properly this time out of spite. - DvF

What p<0.10 means is that the probability of this outcome happening randomly is 10 times in 100. That's it.

Strictly speaking, what it means is that if the underlying model is correct, and if the null hypothesis holds, then the probability of the observed outcome happening is at most 1 in 10.

Of course, as Namazu mentions, Bayesian methods can lead to p values with conceptually different interpretations. - DvF

True enough. Thanks for pointing this out--this is indeed an important point about proper model specification.

[jackit]

...

in medicine, p < 0.05 is adopted as a convention, so that different authors don't move the goal posts.  The word significance is used to literally denote that 0.05 or less has been achieved.

Having said that, it is perfectly reasonable to report correlations and their p values if they hover slightly above 0.05.  'Trending towards significance' is a reasonable phrase for small samples.

It's really not a reasonable phrase for any sample size for the reasons I just cited. A p value is not more or less significant than any other p value. It's merely significant at that level. Then you're left with a rhetorical or evidentiary convention or argument, but not a statistical question.

I said 'trending towards significance', not 'more significant.'  However, I will admit to using 'more significant' when I really mean 'more likely to be a non-random association'.  But isn't the terminology 'most significant predictor' actually completely commonplace in medicine?  I think so.  And we all know what it means:  some p values less than 0.05 but one being less than the others and so being designated 'most significant'.  I know, it's sloppy language.  But there you have it.

If you have a good substitute I will gladly use it!

[prof_smartypants]

just say "significant at" the .01, .05, or .1 level. Interpret as you see fit. It's not "more significant", it's just more likely to be a real relationship.

For me, when learning statistics, I found the term "significant" extremely confusing. What I wanted it to mean was that the X variable was a more important predictor of Y than other variables. I really, really wanted to use terms like "more significant" and "nearing significance". My dissertation advisor, who did exclusively qualitative research, used these terms all the time. Luckily I had other committee members - economists, sociologists, political scientists - who drilled it into my head that this is not correct. I  think a lot of students confuse this because the vocabulary used is misleading. It took me quite some time (and granted, I'm not math head - I had a really tough time with stats) to finally wrap my tiny head around these premises and to fell comfortable talking about not only the p-values, but the coefficients, interactions, etc. Working through my dissertation finally made me comfortable with the language.

I also think there is too much emphasis on the p-value that people forget about everything else - coefficients, model specification and my personal area of study - construct validity. I can't tell you how many conferences I've gone to and watched presentations on highly complicated mathematical models that were based on ridiculous proxy variables (I used "church attendance" as a proxy for "social capital"  --- WHAT???). And all people talked about was how robust their model was and their strong r-square and p-values. It's all pointless if your variables don't make any sense.
 

[sciencephd]

Alternatives: "more better" or "bestest".

[onestep]

OP, did you hypothesize these associations, were their coefficients large, and were their confidence limits wide?  If the answer is yes to all, then it is likely that you have insufficient power, and reporting that you have no association risks type 2 error.  Hence, I would report that your associations were in the hypothesized direction, but that the trend was not statistically significant.  It's important also to recall that many statistical packages default to 2-tailed tests, but  you are justified in using a 1-tailed test with an a priori hypothesis.  

That said, if the effect size was small (and assuming your sample was unbiased), then a larger sample may become statistically significant but still not be informative in the big picture.  That is, with a sample of a billion people, you might find that X is related to Y, but that association would be unimportant in theory and/or practice.  

If you were simply data dredging, I would call it not statistically significant and move on.

So, I would focus on the hypotheses and effect sizes, then state the facts of your findings, and let the reader make up their own mind.  

[madhatter]

That said, if the effect size was small (and assuming your sample was unbiased), then a larger sample may become statistically significant but still not be informative in the big picture.  That is, with a sample of a billion people, you might find that X is related to Y, but that association would be unimportant in theory and/or practice.  

It doesn't take that large of a sample to render significance tests meaningless. I had a survey data set with about 1.5 million records. I could get statistical significance on any pairwise comparison I cared to make, but the actual differentiation in distribution of scores around the means was barely perceptible.

[locutus]

I'm also surprised no one's brought up Bayesian methods yet...but that's another can of worms.

Can we? In my area opinions vary from essentially the same at a practical level all the way to statistical messiah. I think I'm somewhere in the middle. I haven't tried using them in a paper because it would likely be a very uphill battle.

[gennimom]

We talk about about being significant at the .05 level or the .01 level.

A lot of people like to say that something that is .06 (or something similar) is almost significant. As one of my professors taught us, almost significant is like being almost pregnant. Either you are or you aren't.

[neil9]

This difficulty is all due to the ill-conceived p-values. It would be much clearer if confidence limits are used instead.

[tintern]

Just remember that the effect size is the really important thing, especially with small samples.