next »

[shrek]

Well, the problem with "nearing" is you don't really know if it's "nearing" or "far-ing". Let me tell you a little story. One of my doctoral students ran a study where she collected data on some 50 subjects. There was a p = .052 result. Oh how we were tempted to say it was nearing significance (there were other interesting findings, but this one was really really cool, if significant). The effect size was small tho' so that wasn't going to help. I said, add 20 subjects (let me tell you doc students aren't thrilled with me when I say stuff like this), but we were under powered for the number of comparisons and while it was the first study that did what we were doing & we'd have a certain amount of leeway in terms of power it made sense to have clear results. So, we ran 20 more subjects (and had to put off completion of the paper till we did this).

And GUESS WHAT??

p = .09.

Well hell. NOT Significant!

[jackit]

shrek - I'm thinking of stuff like odds ratio = 1.6, but p = 0.07 in a small sample.  If the odds ratio stays > 1, then p eventually reaches significance (< 0.05).

& correlations with p < 0.05 can also go away with new or larger samples, especially if you've been data dredging.

Having said all that, you do have a point.

[hobbit]

Just remember that the effect size is the really important thing, especially with small samples.

I seem to recall that effect sizes can be computed for ANOVA and linear regression (with weighted least squares for small samples, though), but not for some other types of models. Effect sizes apparently have not yet been worked out for mixed effect models, for example, or at least there isn't any consensus on them, as far as I can tell. 

[jackit]

I agree with tintern.  If you have a large effect in a small sample, it can amount to impressive evidence that something important has been found, even if p < 0.05 was not observed.

On the other hand, a small effect in a huge dataset may be relatively unimportant, despite being 'statistically significant.'

[sciencephd]

I agree with tintern.  If you have a large effect in a small sample, it can amount to impressive evidence that something important has been found, even if p < 0.05 was not observed.

On the other hand, a small effect in a huge dataset may be relatively unimportant, despite being 'statistically significant.'

Of course it is possible to make exactly the opposite argument in an equally convincing manner.

[jackit]

I agree with tintern.  If you have a large effect in a small sample, it can amount to impressive evidence that something important has been found, even if p < 0.05 was not observed.

On the other hand, a small effect in a huge dataset may be relatively unimportant, despite being 'statistically significant.'

Of course it is possible to make exactly the opposite argument in an equally convincing manner.

Please do.

[sciencephd]

I agree with tintern.  If you have a large effect in a small sample, it can amount to impressive evidence that something important has been found, even if p < 0.05 was not observed.

On the other hand, a small effect in a huge dataset may be relatively unimportant, despite being 'statistically significant.'

Of course it is possible to make exactly the opposite argument in an equally convincing manner.

Please do.

This is basic statistics, but if you want proactical applications based examples, see the bioinformatics literature.  Fold-change limits in, for example gene chip datasets, which have been quite popular because of such "intuitive" or common sense type arguments,  have been completely debunked as statistically unsound.

[jackit]

I agree with tintern.  If you have a large effect in a small sample, it can amount to impressive evidence that something important has been found, even if p < 0.05 was not observed.

On the other hand, a small effect in a huge dataset may be relatively unimportant, despite being 'statistically significant.'

Of course it is possible to make exactly the opposite argument in an equally convincing manner.

Please do.

This is basic statistics, but if you want proactical applications based examples, see the bioinformatics literature.  Fold-change limits in, for example gene chip datasets, which have been quite popular because of such "intuitive" or common sense type arguments,  have been completely debunked as statistically unsound.

Microarray analysis is typically based on fold-change.  But the level considered significant has to be adjusted based on the large number of comparisons (False Discovery Rate, Bonferroni, etc.).

[daniel_von_flanagan]

I just flipped a coin three times, and the data analysis said that it was twice as likely to fall "heads" as "tails".  The p-value was high, but what a large effect! - DvF

[prof_smartypants]

If you don't have the statistics to back up your claims, you have a problem. If you have a large effect in a small sample with a non-significant p-value, say as much. Do not say it's "nearing significance." This is not something to argue about. Write up your results accurately. "Nearing Significance" is not accurate.

[sciencephd]

I agree with tintern.  If you have a large effect in a small sample, it can amount to impressive evidence that something important has been found, even if p < 0.05 was not observed.

On the other hand, a small effect in a huge dataset may be relatively unimportant, despite being 'statistically significant.'

Of course it is possible to make exactly the opposite argument in an equally convincing manner.

Please do.

This is basic statistics, but if you want proactical applications based examples, see the bioinformatics literature.  Fold-change limits in, for example gene chip datasets, which have been quite popular because of such "intuitive" or common sense type arguments,  have been completely debunked as statistically unsound.

Microarray analysis is typically based on fold-change.  But the level considered significant has to be adjusted based on the large number of comparisons (False Discovery Rate, Bonferroni, etc.).

The argument is about small sample sizes and large effects being "better".  With small sample sizes this is more likely to be do to random chance.  The fact that the effect is large is then irrelevant.

The microarray example was a perhaps too complex example.  However, just to clarify, part of the point was that the days of fold-change limits being a legitimate analysis "device" in microarrary analysis are over.  Part of the reason is EXACTLY this point about small sample size and large fold changes being "better" is a fallacy.

[jackit]

Reality check:  early phase studies in medicine often cost a million dollars or more, take several years to conduct, and require very sick people to make potentially life-changing choices about treatment.

And by gawd if n=30, p = 0.07 and the odds ratio = 2 for an effect, I'm going say something like it was 'nearly significant.'  Burying a result, that potentially may help many patients, that is 93% likely to not be due to to chance, when the bootstrap uncertainty on the p-value itself is greater than 0.02, is not something I think is very helpful to either science or medicine. 

Ya'll can be as pedantic as you like.

[sciencephd]

The cost of the study is an interesting argument as to why the study sample size is small, and certainly is the reason why most microarrary studies are also severely underpowered.  However, I have never understood how the cost of the study or other such limitations should relate to how the data is analyzed and in particular what the statistical parameters are.

Nobody has suggested burying the results.  Adding a high level of emotional charge to an argument that is supposed to be about the scientific merits of a particular type of data analysis, particularly when "dying patients" are invoked, can lead to severe bias. 

Strangely enough, I am looking at a clinical dataset today with very weak statistics.

[jackit]

If cost (monetary and human) were no issue we would all happily have more impressive statistical thresholds.

[qrypt]

And if it became widespread practice to act on results that come with a p-value of .07, we would increase the ratio of certain kinds of errors from 1 in 20 to 1 in 14.  Okay, we'd also reduce the incidence of other kinds of errors.  At which point it becomes a matter of weighing all sorts of consequences in ways that can't be summarized in a general rule, not even in a single field e.g. medicine let alone across the board. 

The point is still that "nearing significance" is a phrase that carries implications it doesn't warrant.  As sciencephd says, this doesn't mean burying results.  It simply means reporting them in ways that specify what you actually mean, rather than using a confusing and potentially misleading phrase.