next »

[theblondeassassin]

Well, men might be underreporting their income, but aren't all "studies" possibly invalid in some way, because some people are just liars?

I know nothing about statistics (really, I don't! I don't even get on the fora and claim to know anything!)--

BUT is there some term or concept for the recognition that some respondents will lie, forget, or screw up. Is that what "margin of error" is supposed to mean?

The Fiona

I once found that an item had been accidently repeated on a survey with a large number of questions (~100). That is, the exact same wording, to the exact same respondent, with the exact same response scale, on the same questionnaire.

My analysis of the responses by the same respondents, about 20 questions apart, suggests that the responses to any individual item (or even scales) should be taken with a BIG grain of salt, even if phenomena such as the "helpful respondent" or "socially desirable" responding are not present.

More generally, extensive work on instrument development, along with triangulation/multi-method approaches, is generally warranted to have confidence in self-report results.

[busyslinky]

Well, men might be underreporting their income, but aren't all "studies" possibly invalid in some way, because some people are just liars?

I know nothing about statistics (really, I don't! I don't even get on the fora and claim to know anything!)--

BUT is there some term or concept for the recognition that some respondents will lie, forget, or screw up. Is that what "margin of error" is supposed to mean?

The Fiona

I once found that an item had been accidently repeated on a survey with a large number of questions (~100). That is, the exact same wording, to the exact same respondent, with the exact same response scale, on the same questionnaire.

My analysis of the responses by the same respondents, about 20 questions apart, suggests that the responses to any individual item (or even scales) should be taken with a BIG grain of salt, even if phenomena such as the "helpful respondent" or "socially desirable" responding are not present.

More generally, extensive work on instrument development, along with triangulation/multi-method approaches, is generally warranted to have confidence in self-report results.

I'm not espousing to be an expert on questionnaires or any such thing or even think I am worthy of giving an opinion on these items.  But, I would say I agree with you.  I am not sure if I am understanding your first point.  But the same question for the same survey for the exact same response scale, may have been part of a careful and extensive development of a survey.  It may actually be a "marker" question to determine the reliability of the responses.  If they respond differently to the same question then a reliability issue arises.  I think reliability and validity are the terms.  But don't hold me to that.

[theblondeassassin]

Well, men might be underreporting their income, but aren't all "studies" possibly invalid in some way, because some people are just liars?

I know nothing about statistics (really, I don't! I don't even get on the fora and claim to know anything!)--

BUT is there some term or concept for the recognition that some respondents will lie, forget, or screw up. Is that what "margin of error" is supposed to mean?

The Fiona

I once found that an item had been accidently repeated on a survey with a large number of questions (~100). That is, the exact same wording, to the exact same respondent, with the exact same response scale, on the same questionnaire.

My analysis of the responses by the same respondents, about 20 questions apart, suggests that the responses to any individual item (or even scales) should be taken with a BIG grain of salt, even if phenomena such as the "helpful respondent" or "socially desirable" responding are not present.

More generally, extensive work on instrument development, along with triangulation/multi-method approaches, is generally warranted to have confidence in self-report results.

I'm not espousing to be an expert on questionnaires or any such thing or even think I am worthy of giving an opinion on these items.  But, I would say I agree with you.  I am not sure if I am understanding your first point.  But the same question for the same survey for the exact same response scale, may have been part of a careful and extensive development of a survey.  It may actually be a "marker" question to determine the reliability of the responses.  If they respond differently to the same question then a reliability issue arises.  I think reliability and validity are the terms.  But don't hold me to that.

It might have been intentional, but it wasn't.

I was part of the research team that administered the questionnaires and analysed the responses.

(The total number of questions being asked was in the thousands and the number of respondents per response unit was in the dozens.)