Methods Matter: Phone vs Web Surveys

[Michael Hardner]

Ah, ok. I stand corrected by TimG and Cyber both.

If I read it correctly, though, there is an *effect* that is understood but the reasons behind it are not. If that's the case, then we're discussing more of the mysteries of human behavior, which aren't really solvable. For the purposes of accurately answering questions like "what do people think ?", the effect is understood enough for good polling to be executed.

[TimG]

Where your argument leads is rejecting everything because something new might come up later, instead of accepting the best available explanation now until something better comes along.

Who says I reject it? All I said is it is nothing but an assumption that might be true or it might not. You are the one insisting that this assumption be treated as a fact because "no one has any better idea".

Again, your brand of radical skepticism that rejects all knowledge because what we know can change at some non-descript point in the future borders on absurdity, isn't at all helpful nor enlightening

I see it as common sense. It is called 'understanding the limits of knowledge' and it is very important to advancing knowledge.

You also keep seem to missing my point. I am not saying that the assumption that web surveys are more 'truthful' is necessarily wrong. I am only saying that you should acknowledge that it is an assumption and that there is no direct evidence supporting the claim. Why is that so difficult? You can say that it is best assumption given the information available if you like. The argument only comes up because you insist that this assumption be treated as a fact when it is not.

you would simply reject evolution as "just an assumption," even though it explains what we observe in nature perfectly. That's what you're arguing here.

Nope. Evolution is a theory that has predicted outcomes. For example, evolutionary theory says populations will develop a tolerance for poisons over time and that is what we see with various organisms that are exposed to poisons. Obviously it is a broad topic and narratives and assumptions are part of it but because of the ability to predict outcomes we can have more confidence than we would have with other ideas. Edited May 19, 2015 by TimG

[cybercoma]

Who says I reject it? All I said is it is nothing but an assumption that might be true or it might not. You are the one insisting that this assumption be treated as a fact because "no one has any better idea".

Everything is an assumption that might or might not be true according to your logic.

I see it as common sense. It is called 'understanding the limits of knowledge' and it is very important to advancing knowledge.

Except your limits involve rejecting everything on the grounds that our understanding may change in the future.

You also keep seem to missing my point. I am not saying that the assumption that web surveys are more 'truthful' is necessarily wrong. I am only saying that you should acknowledge that it is an assumption and that there is no direct evidence supporting the claim. Why is that so difficult? You can say that it is best assumption given the information available if you like. The argument only comes up because you insist that this assumption be treated as a fact when it is not.

It's so difficult because there is direct evidence supporting the claim. There's statistically valid research attached here. The theory is simply explaining the proof.

Nope. Evolution is a theory that has predicted outcomes. For example, evolutionary theory says populations will develop a tolerance for poisons over time and that is what we see with various organisms that are exposed to poisons. Obviously it is a broad topic and narratives and assumptions are part of it but because of the ability to predict outcomes we can have more confidence than we would have with other ideas.

It's merely an example of the absurdity of your radical skepticism position. People should be skeptical, but radical skepticism rejects all knowledge. Using your language for the OP study, evolution is an assumption. It's an assumption that's demonstrated and supported by observations, but that doesn't matter. You want someone to acknowledge it as an assumption. That's exactly what's going on in the OP. Social desirability bias is an assumption, like evolution, that's demonstrated and supported by observations (the empirical data in this study). You push for this to be called an assumption because it's a dysphemism for the strength of the theory here. Why you're doing that is beyond me, but again I have to assume from your past behaviour and arguments that you're simply pushing your radical skepticism ideology again. Edited May 19, 2015 by cybercoma

[bush_cheney2004]

A practical lesson in faked interwebs survey data:

http://www.cbc.ca/radio/asithappens/as-it-happens-tuesday-edition-1.3079544/researcher-retracts-landmark-same-sex-marriage-study-claims-co-author-fabricated-data-1.3080637

[Michael Hardner]

A practical lesson in faked interwebs survey data:

http://www.cbc.ca/radio/asithappens/as-it-happens-tuesday-edition-1.3079544/researcher-retracts-landmark-same-sex-marriage-study-claims-co-author-fabricated-data-1.3080637

Ok, but that's just out-and-out fraud in academia - could have happened with any medium. Poor Michael J. LeCour, looks like peer review worked once again, and now his career is done. Too bad he didn't pick a career with a low moral stature, like international banking.

[bush_cheney2004]

Ok, but that's just out-and-out fraud in academia - could have happened with any medium.

True...but it did happen with internet surveys...compared to shoe leather.

[Michael Hardner]

True...but it did happen with internet surveys...compared to shoe leather.

Except that it didn't. It was just Mike LeCour SAYING he had done a survey. He could have said he interviewed everybody one-on-one too. But ok.

[bush_cheney2004]

Except that it didn't. It was just Mike LeCour SAYING he had done a survey. He could have said he interviewed everybody one-on-one too. But ok.

No he couldn't...not without faking supporting evidence and people as well. It is much easier with the anonymous internet.

[dre]

If you want to make a claim it is not enough to assume something that sounds good and claim it is the truth because no one has a better explanation. If you want to make a claim YOU must prove the claim and I have absolutely no obligation to provide a counter explanation. If I look at the the claim and find that it does not have any real supporting evidence and then I can say that it is simply an assumption whether I offer an alternative or not.

Nobody said you had an obligation. But if you wanted to be part of a conversation you could provide one... instead of pages of mindless babble thats not even germaine to the OP.

[Michael Hardner]

It is much easier with the anonymous internet.

I suppose so. I do want to point out, though, that well designed internet polls are not just like the multi-choice polls you put on MLW. I was just paid $10 for a poll that only took about 15 mins !

[TimG]

Nobody said you had an obligation. But if you wanted to be part of a conversation you could provide one.

I did provide one: the common knowledge that people lie and or misrepresent themselves when they believe they are anonymous. but cybercoma simply declared my alternative "unacceptable". No matter how many times cyper repeats the claim, the study has no basis for the claim that "online surveys" are more "truthful". It only has the assumption that since people can self censor that anonymous surveys must be more accurate because they are not "self-censored". A premise that ignores the possibility that people can have other reasons to lie or misrepresent facts on web surveys which cancel out any increased accuracy from reduced "self censorship". Edited May 22, 2015 by TimG

[cybercoma]

I did provide one: the common knowledge that people lie and or misrepresent themselves when they believe they are anonymous. but cybercoma simply declared my alternative "unacceptable".

There's absolutely no reason to believe people would lie more in an online form than in a personal interview and it's certainly not as thorough or well established as the alternative that was offered by Pew. That's why I reject your explanation. It's just not very good. You're arguing the exact opposite, that people are more likely to lie when anonymous, and it simply doesn't make any sense when people have far more incentive to lie to another person due to socionormative pressure. You yourself said that people self-censor around others. So you even agree with the explanation that Pew gave, but you oddly want me to give equal consideration to the exact opposite when it doesn't even come remotely close to making as much sense.

Edited May 22, 2015 by cybercoma

[cybercoma]

And I should add, more importantly, it doesn't fit with the kind of questions that people "lie" most on. The biggest disparities are amongst those responses that are socially contentious. Why would people lie to give MORE socially contentious answers in an anonymous form, when it makes far more sense that they would lie to an interviewer to hide their controversial opinions?

[TimG]

You're arguing the exact opposite, that people are more likely to lie when anonymous, and it simply doesn't make any sense when people have far more incentive to lie to another person due to socionormative pressure.

Memory is a funny thing. Let's take the question "have you experienced racism?". An interview with a person the social pressure could force the person to assess their memory carefully and provide an answer that closely reflects reality. In an anonymous survey that same person could be inclined to exaggerate because they have no incentive assess their memory carefully before answering. IOW, the person is not necessarily lying or even aware that they are exaggerating but the web form could be less accurate because of the way human memory recall works. Edited May 22, 2015 by TimG

[cybercoma]

You have a lot of personal experience with racism to make the argument that it's difficult to recall? Because frankly, I think anyone who has faced racism probably remembers it very clearly.

[cybercoma]

Let's put this another way, Tim. Do you think the KKK hides behind white hoods so they can lie about their racism?

[TimG]

You have a lot of personal experience with racism to make the argument that it's difficult to recall? Because frankly, I think anyone who has faced racism probably remembers it very clearly.

The original question (which i did not reference correctly in my post above) asked people to quantify the amount of racism they encountered. An frankly, yes, people are very bad are quantifying such things. A person who is overly sensitive to the issue will exaggerate the frequency of such events while a person who does not care much about it may under estimate them. In such a situation is possible, if not plausible, that more exaggeration would occur when someone is anonymous than when someone is talking to someone. IOW, self censorship can lead to more accuracy - not less as your OP assumes.

[cybercoma]

A person who is overly sensitive to the issue will exaggerate the frequency of such events

A person who is "overly" [sic] sensitive to the issue will "exaggerate" [sic] regardless of the format.

[TimG]

A person who is "overly" [sic] sensitive to the issue will "exaggerate" [sic] regardless of the format.

The issue at question is not whether they exaggerate but whether being anonymous leads to more exaggeration. Given how common it is for people to misrepresent themselves when online and anonymous it is not unreasonable to suggest that the differences can be explained by greater exaggeration when anonymous. This, of course, is the exact opposite of the assumptions made by the Pew study which is why I pointed out that the Pew study is simply assuming a conclusion that does not automatically follow from the data they collected. Edited May 22, 2015 by TimG

[cybercoma]

It is unreasonable to assume that. That's what I've been pointing out for pages and pages. People misrepresent themselves to others in online forums and by trolling. That's not the same thing as being part of a Pew Research panel. Again, since I have to repeat myself over and over to you, the only way your argument makes sense is if you believe that all surveys are useless because people will commonly lie on them. If that's your contention, then provide some kind of research to support that argument because I'm simply not buying it. People are generally truthful in surveys. In situations where they're not, such as when it asks intimate details about their life or about illegal activities, there are methods to determine the truthfulness of their responses.

Anyway, it seems the problem is you just don't understand the background of the problem. You're rejecting something you've read nothing about. I would suggest reading up on social desirability bias. You can start with the link below for more information, if you're so inclined. Bear in mind that SDR isn't just something new that Pew made up off the top of their heads. It's something that has been observed and written about since the early 1960s. We're talking about a theory that was established over 50 years ago and you're just dismissing it out of hand because "well people lie online too!" You're missing the point and completely misunderstanding the theory and why the data supports it.

http://neuron4.psych.ubc.ca/~dpaulhus/research/SDR/downloads/CHAPTERS/ETS%20chapter.pdf

Edited May 22, 2015 by cybercoma

[TimG]

That's not the same thing as being part of a Pew Research panel.

That is actually worse because no one volunteers to be a member of such as panel unless they have opinions they want to share. It is incredibly naive to assume that the desire to expression opinions will not lead to exaggeration and misrepresentation on anonymous forms - a tendency which the self-censorship phenomena could counter act when interviews are done in person.

[dre]

I found this online with some links... Most of them seem to suggest theres very little difference if any.

A majority of the literature on the net seems to support that non-anonymous and anonymous-surveys give very similar results.

The paper "A comparison of confidential versus anonymous survey procedures: Effects on reporting of drug use and related attitudes and beliefs in a national study of students." studies anonymous vs. confidential surveys in a specific case (drug usage reported in 8th and 10th graders), and concluded that there was a very modest or possibly no effect on the data.

From the abstract:

The same conclusion is drawn in more countries in the paper "Adolescent Substance Abuse in Mexico, Puerto Rico and the United States: Effect of Anonymous versus Confidential Survey Formats".

Similarly another study "The Effect of Anonymous Vs. Nonanonymous Rating Conditions on Patient Satisfaction and Motivation Ratings in a Population of Substance Abuse Patients" concluded that

Anonymity had either no effect on ratings or accounted for <1% of the variance.

In a study about collecting data regarding bullying, "The Efficacy of Non-Anonymous Measures of Bullying" concluded that

The findings supported the hypotheses that the respondents did not differ in their report of the incidence of either bullying or victimization, regardless of whether they were required to identify themselves by writing down their names on the questionnaire forms.

Another study, "Differences between ‘talking about' and ‘admitting' sensitive behaviour in anonymous and non-anonymous web-based interviews" which (among other things) compared difference between two questionnaire modes- anonymous web-based forms or dektop-based video interviews concluded that:

Nevertheless, the expected differences between the interview modes were not observed.

On the other hand:

A study called "High risk behaviour and fertility desires among heterosexual HIV-positive patients with a serodiscordant partner – two challenging issues"(PDF) says that

Non-anonymous data collection on condom use may underestimate high risk behaviour

Note that the last study was the only one that found the non-anonymous data was not as good. And it kind of makes sense if you think about it because the nature of that subject would involve more social pressure.

Edited May 22, 2015 by dre

[bush_cheney2004]

Ok, but that's just out-and-out fraud in academia - could have happened with any medium. Poor Michael J. LeCour, looks like peer review worked once again, and now his career is done. Too bad he didn't pick a career with a low moral stature, like international banking.

....or "climate change scientist". These guys must have learned from them:

http://www.cbc.ca/news/technology/science-magazine-retracts-study-on-voters-gay-rights-views-1.3091807

[cybercoma]

Take your off topic routine somewhere else.

[cybercoma]

I was doing some reading and came across this statement on Ipsos' online panelist recruitment.

Respondent Integrity

Online panel members are verified via a double opt in process, and stringent quality checks are performed regularly. Online panelists with contradictory answers, under/over clicking, straight-lining responses, or speeding through surveys consistently are removed from the panel. Technical quality checks identify duplicate IP addresses and automated registrations.

This speaks directly to TimG's claims that people in online surveys lie or otherwise don't take the time to respond appropriately. There are built in methods for ensuring respondent integrity. Those who fail are routinely booted from the panel.