Methods Matter: Phone vs Web Surveys

[TimG]

TimG, I think, has done this in the past. He's just skeptical of social science, to my memory, especially when it disagrees with his gut feelings about how the world is. Again - this is just my memory of how he operates.

You misunderstand my point. The issue is I frequently notice people taking what is really just an opinion, dressing it up with some "study" and then claim it is proven fact. This happens most frequently in social sciences but happens in almost any field. In this case, cybercome simply repeats the **assumption** that in person interviews must be less accurate because that is what matches how he thinks the world works. If he just wants to say it is his opinion then that is the end of the discussion. The problem is he wants claim that his opinion should be treated as a known fact because some study produced data that does not directly contradict his opinion but offers no direct support for it either.

As for web polls: look at trolls in forums like this. How many trolls do you think actually believe their own posts and how many are exaggerating or lying just to generate a reaction? I personally think the same dynamic can bias the responses to anonymous web surveys and could lead to them being **less** accurate than the in person calls. That said, I don't know. I am just pointing out that multiple explanations for data exists and it quite myopic to insist that the only explanation is the one that conforms to cybercoma's personal prejudices.

Edited May 18, 2015 by TimG

[cybercoma]

As are many actual scientists.

Yeah. That's not insulting or anything to people who do social science. Just because you don't understand its reliability and validity doesn't mean that it's not a "real" science or that it doesn't have any sort of value.

[Smallc]

Yeah. That's not insulting or anything to people who do social science. Just because you don't understand its reliability and validity doesn't mean that it's not a "real" science or that it doesn't have any sort of value.

I've read a great many articles from scientists on why social science is bunk. I used to share your opinion, but I was persuaded to change my mind.

[cybercoma]

cybercome simply repeats the **assumption** that in person interviews must be less accurate because that is what matches how he thinks the world works. If he just wants to say it is his opinion then that is the end of the discussion. The problem is he wants claim that his opinion should be treated as a known fact because some study produced data that does not directly contradict his opinion but offers no direct support for it either.

Actually, it's not my opinion at all. It's the explanation given by Pew Research for the patterns in the statistics. I offered you an opportunity to give a better explanation that fits with the observations, but you can't. That's exactly how science works: observation and theory. You want to reject the explanation outright and provide nothing of substance in return. Radical skepticism is a pointless endeavour and I might add completely unreasonable.

More importantly, what you're denying is a social theory that people will generally censor themselves in the company of others, especially a stranger interviewer. I could provide you with additional sources that show this social fact to be true, but I doubt that would even matter because you would then just continue with your radical skepticism schtick.

Now here's the biggest problem with your routine here. You fail to extend that skepticism to your own assumptions and biases. What we have here in this study are not assumptions. What you are calling an assumption is an explanatory theory that describes how and why the patterns in the data occur. Just like any other science, there is an opportunity for a better explanation to arise. You haven't provided one. You haven't come even remotely close to providing one. In fact, I'm not entirely certain you even understand the current theory that does explain the data because you keep calling it "[my] opinion."

At this point, I'm not entirely sure why I keep responding to you. It's clear that your personal bias towards scholarly research is clouding your judgment. I could appeal to reason here until I'm blue in the face and you just won't see the point here because you're more interested in rejecting things outright. Don't get me wrong here. I'm all for someone rejecting Pew Research's theory, but in doing so they ought to provide a more compelling explanation for the patterns that appear in the data. Your theory is this (and correct me if I'm wrong), "people lie on web polls more than telephone polls because it's easier." That's not sufficient to explain the reason the disparity in responses follows a particular pattern and is most evident in questions involving socially contentious issues. Pew Research's explanation, invoking the notion of social desirability bias, does. And that's why I reject your explanation. Not because I disagree with you or because you simply have a different opinion. I reject your explanation because it doesn't do a better job of explaining the observed variability.

Now you reject Pew's explanation why? I don't understand. It's certainly not because you've found a better explanation. "It's easier to lie on web polls" is clearly not a better explanation to any reasonable person. In fact, it's demonstrably false as it's the exact opposite of what the data shows. The data shows that social interaction encourages people to lie due to social desirability bias. That means it's actually easier and more likely that people will lie in a face-to-face interview, which is the exact opposite of your argument. Were the biased responses all over the place and had no pattern indicating that people self-censor on topics that are socially contentious, then you might have a point that there's no way of knowing which survey method is biased or why. It's the patterns in the data though that are explained most clearly by the social desirability bias theory.

Anyway, I'm still amenable to alternate explanations for the observed data variability, should you come up with something better. Until then, I'm not going to engage with your silly little crusade against scholarly research that we've seen time and time again.

[cybercoma]

I've read a great many articles from scientists on why social science is bunk. I used to share your opinion, but I was persuaded to change my mind.

I've read a great many articles about why the "natural" sciences are bunk. *shrugs* What's your point? You think statistical analysis in economics is garbage? Economics is a social science. You think the predictive models that they build are completely useless? What's the end game here? Just reject observations and explanations about society and the behaviour of people entirely? There are predictive patterns that can be gleaned from social research. There's valuable information to be learned from these studies, which allow policies and other interventions to be put into place to help people.

I guess I can see rejecting that if you're a radical rightwing extremist who believes that it's a dog-eat-dog world and it should be everyone for themselves though. I suppose if you're a radical social Darwinist who believes people sow their own fates and that the poor and suffering should be culled for the strength of society, then you wouldn't want to be able to predict problems and help others.

But I'm sure you're none of those things, so why reject it outright? I have to wonder if you understood the exceptions that these "scientists" had to social science. Because believe me, there is bad social science, just as there's bad natural science. That doesn't mean throwing the baby out with the bathwater though.

And more importantly it's pretty much off topic. You're simply saying you reject Pew's study because it's social science. That's pretty much a conversation killer. So if you don't believe in social science, just move along.

[Smallc]

I've read a great many articles about why the "natural" sciences are bunk.

I doubt that.

[cybercoma]

I doubt that.

They study different things and have different methodologies and problems. You're taking a hockey player and saying they would completely suck at MMA. It's not enlightening and offers nothing to the discussion. The value of the social sciences is evident. If you want to argue about the validity of social science start another thread for it. This thread is about polling methods. Edited May 18, 2015 by cybercoma

[Smallc]

If you start a topic, you can't really control where that topic is going to go. The validity of social science is very much a consideration when discussing studies in the field.

[TimG]

More importantly, what you're denying is a social theory that people will generally censor themselves in the company of others, especially a stranger interviewer.

People will be more careful of what they say when they are in situations where they are not anonymous. I don't dispute this at all. This can show up as self censorship but it could show up as a tendency to exaggerate less and to be more truthful. It really depends on the person and the topic. That is why I think it is wrong to jump to the conclusion that web polls are necessarily more accurate simply because the results differ from the phone polls.

I could provide you with additional sources that show this social fact to be true, but I doubt that would even matter because you would then just continue with your radical skepticism schtick.

Why would it be relevant to my point? I am sure data says that people are more careful about what they say and that they seek to say things that they think the listeners want to hear. This data does not establish that the things said would be less truthful than the answers acquired via a web form because someone filling in an anonymous web form faces a different set of incentives that could result in answers that are just as inaccurate but inaccurate in different ways.

What you are calling an assumption is an explanatory theory that describes how and why the patterns in the data occur.

IOW, a hypothesis based on the preconceptions of the researchers. The only time something gets to be called a theory is when it is testable. In this case, we would need some way to determine the 'truth' in order to evaluate which form of survey is more accurate. But we can't know what the truth is so the researchers are reduced to making assumptions about the truth and constructing a narrative around their assumptions. That is why I say it is nothing but an assumption no matter how much they wish to dress it up. Edited May 18, 2015 by TimG

[Michael Hardner]

However, most people would fall asleep at any public discussion about methods.

Aha ! Close, but not actually true. Most members of "the" public, or perhaps "the" masses, would indeed fall asleep. But that's a problem with the public that has emerged from the design of this dialogue. I most certainly would NOT fall asleep, and would stand in line for a chance to have a meaningful and impactful discussion on these things.

I, in fact, am a member of several call lists and regularly answer telephone and web polls. The more I answer, the more they call me, the more other firms call me too. So my public is more influential than, say, yours already.

The idea of "the" public is going away as fast as mass TV audiences are, and the future will provide more input from members of "a" public, such as I am.

IMO, it's important to understand how the methods affect the responses.

Sure. I think the anonymous/nonymous aspects have been covered in the past, and Tim's suspicion of social science won't stop it from producing important results.

We can all help things along by encouraging uninformed people, and Fox News watchers, to vote with their remotes, ie. stay home and watch TV on voting day !

[Michael Hardner]

You misunderstand my point. The issue is I frequently notice people taking what is really just an opinion, dressing it up with some "study" and then claim it is proven fact.

For sure this happens a lot.

The problem is he wants claim that his opinion should be treated as a known fact because some study produced data that does not directly contradict his opinion but offers no direct support for it either.

Well, sure but it's probably true anyway...

That said, I don't know.

Yes, me neither really. It *sounds* believable to me. I don't think the answer helps the left, or the right wing though. It just helps anybody who uses the (potential) correct information properly.

it quite myopic to insist that the only explanation is the one that conforms to cybercoma's personal prejudices.

Or that the truth is incorrect because it conforms to Cyber's personal prejudices !

[Bonam]

I've read a great many articles about why the "natural" sciences are bunk. *shrugs* What's your point? You think statistical analysis in economics is garbage? Economics is a social science. You think the predictive models that they build are completely useless? What's the end game here? Just reject observations and explanations about society and the behaviour of people entirely? There are predictive patterns that can be gleaned from social research. There's valuable information to be learned from these studies, which allow policies and other interventions to be put into place to help people.

Economics? Yes, it's pretty much garbage. Take a room full of 100 economists, and you'll get 100 answers on where they think the markets, the economy, etc, are going.

[TimG]

Well, sure but it's probably true anyway...

And that would be an assumption. Assumptions are not necessarily false. They are simply not facts.

Yes, me neither really. It *sounds* believable to me. I don't think the answer helps the left, or the right wing though. It just helps anybody who uses the (potential) correct information properly.

This has nothing to do with the left-right politics. This is about evaluating information and understanding the difference between an assumption and a fact.

[Michael Hardner]

This has nothing to do with the left-right politics.

I agree with that. The part about face-to-face vs anonymous responses, though, I have heard before... a long time ago. It doesn't sound far-fetched at all to me.

[cybercoma]

Economics? Yes, it's pretty much garbage. Take a room full of 100 economists, and you'll get 100 answers on where they think the markets, the economy, etc, are going.

The markets and the economy are a fraction of what economists study.

[Bonam]

The markets and the economy are a fraction of what economists study.

True. Doesn't affect the point, though.

[cybercoma]

I agree with that. The part about face-to-face vs anonymous responses, though, I have heard before... a long time ago. It doesn't sound far-fetched at all to me.

It's not. This theory has been around for the better part of 3 and a half decades when it comes to data collection in the social sciences. What's new here is that Pew Research has quantified it and shown that new technologies, i.e., polls over the internet, have some value in mitigating this effect. Tim's trying to rewrite a generation worth of knowledge by calling these things assumptions and opinions.

Also, Tim, I don't claim there's only one explanation. I claimed the explanation given was the best one and have asked you numerous times here to provide a better one. What you're calling an assumption is an explanation of the observed patterns in the data. Give a better explanation. I'm completely open to that. What I'm not open to are people who don't have anything better to offer, but reject a perfectly sensible explanation outright–even when they admit that is not only sensible, but likely.

[bush_cheney2004]

Sensible explanations do not substitute for proof....but are great for assumptions and speculation.

[TimG]

I claimed the explanation given was the best one and have asked you numerous times here to provide a better one.

If you want to make a claim it is not enough to assume something that sounds good and claim it is the truth because no one has a better explanation. If you want to make a claim YOU must prove the claim and I have absolutely no obligation to provide a counter explanation. If I look at the the claim and find that it does not have any real supporting evidence and then I can say that it is simply an assumption whether I offer an alternative or not.

Your kind of thinking is a good example of lazy science that drives me nuts. It is basically science by story telling. i.e. invent a narrative that sounds good and claim it is a 'fact' because nothing contradicts that narrative. It is silly pseudo-science that is driven primarily by what the researchers want to believe rather than what the data says. Calling it nothing but an opinion is the most polite way to describe it.

Developing credible scientific explanations for the natural world requires testable hypotheses. i.e. it must be possible to consistently predict future outcomes by using the hypothesis. For example, if you jump off a building, I can use the theory of gravity to predict what your speed will be when you hit the ground. The ability to predict outcomes makes gravity a theory rather than an opinion.

Now it is true that every scientific theory starts out as a opinion/assumption/hypothesis. But that does not mean that every opinion/assumption/hypothesis should be treated like a theory because many opinions/assumptions turn out to be wrong. In this case the claim that web form forms are more truthful is an untestable assumption because we have no way to measure the "truth".

Edited May 19, 2015 by TimG

[Michael Hardner]

Sorry, correct me if I'm wrong but we're talking about proof of *why* this effect happens, correct ?

Isn't it understood that the effect is that people are more truthful when anonymous, but the *why* is not exactly understood ?

[TimG]

Isn't it understood that the effect is that people are more truthful when anonymous, but the *why* is not exactly understood?

Actually, the evidence is people are more careful about what they say when they are not anonymous and will say different things when anonymous. It is does NOT automatically follow that things said while anonymous are more truthful. To support the claim there needs to be evidence that things said when anonymous are actually more truthful. This has nothing to do with why. It is about finding evidence that actually supports the claim. So far all I have seen is the **assumption** that if people are careful about what they say when not anonymous then things said when anonymous must be more "truthful". There is no evidence to support this assumption even if the assumption is not unreasonable. Edited May 19, 2015 by TimG

[cybercoma]

Sensible explanations do not substitute for proof....but are great for assumptions and speculation.

The "proof" is in the data. The explanation is just that, an explanation of the proof.

[cybercoma]

Sorry, correct me if I'm wrong but we're talking about proof of *why* this effect happens, correct ?

Isn't it understood that the effect is that people are more truthful when anonymous, but the *why* is not exactly understood?

People aren't necessarily more truthful when they're anonymous. The study showed that it was particular kinds of questions that had a bigger effect. There was a greater difference between phone and web polls on questions that were socially contentious. Why that happens is wrapped up in the explanation that people will self-censor around others for either 1) what the respondent perceives as in the interests of the interviewer's own comfort or 2) what the respondent perceives to be a response that will paint them in a less negative light. It seems people don't generally consider how others would perceive them when they're taking an impersonal web poll. This effect isn't present for all questions, just the ones where the fundamentals of social interactionism take place, i.e., what the respondent thinks the interviewer would think of what the respondent thinks. It gets complex because it doesn't necessarily have anything to do with what the interviewer actually thinks. It's the social situation alone that alters the way the respondent answers the questions. The proof is in the data. We know this happens because we can observe it. We can explain why it happens with the social desirability bias theory because it encompasses all of the nuances of the variability in the data. And I will mention it again, if TimG can provide a better explanation as to why, then I'm all ears. I would love to hear it. But rejecting a solid theory that explains the data perfectly on the assumption that there could possibly be a better theory out there, but he just doesn't know it yet, is painfully stupid to be blunt.

[cybercoma]

Now it is true that every scientific theory starts out as a opinion/assumption/hypothesis. But that does not mean that every opinion/assumption/hypothesis should be treated like a theory because many opinions/assumptions turn out to be wrong. In this case the claim that web form forms are more truthful is an untestable assumption because we have no way to measure the "truth".

Where your argument leads is rejecting everything because something new might come up later, instead of accepting the best available explanation now until something better comes along. Again, your brand of radical skepticism that rejects all knowledge because what we know can change at some non-descript point in the future borders on absurdity, isn't at all helpful nor enlightening, and would set us back generations if it actually were the way science worked. In the natural sciences, you would simply reject evolution as "just an assumption," even though it explains what we observe in nature perfectly. That's what you're arguing here.

[cybercoma]

Actually, the evidence is people are more careful about what they say when they are not anonymous and will say different things when anonymous. It is does NOT automatically follow that things said while anonymous are more truthful. To support the claim there needs to be evidence that things said when anonymous are actually more truthful. This has nothing to do with why. It is about finding evidence that actually supports the claim. So far all I have seen is the **assumption** that if people are careful about what they say when not anonymous then things said when anonymous must be more "truthful". There is no evidence to support this assumption even if the assumption is not unreasonable.

What's unreasonable is to assume that people would automatically lie in surveys for no reason whatsoever. Of course the assumption is when a survey is given people are going to be truthful, otherwise we wouldn't be able to survey people at all. So now you're rejecting the entire idea that anyone ever gives a truthful response.

Again, if you want to flip the explanation and say that the more truthful response is the phone survey and the people are actually lying on the web survey, provide a coherent explanation as to why that would be, particularly given the kinds of responses where the difference are the greatest. You yourself even admit that people are not honest around others because they will self-censor. You've admitted that the theoretical explanation they gave is actually valid. Yet you continue to argue that they're wrong. Your argument here is completely incoherent. All you have to do to support it is provide an explanation as to why it would be the web survey that's less truthful on those questions which are most socially contentious. You haven't yet and you can't because it doesn't make sense that someone would tell the truth in a face-to-face interaction, yet try not to be so offensive in an impersonal poll. Who would they be trying not to offend when they're not in that face-to-face interation? The idea of someone reading this later? They know they're anonymous. They're not looking someone in the eye and giving them an answer.

The explanation given for the patterns in the data is not even remotely contentious. You're literally making an issue out of absolutely nothing because you are radically skeptical about any and all scholarly research. Your thinking leads to rejecting everything because you don't know if something new and better will come along. Like I said, it's like rejecting evolution because you think a better explanation might come along. It's like rejecting gravity because you think a better explanation might come along. You're more than welcome to reject those things and in fact it would be groundbreaking work if you did. The problem is you have to provide a better explanation in the process. You can't and certainly haven't here.