« 'Tis the Season | Main | RIP Louis Auchincloss »
Wednesday, January 27, 2010
"Cognitive infiltration"
A few days ago, over at Salon.com, Glenn Greenwald put up this post, in which he described and criticized a 2008 paper (by Cass Sunstein (now, as Greenwald notes, the head of the President's Office of Information and Regulatory Affairs). Here is the SSRN abstract of the paper:
Many millions of people hold conspiracy theories; they believe that powerful people have worked together in order to withhold the truth about some important practice or some terrible event. A recent example is the belief, widespread in some parts of the world, that the attacks of 9/11 were carried out not by Al Qaeda, but by Israel or the United States. Those who subscribe to conspiracy theories may create serious risks, including risks of violence, and the existence of such theories raises significant challenges for policy and law. The first challenge is to understand the mechanisms by which conspiracy theories prosper; the second challenge is to understand how such theories might be undermined. Such theories typically spread as a result of identifiable cognitive blunders, operating in conjunction with informational and reputational influences. A distinctive feature of conspiracy theories is their self-sealing quality. Conspiracy theorists are not likely to be persuaded by an attempt to dispel their theories; they may even characterize that very attempt as further proof of the conspiracy. Because those who hold conspiracy theories typically suffer from a crippled epistemology, in accordance with which it is rational to hold such theories, the best response consists in cognitive infiltration of extremist groups. Various policy dilemmas, such as the question whether it is better for government to rebut conspiracy theories or to ignore them, are explored in this light.
Let's put aside (what seems to me to be) the fact that calls by a prominent Bush Administration official -- especially one with the "Information and Regulatory Affairs" portfolio -- to do what Sunstein urged (before he was in the Administration) in the paper be done would be met with howls of protest. Greenwald concludes:
It is [the] history of government deceit and wrongdoing that renders Sunstein's desire to use covert propaganda to "undermine" anti-government speech so repugnant. The reason conspiracy theories resonate so much is precisely that people have learned -- rationally -- to distrust government actions and statements. Sunstein's proposed covert propaganda scheme is a perfect illustration of why that is.
My first (and probably second and third) reaction to Sunstein's paper is like Greenwald's. That is, the idea of the government intentionally employing covert agents to infiltrate suspect web sites, online conversations, etc., and pose as independent experts in order to bend these sites and conversations more toward the government's liking -- even assuming (as Sunstein does) a "well-motivated government that aims to eliminate conspiracy theories, or draw their poison, if and only if social welfare is improved by doing so" -- seems deeply troubling.
And yet . . . it surely is in the government's -- or, the political community's -- interest that "conspiracy theories" be engaged and refuted and often government officials -- again, we assume "well-motivated ones" -- are often going to be in a good position to have "the truth of the matter." As I discussed in more detail in this paper, the notion that sometimes finds its way into free-speech manifestos and First Amendment opinions -- i.e., that it is none of the government's business what people think -- seems clearly misguided. Ideas matter. So, why is it that Sunstein's "infiltration" proposal is unsettling, even frightening? Is it because the enterprise itself is illegitimate, or because we worry that the risk of abuse is too great? If the latter, can the risks be reduced (in ways that do not require the assumption of "well-motivated government"?) Any thoughts?
Posted by Rick Garnett on January 27, 2010 at 10:32 AM | Permalink
TrackBack
TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8341c6a7953ef0128771abb4d970c
Listed below are links to weblogs that reference "Cognitive infiltration":
Comments
Sunstein's proposal is a rather striking rejection of the soft power benefits of simple good government openness.
For example, post-George W. Bush revelations have shown that while the torture methods used to interrogate terrorism suspects did not meet the something short of torture "smell test" and were not as effective as claimed, the size of the program was smaller than either its supporters or opponents had suspected. More openness on the scale of this program probably would have undermined opposition to it.
Similarly, a greater willingness to reveal instances of stupidity by national security actors would reduce the likelihood that mainstream observers would attribute malice to them.
Nutty conspiracy theorists are an overrated threat to public policy. The threat arises only when more mainstream people give credence to them. But, these people do not have a "crippled epistemology" to the same degree. If there are credible people providing real facts, normal people won't take the bait.
Posted by: ohwilleke | Feb 1, 2010 3:31:05 PM
Sunstein's proposal is both rational and creepy. Were it implemented and then discovered, the creepiness would be more salient, if it isn't already. There also used to be some notion that in a democracy even the best ends do not justify the use of any means whatsoever to achieve them.
Posted by: A.J. Sutter | Jan 28, 2010 7:08:46 AM
"The reason conspiracy theories resonate so much is precisely that people have learned -- rationally -- to distrust government actions and statements."
Mark's point gets it just right. A posture of distrust of the government is only rational if Greenwald's empirical assertion about the government's inherent badness is accurate.
What puzzles me about this is that government seems more rightly regarded as just a neutral and inevitable means of administering public policy (and by "neutral" I mean without necessary political valence). Like any human, it's fallible, but also like any human, capable of great good too. Some government action is very popular, like creating a strong military or building interstate highways. Those who assert that government is inherently bad never seem to incorporate these benign or right-leaning state activities into their worldview.
Why is this? Perhaps the answer lies selection bias. If you are attracted to or convinced of the idea that government is inherently bad, then you'll likely discount evidence that suggest it isn't, and identify as typical of government only that state conduct you dislike. Hence Bush II loyalists were a lot less vociferous about the ills of big government from 2001-08, for understandable reasons.
This doesn't mean that there isn't something surprising and possibly even disturbing about Sunstein's idea. All of "On Rumors" (and engaging and frequently unsettling little book) makes more or less this point: that public dialogue is so fraught with various biases that it may not make things better, but worse. This is a foundationally concerning idea for those of us who developed a belief in Rawlsian public reason and the marketplace of ideas theory of the First Amendment, and weirder still, there's some decent evidence that it may be right (though this doesn't necessarily warrant policies that squelch public dialogue).
But Sunstein's idea is also inherently not partisan. The left as well as the right is susceptible to the kinds of biases that can cause public dialogue to mislead rather than edify, and Sunstein makes this point repeatedly in his work.
Posted by: Dave | Jan 28, 2010 2:24:45 AM
Thanks, Mark. As you say, "it could be that the risk that the government will not be well-motivated is high enough that on rule-utilitarian-like grounds we ought to preclude the practice entirely[,]" and (I thought that) the point of my own post, as opposed to Greenwald's, was to ask whether -- given that the political community does have an "interest" in reducing the stock and purchase of "conspiracy theory" ideas -- there are ways to reduce the risk.
On your second point -- I don't think my post came close to a "howl." At least, I hope it didn't!
Posted by: Rick Garnett | Jan 27, 2010 1:38:02 PM
All of politics and ideology is some form of conspiracy theory. Traditional populist assaults on a Yankee clique of rich Boston Bankers oppressing ordinary folk, the pre Citizens United view of giant corporations corrupting politicians with truckloads of money, the idea of Eastern effete limousine liberals. There are dozens of cliched conspiracies that persist over time, although certain elements vary with culture. The conventional leftist Marxism (history as a clash of economic classes driven by dialectical materialism) is out of favor these days and Protocols of the Elders of Zion is down some but certainly not out.
Once an ideological preconception is established, then every new event is filtered through that world view. You see what you expect to see because everything will be reformulated to confirm what you already believe. If you imagine an all powerful Mossad then the Jews are responsible for 9/11 and everything else. Of course, if you believe in little green men in flying saucers from the planet Zoron, then they are probably at fault.
However, the extreme cognitive dysfunction that we can easily see in others is simply an extended version of the normal day to day cognitive distortion that we live by. The same predisposition that drives conservatives, liberals, Republicans, Democrats, Sarah Palin, and Nancy Pelosi. That is why the concept of any government intervention is scary. Representative government is supposed to be a relection of whatever conspiracy theory works best every two years in attack ads during the election campaign. If government begins to systematically influence these delusions where they are deemed to be really bad, then it will slowly deal just with the bad ones. Unfortunately, yours are bad and mine are good.
Posted by: Howard GIlbert | Jan 27, 2010 1:20:48 PM
Isn't the "problem" that people aren't willing to keep at the front of their minds the assumption (or condition) that the government doing this is well-motivated? The suspicions, history, etc., that people have and invoke arise because they are unwilling to assume that the government is well-motivated. But, IF the government is well-motivated and, as Sunstein and Vermeule assert, those who hold mistaken conspiracy theories discount information and arguments they know come from government officials, what's the problem with cognitive infiltration? (This is different from agents provocateurs in a number of ways, but the fact that the engagement occurs only in cyberspace and not in physical space is, I think, important.) Of course, it could be that the risk that the government will not be well-motivated is high enough that on rule-utilitarian-like grounds we ought to preclude the practice entirely, but that's outside the parameters of the Sunstein-Vermeule paper.
And, incidentally, exactly why do you seem to say that the paper is not being "met with howls of protest"? See, e.g., your post itself.
Posted by: Mark Tushnet | Jan 27, 2010 11:23:24 AM
The comments to this entry are closed.