The new Kahneman et al book titled NOISE has generated considerable, well, noise, as evidenced by the Amazon reviews so far which are all over the place. Also, Andrew Gelman on his Statistical Modeling, Causal Inference, & Social Science blog has taken an observation by Rachel Meagher to question the authors’ statistical chops. His comment prompted many comments (and counting) even by the standard of his blog. On Facebook, in the Behavioral Economist group, a contributor pointed out the questionable evidence that Kahneman paraded in his previous best-seller Thinking Fast and Slow (TFS) as well as Kahneman’s failure to correct the record and either retract the book or revise it, and wondered why she should waste her time on reading his new book … . Fair question that, no?
To answer my own question, yes, I think the contributor’s question is fair. As pointed out by Schimmack and colleagues, in their deconstruction of the (in)famous chapter on priming in TFS, Tversky and Kahneman (1971) wrote “we refuse to believe that a serious investigator will knowingly accept a .50 risk of failing to confirm a valid research hypothesis.” Yet, here we are (and we are not even talking about the equally questionable track record of Sunstein.)
As Kristal and her colleagues have noted recently, it is scientists’ responsibility to fess up when they are wrong and know it. Tall poppies such as Kahneman and Sunstein have a particular responsibility of getting the science right. Also, their output deserves increased scrutiny last but not least because they are so influential.
Contrary to the claim made repeatedly in the book, none of the material here is really all that new. As so often (e.g., in his decades of ongoing battle with Gigerenzer or here in places such as the section on simple rules starting on p. 127), Kahneman fails to give credit where surely he should. Brighton & Gigerenzer (2015), for example, have written about the Overall Error equation and its NoiseSquared summand (and provided many references to related work in marketing and finance). It is hard to believe that Kahneman has not come across this work, as “a few minutes of research would reveal [it]” (p. 167). So what to make of this omission? There are two possible explanations: Kahneman either did not know, or he did and chose to ignore this previous work. Both explanations are disappointing and demonstrate intellectual dishonesty, right up there with the failure to correct the record on which TFS is based when he had about a decade to do so. This kind of behavior injects unnecessary noise into the presumably common project science and undermines science’s self-correction mechanism. So much for noise audits and decision hygiene. (More about them later.)