yesterday afternoon, a colleague sent me an article from the daily mail and asked me if it could possibly be true. The article in question is an expose on snopes.com, the fact-checking site used by journalists and citizens around the world and one of the sites Facebook recently partnered with to fact-check news on its platform. the daily mail article makes a number of claims about the site’s principles and organization, drawing heavily on the proceedings of a contentious divorce between the site’s founders and questioning whether the site could act as a trusted, neutral arbiter of the truth”. /p>
When I first read the daily mail article, I immediately suspected that the story itself must be “fake news” due to how devastating the claims were and since snopes.com was heavily used by the journalistic community, if any. If the claims were true, someone would have already written about them and companies like Facebook wouldn’t be associated with them. I also noticed that despite being online for several hours, no other major media outlets had written about the story, which is usually a strong sign of a false or misleading story. Yet at the same time, the daily mail seemed to base its claims on a series of emails and other documents from a court case, some of which it reproduced in its article, and, perhaps most bizarrely, neither the snopes nor its principles had issued none. type of statement through their website or social media channels denying the story.
On the surface, this seemed like a classic case of fake news: an outrageous and highly shareable story, incorporating sources and materials that seemed official, but no other mainstream media even mentioned the story. I myself told my colleague that I just didn’t know what to think. Was this a complete fabrication by a disgruntled snopes target, or was this really an explosive exposure that pulled back the curtain on one of the world’s most respected and famous fact-checking marks?
In fact, one of my first thoughts after reading the article is that this is precisely how the “fake news” community would fight fact-checking: by running a trickle of explosive false or misleading stories to discredit and cast doubt. about fact checkers.
In the world of counterintelligence, this is what is known as a “mirror desert”: it creates a chaotic information environment that blends truth, half-truth, and fiction so seamlessly that even the best can no longer they can tell what is real and what is not.
so, when I contacted david mikkelson, the founder of snopes, for comment, I expected him to respond with a lengthy email in snopes’ trademark point-by-point format, completely refuting each and every one of the claims in the daily mail article and dismiss the entire article as “fake news”.
so it was an incredible surprise to get david’s one-sentence answer that read in its entirety “I’d love to talk to you, but I can only cover a few things in general because the terms exclude me.” of a binding settlement agreement to discuss the details of my divorce.”
this totally blew me away. Here was one of the world’s most respected fact-checking organizations, soon to become the final arbiter of “truth” on Facebook, saying it can’t respond to a fact-checking request due to a non-disclosure agreement.
In short, when someone tried to verify the fact with the fact checker, the response was the equivalent of “it’s secret”.
It is impossible to underestimate how antithetical this is to the world of fact-checking, in which absolute openness and transparency are necessary prerequisites for trust. How can fact-checking organizations like snopes expect the public to trust them if when questioned, their response is that they can’t answer?
When I presented David with a set of post-clarification questions, he provided answers to some and not others. Of particular interest, when pressed about the Daily Mail’s claims that at least one Snopes employee has run for political office and that this presents at least the appearance of potential bias in Snopes’ fact checks, David responded: “It’s pretty much a given that anyone who has ever run for (or held) political office did so under some kind of party affiliation and said something critical about their opponents and/or other politicians at some point. Does that mean that anyone who has ever run for office is manifestly unsuitable to be associated with a fact-checking effort, in any capacity?”
Actually, that’s a fascinating response from a fact-checking organization that prides itself on its supposed neutrality. Think of it this way: what if there was a fact-checking organization whose fact-checkers all came from the ranks of Breitbart and Infowars? most liberals would probably dismiss such an organization as partisan and biased. Likewise, an organization whose fact-checkers all hailed from Occupy Democrats and Huffington Post might be dismissed by conservatives as partisan and biased. in fact, when I asked several colleagues their opinion on this matter this morning, the unanimous response was that self-declared strongly politically inclined people on either side should not be part of a fact-checking organization and they had all incorrectly assumed that snopes would have felt the same way and would have a general policy against placing partisan individuals as fact checkers.
Indeed, this is one of the reasons fact-checking organizations need to be transparent and open. if an organization like snopes feels it is okay to hire partisan employees who have run for public office on behalf of a particular political party and employ them as fact checkers where they are likely to be asked to weigh in on aligned or contrary material depending on their views, how can they reasonably be expected to act as neutral arbiters of truth?
Put another way, some Republicans firmly believe that climate change is a falsehood and that humans are in no way responsible for climate change. Those in the scientific community might object to an anti-climate change Republican serving as a fact checker for climate change stories in snopes and marking every article about a new scientific study on climate change as fake news. however, we have no way of knowing the biases of fact-checkers on snopes: we simply have to trust that the site’s views on what constitutes neutrality are the same as our own.
When I asked for a comment on the specific and detailed criteria that snopes uses to screen its candidates and decide who to hire as a fact checker, surprisingly, david demurred, saying only that the site looks for candidates in all fields and skills. Specifically, it did not provide any details whatsoever regarding the selection process and how Snopes evaluates potential hires. David also did not respond to further emails asking if, as part of the selection process, snopes has applicants check a set of articles to assess their reasoning and research skills and to gain insight into their thought process.
This was highly unexpected, as I assumed that a reputable fact-checking site like snopes would have a detailed written formal screening process for new fact-checkers that would include them performing a set of fact-checks and would include a set of extensive set of interview questions designed to test your ability to identify potential or perceived conflicts of interest and work through potential biases.
Even stranger, despite asking in two separate emails how snopes assesses its fact checkers and if it performs inter-rater and intra-rater reliability assessments, david replied only that fact-checkers work collaboratively together and did not reply to further requests for more detail and did not respond to whether snopes uses any sort of evaluation score or ongoing testing process to evaluate its fact checkers.
this raises exceptionally serious concerns about the inner workings of snopes and why it is not more forthcoming about its evaluation process. To argue that because multiple fact checkers may work on an article that reliability is not a concern is a false argument that shows a worrying lack of understanding about reliability and accuracy. imagine a team of 50 staunch climate deniers working collaboratively to debunk a new scientific study that shows a clear link between industrial pollution and climate change. the sheer size of the team does not make up for the lack of diversity of opinion. however, david did not provide any comment on how snopes does or does not explicitly force diversity of opinion in their ad-hoc fact-checking teams.
A robust human rating workflow should regularly assess the accuracy and reproducibility of the scores generated by your human raters, even when working collaboratively together. Typically, this means that on a regular basis, each fact-checker or team of fact-checkers is given the same item to fact-check, and the results are compared between groups. if one person or group regularly generates different results than the others, then it is evaluated to understand why. Similarly, an individual or group is also periodically given the same or nearly identical story from previous months to see if they give it the same rating as last time; this tests whether they are consistent in their scoring.
More concerning is that we simply don’t know who contributed to a given fact check. David noted that the “snopes process is highly collaborative in that several different people can contribute to a single article” but that “the result is usually credited to whoever wrote the initial draft.” david did not respond to a request for comment about why snopes only lists a single author for each of its fact checks, instead of providing an acknowledgment section that lists everyone who contributed to a given fact check. /p>
It could be argued that newspapers don’t recognize their fact checkers in story lines either. however, in a newspaper’s workflow, fact-checking typically occurs as an editorial function, double-checking what a reporter wrote. In snopes, fact-checking is the primary function of an article, and so if multiple people contributed to a fact-check, it is surprising that they are not mentioned at all, given that in a newspaper all reporters are listed. that contribute to a story. not only does this take credit away from those people, but perhaps more critically, it makes it impossible for outside entities to audit who is contributing to which fact-check and ensure that fact-checkers who identify themselves as highly supportive or against particular topics are not assigned to verify those topics to avoid the appearance of conflicts of interest or bias.
If the privacy or security of fact-checkers is a concern, the site could simply use the first name and last initials or pseudonyms. Having a master list of all fact-checkers who contribute in some way to a given fact-check would go a long way in establishing greater transparency in the fact-checking process and internal snopes controls on conflicts of interest and bias.
David also did not respond to a request for comment about why snopes fact checks rarely mention that they contacted the authors of the article being checked to get their side of the story. In fact, Journalism 101 teaches him that when he writes an article that casts someone or something in a negative light, he should give them a chance to respond and provide their side of the story. instead, snopes usually focuses on the events that are described in the article and contacts the people and entities named in the story, but snopes fact-checks usually don’t mention contacting the authors of the articles about those events to see if those reporters claim to have additional corroboration. material, perhaps unofficially revealed.
Essentially, in these cases, snopes performs a “far-flung fact check,” passing judgment on the news without giving original reporters a chance to comment. david did not respond to a request for comment on this or why the site does not have a dedicated appeals page for authors of stories that snopes has labeled false to contest that label and he also did not respond to a request to provide more details on whether snopes has a formal written appeal process or how it handles such requests.
Putting all this together, we just don’t know if the Daily Mail story is completely false, completely true, or somewhere in between. Snopes has not issued a formal response to the article, and its founder, David Mikkelson, responded by email that he was unable to address many of the claims due to a confidentiality clause in his divorce settlement. this creates a deeply unsettling environment where when one tries to verify the fact, the answer is the equivalent of “his secret”. Furthermore, David’s responses regarding the hiring of strongly partisan fact-checkers and his lack of response on screening and selection protocols present a deeply troubling picture of a secretive black box acting as the final arbiter of truth, but revealing little of its inner workings. this is precisely the same approach used by facebook for its old current affairs team and, more recently, for its policies on hate speech (the company did not respond to a request for comment).
from the outside, silicon valley looks like a gleaming tower of technological perfection. however, once the curtain rises, we see that behind that gleaming facade is a storehouse of good old-fashioned humans, subject to all the same biases and fallibility, but with their results now washed through the glitter of computerized infallibility. even my colleagues who work in the journalistic community and who are by nature skeptical, had assumed that busybodies must have rigorous selection procedures, constant inter-rater and intra-rater evaluations and ongoing evaluations and a mandate for full transparency. however, the truth is that we simply have no visibility into the inner workings of the organization and its founder declined to shed any further light on his operations for this article.
regardless of whether the daily mail article is correct in its claims about snopes, at least what emerges from my exchanges with the founder of snopes is the image of the ultimate black box presenting a shiny appearance of the final arbitration of the truth, but with absolutely no idea of its inner workings. While techies decry the algorithmic black boxes that increasingly power companies like Facebook, they’ve forgotten that even human-powered sites give us little visibility into how they work.
at the end of the day, it’s clear that before we rush to put fact-checking organizations like snopes in charge of arbitrating what is the “truth” on facebook, we need to understand a lot more about how they work internally and more transparency in your job.