The deep inequalities of Facebook’s secretive cross-check moderation program

Today, let’s talk about a new opinion from Meta’s Oversight Board that attempts to square a tension at the heart of any big social network: on one hand, a desire to treat its users with equality; and on the other, an acknowledgement that in practice some groups of users deserve special treatment.

No company wants to say outright that some of its customers matter more than others. But in practice, on social networks in particular, some customers do — at least when it comes to how their accounts and posts are treated. If the president of a country regularly communicates with their citizens on a platform (particularly a country that might ban or otherwise restrict your business), you want to be extra careful with any steps you take to remove their posts or suspend their account.

In short, you don’t want to make a mistake.

It was in this desire not to make mistakes that the company then known as Facebook created “cross check” — sometimes styled “Xcheck” — and implemented it around the world. The program’s existence was first documented by the Wall Street Journal as part of Frances Haugen’s revelations last year. Jeff Horwitz’s article revealed that some employees felt that the program offered undue protections to some of the platform’s most prominent users, often allowing harmful content to stay live for many days before Facebook intervened.

A 2019 internal review of Facebook’s whitelisting practices, marked attorney-client privileged, found favoritism to those users to be both widespread and “not publicly defensible.”

“We are not actually doing what we say we do publicly,” said the confidential review. It called the company’s actions “a breach of trust” and added: “Unlike the rest of our community, these people can violate our standards without any consequences.”

Days after the article’s publication, the Oversight Board — an independent body funded by Meta with the intention of serving as a check on its content moderation and policy decisions — said it would investigate. A week after that, Meta formally asked the board to issue an advisory opinion on what it ought to do with cross-check and the basic idea that it will give some posts and accounts an extra layer of review in an effort not to screw up.

More than a year later, on Tuesday the board delivered its opinion. I’ve recently been critical of the board’s slowness — its 23 members managed to produce just three opinions last quarter — but in this case it wasn’t entirely the board’s fault.

As described in the 57-page opinion, Meta dragged its feet during the board’s fact-finding process, in one case waiting almost five months to answer a question about what sorts of accounts qualify for cross-check protection. Even then, Meta only offered “limited aggregate data about each listed entity on the current,” the board said.

(Meta cited privacy concerns, as companies often do in these situations, but the board said it “believes, and has pointed out to Meta, that these concerns could have been mitigated and more extensive disclosures provided.” In the relatively dry, somber world of communications between Meta and the board, I thought this qualified as a fairly sick burn.)

Despite the dodges and delays, the board eventually obtained a great deal of information about cross-check, and the opinion lays it out in admirably thorough detail. And after laying out the shape of the program, the board reached some important conclusions.

We’ll get to those conclusions in a minute. But first, it’s worth giving a bit more explanation to what cross-check is. Notably, after the Journal’s report, Meta broadened the scope of the program in an effort to reduce the inequality inherent in the system. The board summarizes all this in one neat paragraph:

According to Meta, making decisions about content at this scale means that it sometimes mistakenly removes content that does not violate its policies. The cross- check program aims to address this by providing additional layers of human review for certain posts initially identified as breaking its rules. When users on Meta’s cross-check lists post such content, it is not immediately removed as it would be for most people, but is left up, pending further human review. Meta refers to this type of cross-check as “Early Response Secondary Review” (ERSR). In late 2021, Meta broadened cross-check to include certain posts flagged for further review based on the content itself, rather than the identity of the person who posted it. Meta refers to this type of cross-check as “General Secondary Review” (GSR).

Creating a second kind of cross-check program and nominally expanding it to all users had obvious public-relations benefits for Meta, which could then argue that the protections it offers for wrongful removal of content extend to all users. In practice, though, the board found that the “general secondary review” queue had too few people working on it, and that posts marked for such review are often never given one.

That leads us to the board’s conclusions. The first and most important of these tracks with the central claim in the Journal’s reporting: it leads to “unequal treatment of users.” The board writes:

Cross-check grants certain users greater protection than others. If a post from a user on Meta’s cross-check lists is identified as violating the company’s rules, it remains on the platform pending further review. Meta then applies its full range of policies, including exceptions and context-specific provisions, to the post, likely increasing its chances of remaining on the platform. Ordinary users, by contrast, are much less likely to have their content reach reviewers who can apply the full range of Meta’s rules. This unequal treatment is particularly concerning given the lack of transparent criteria for Meta’s cross-check lists. While there are clear criteria for including business partners and government leaders, users whose content is likely to be important from a human rights perspective, such as journalists and civil society organizations, have less clear paths to access the program.

And it’s not just that the program leads to unequal treatment. It’s that, thanks to the fact that Meta wouldn’t share complete data on who has qualified for ERSR, we don’t even know how unequal it is. It’s possible that, given how little Meta eventually chose to share on this subject, the inequality runs much deeper than even the board assumes here.

A second conclusion is that this inequality causes harm. If a prominent account posts something awful on Facebook, ERSR means that it won’t be removed by automated systems. Facebook’s goal in these cases is to remove content within 12 hours to five days. In practice, though, the board found that even in the United States, it takes the company “approximately 12 days on average to make a decision.” In one case, the board found that the company took a staggering 222 days to evaluate a post.

To state the obvious here, that’s a long time for hate speech, violence, and other violations of the company’s community standards to be live on the site.

A third important conclusion the board reaches is that, however important cross-check is to the company, it doesn’t really measure the program’s effectiveness. “For example, Meta did not provide the board with information showing it tracks whether its decisions through cross-check are more or less accurate than through its normal quality control mechanisms,” the board says. And if you’re going to leave potentially harmful stuff up on Facebook for 222 days, wouldn’t you want to know if it is?

In short, while Meta often dresses up its policy decisions in the language of human rights, the board finds that — as you would probably expect — cross-check is primarily designed to serve business needs.

And so what do you do about it?

If it were up to the board, Meta would do a lot about it. Members made 32 different recommendations for the company, to which is now obligated to respond. (The company requested, and received, an extra-long 90-day period to consider all of the board’s requests.)

Importantly, though, the board fundamentally supports the idea of a secondary review system. It acknowledges that large-scale content moderation systems will inevitably make mistakes, and that Meta should act to reduce them. And it accepts that this means designating some accounts for special protections.

At the same time, the board finds that Meta’s implementation of such a system is deeply problematic, and arguably a scandal of its own. For example, while just 9 percent of Facebook’s daily active users are from the United States or Canada, 42 percent of all the ERSR content reviewed by the company came from those two countries. It’s a powerful illustration of something we have long known: that as inadequate as content moderation can feel to us North Americans, the situation is much worse in the rest of the world. And that can and does lead to very real harms.

The board calls for developing much stronger criteria for which accounts should be eligible for ERSR protections, making those criteria public, and allowing individuals to proactively apply for the program. It says Facebook should require anyone who is included to first review the community standards and agree to uphold them. And it says Facebook should proactively communicate changes to those standards so users will know how to comply with them.

Once an account has been accepted into the program, the board says its status should be visually communicated to users. That way, if you see a harmful post from a government official or celebrity left up for 222 days, you’ll have some idea of why it hasn’t yet been removed. And, if you report a post on such an account, the board says that you should be told the account you’ve reported has special protections.

These are really good ideas!

As for the other half of cross-check — GSR, which attempts to elevate potential mistakes for secondary review no matter which account posted them — the board recommends that Facebook build more capacity to ensure it’s actually reviewing those posts, rather than just funneling them into a queue that in practice is often ignored. That’s a direct request for Facebook to shift funding resources, and I’m fairly sure that’s a first for the board.

How will the company respond? I talked with Meta on the ground rules that I couldn’t name who I talked to, or quote them directly. Mostly I wanted to know what the company makes of the critiques and recommendations in the opinion, which it had also been briefed on. But the company didn’t want to say anything until the opinion was public. A statement from Nick Clegg, the company’s vice president of global affairs, outlined changes the company has made to the program already and said only that it would share more in 90 days.

Last year, the mere existence of cross check struck some observers as an outrage. But to the board, the idea of secondary review turned out to be a practical necessity for a large-scale social network.

At the same time, there’s plenty in cross-check that’s ugly to see. And the board’s recommendations for transparency, auditing, and heightened focus on the system’s design would go a long way toward making content moderation more equitable.

Of course, these can seem like minor concerns in a world where some states have passed laws decreeing that content moderation of nearly any sort should be illegal. But as long as platforms still have the right to remove bad posts — and frequently make mistakes in doing so — the rest of us deserve better than we’re currently getting, on Facebook and everywhere else.

Here’s hoping the company takes its board’s recommendations seriously, and brings more equality to a platform where that remains in disappointingly short supply.