Facebook Stopped Employees From Reading An Internal
Report About Its Role In The Insurrection. You Can Read It Here.
After BuzzFeed News reported on an
internal document that examined the social network’s failings leading up to the
Capitol riot, many of Facebook's employees were prevented from accessing it.
Posted on April 26, 2021, at 2:23 p.m. ET
Last Thursday, BuzzFeed News revealed that an
internal Facebook report concluded that the company had failed
to prevent the “Stop the Steal” movement from using its platform to subvert the
election, encourage violence, and help incite the Jan. 6 attempted coup on the
US Capitol.
Titled “Stop the Steal and Patriot
Party: The Growth and Mitigation of an Adversarial Harmful Movement,” the
report is one of the most important analyses of how the insurrectionist effort
to overturn a free and fair US presidential election spread across the world’s
largest social network — and how Facebook missed critical warning signs. The
report examines how the company was caught flat-footed as the Stop the Steal Facebook group supercharged a movement to undermine democracy, and
concludes the company was unprepared to stop people from spreading hate and
incitement to violence on its platform.
The report's authors, who were part
of an internal task force studying harmful networks, published the document to
Facebook's internal message board last month, making it broadly available to
company employees. But after BuzzFeed News revealed the report's existence
last week, many employees were restricted from accessing it.
“Is
there a reason the Workplace Note has been taken down?” one employee wrote on
the message board after the report became restricted. “I suspect employees
would prefer to read it for themselves and draw their own conclusions.”
“It’s
pretty common that critical writing about the company gets removed under some
trumped-up excuse if it gains any internal or external traction, it’s not about
the public visibility but the morale effects I imagine,” another worker said.
“I
suspect employees would prefer to read it for themselves and draw their own
conclusions.”
Given
the newsworthiness and historical significance of the report and its
revelations about the events of Jan. 6, BuzzFeed News is publishing the full
text below.
“The
authors never intended to publish this as a final document to the whole
company," a Facebook spokesperson said in a statement. "They
inadvertently published it to a broad audience and they simply restricted it to
the internal working group it was intended for."
The
spokesperson added that it was the authors who restricted access to the report.
The
company has defended its work to protect the 2020 election. Last month in
testimony before the House Energy and Commerce Committee, Facebook CEO Mark
Zuckerberg said that though the company had not caught all election
interference before the insurrection, it had “made our services inhospitable to
those who might do harm.”
“We
are committed to keeping people safe on our services and to protecting free
expression, and we work hard to set and enforce policies that meet those
goals,” he wrote in prepared comments to
that committee. “We will continue to invest extraordinary resources into
content moderation, enforcement, and transparency.”
On
Tuesday, Monika Bickert, Facebook’s vice president of content policy, is set to
testify in a Senate Judiciary Committee hearing on algorithmic amplification on
technology platforms alongside executives from YouTube and Twitter.
Here
is the full text of Facebook’s internal report. Some graphics were not
reproduced due to their technical nature.
Stop the Steal and Patriot Party: The
Growth and Mitigation of an Adversarial Harmful Movement
[The
Facebook report included a cover image here, featuring a burning US Capitol
building and a cartoon corgi dressed as a firefighter.]
TLDR
- Stop the Steal
(StS) grew rapidly after the election as a movement, but our enforcement
was piecemeal.
- Treating StS as
a network allowed us to understand coordination in the
movement and how harm persisted at the network level. This harm was more
than the sum of its parts.
- Examining the
StS network allowed us to observe the growth of Patriot Party.
- We learned a lot
from these cases. We’re building tools and protocols and having policy
discussions to help us do this better next time as part of the
Disaggregating Networks taskforce.
Intro
Many
of us remember election night and the few days following. The satisfaction at
having made it past the election without major incident was tempered by the
rise in angry vitriol and a slew of conspiracy theories that began to steadily
grow. At the time, veterans of 2016 recalled the spike in fear, anger, and
uncertainty, the growth of mega-groups like Pantsuit Nation. We all asked ourselves
whether what we were seeing in the wake of the election was the same thing, or
something more nefarious. Hindsight is 20/20, at the time it was very difficult
to know whether what we were seeing was a coordinated effort to delegitimize
the election, or whether it was protected free expression by users who were
afraid and confused and deserved our empathy. But hindsight being 20/20 makes
it all the more important to look back to learn what we can about the growth of
the election delegitimizing movements that grew, spread conspiracy, and helped
incite the Capitol Insurrection.
The
first Stop the Steal Group emerged on election night. It was flagged for
escalation because it contained high levels of hate and violence and incitement
(VNI) in the comments. The Group was disabled, and an investigation was kicked
off, looking for early signs of coordination and harm across the new Stop the
Steal Groups that were quickly sprouting up to replace it. With our early
signals, it was unclear that coordination was taking place, or that there was
enough harm to constitute designating the term. It wasn’t until later that it
became clear just how much of a focal point the catchphrase would be, and that
they would serve as a rallying point around which a movement of violent
election delegitimization could coalesce.
“Delegitimization”
(D14N) as a concept is new territory, both for analysis and policy. Many D14N
workstreams were spun up in the wake of election night, but few policies or
knowledge existed around the issue. Our research during the US2020 IPOC came
from rapid work on topic classifiers, CIRD pipelines, regex and classifier
tracking in HELLCAT, and manual analysis via CORGI modeling, and we were able
to launch demotions and some enforcement directed at the issue, but work
remains to develop a firm policy framework around addressing the issue. In this
note we will describe the harms we were later able to observe within the StS
movement, how follow-on movements like Patriot Party (PP) were able to grow in
its wake, and how we might use what we learned to better capture coordinated
harm in the future.
Early
Indicators of Harm
From
the earliest Groups, we saw high levels of Hate, VNI, and delegitimization,
combined with meteoric growth rates — almost all of the fastest growing FB
Groups were Stop the Steal during their peak growth. Because we were looking at
each entity individually, rather than as a cohesive movement, we were only able
to take down individual Groups and Pages once they exceeded a violation
threshold. We were not able to act on simple objects like posts and comments
because they individually tended not to violate, even if they were surrounded
by hate, violence, and misinformation. After the Capitol Insurrection and a
wave of Storm the Capitol events across the country, we realized that the
individual delegitimizing Groups, Pages, and slogans did constitute a cohesive
movement.
Some
of our first indicators use off-platform signals, finding that designated
organized hate groups were involved in organizing Storm the Capitol (StC)
events using CORGI fanouts, and were involved in pushing Stop the Steal. We
also found that there was high membership overlap between StS Groups and Proud
Boy (a designated DOI org) and militia Groups.
We
looked at the content of Groups and Pages, comparing the rates of hate speech,
vni, and DOI references in StS, PP, and StC Groups using the HELLCAT tables,
which aggregate a myriad of integrity-based content signals to the complex
entity level. This allowed us to see that StS groups had considerably more
hate, vni, and references to conspiracy and militias than the average civic
Group as a whole.
In
addition to HELLCAT, we were built fast turnaround classifiers and CIRD
pipelines to identify high risk Groups and other complex entities. These CIRD
pipelines were wired to demotions, as well as aggregated to surface high risk
complex entities. Misinfo escalations were also frequent, although the volume
far outstripped 3PFC or escalation review capacity. Together, these approaches
allowed us to flag individual Groups and Events with high levels of harm for
review through HEROCO or the Events queue.
BuzzFeed
News recreated a data table from an internal Facebook report comparing hate
content within Stop the Steal (STS) and Patriot Party groups to all civic and
political groups on the social network.
These
content-based approaches allowed us to observe how harm manifested in the
movement as a whole, showing that the terms were steeped in hate and VNI. This
helped us see that there was a problem, but network analysis helped us
understand coordination in the movement, and how the harm was able to spread as
a network. Understanding the growth of the network will help us to better
tackle harmful networks in the future.
Coordination
We
were able to observe direct coordination for Stop the Steal through
investigative work, relying on external sources for leads.
The
terms Stop the Steal and Patriot Party were amplified both on platform and off.
Ali Alexander and the Kremer sisters repeated slogans at rallies, and spread
them through super Groups like Women4Trump and Latinos for Trump. The Kremer
Sisters were admins of both Women4Trump, and the original Stop the Steal Group.
After January 6th, Amy Kremer confirmed on platform that she was an organizer
for the Stop the Steal rally that precipitated the Capitol Insurrection.
Ali
Alexander worked on and off platform, using media appearances and celebrity
endorsements. We also observed him formally organizing with others to spread
the term, including with other users who had ties to militias. He was able to
elude detection and enforcement with careful selection of words, and by relying
on disappearing stories.
This
sort of deep investigation takes time, situational awareness, and context that
we often don’t have. What sort of behavioral signals might we be able to
leverage to observe coordination when we lack the time or background for in
depth investigations? What sort of analyses and models might we build to help
us identify these networks in the future?
Group
Inviters
One
way to observe coordination in a movement is by looking at growth hacking.
Growth hacking might not always be bad. A democratic movement, a movement
seeking human rights, or even an advertising movement, may all employ
legitimate techniques to grow their audience quickly. However, when the growth
is mixed with the signals of harm we described above, this rapid growth
indicates the spread of harm, and may indicate coordinated harm.
Stop
the Steal was able to grow rapidly through coordinated Group invites: 67% of
StS joins came through invites. Moreover, these invites were dominated by a
handful of super-inviters: 30% of invites came from just 0.3% of inviters.
Inviters also tended to be connected to one another through interactions — they
comment on, tag, and share one another’s content. These were inviters with more
than 500 invitees each. In the top StS Group, there were 137 super inviters,
with 500 invitees each. Of these users, 88 were admins of other StS Groups,
suggesting cooperation in growing the movement. These super-inviters had other
indicators of spammy behavior: 73% had bad friending stats, with a friend
request reject rate above 50%. 125 of them likely obfuscated their home
locations. 73 of them were members of harmful conspiracy Groups. We also saw
that inviters to these Groups tend to be connected. At the beginning of
January, before the post-insurrection spike in StS and PP Groups, half of all
inviters with > 100 invitees also engaged with one another either directly
through messaging and tagging, or with one another’s content in the previous
month, suggesting that many inviters were connected to one another.
[The
Facebook report included a graphic here, showing a network of how “most heavy
inviters are connected to one another.”]
This
growth occurred despite our attempts to prevent it: the Groups Task Force
identified risks around Group inviting leading to the rapid growth of
anti-quarantine Groups. Super-inviters were able to quickly grow new Groups,
both allowing the rapid growth of harmful Groups, and helping to avoid
enforcement as backup Groups replaced disabled Groups. In response, a cap of
100 invites/person/day was implemented. We released an additional new invite
rate limit of 30 adds/hour (now deprecated) during the growth of Stop the Steal
Groups for users adding new friends (< 3 days) to new groups (< 7 days)
to Groups with some certain ACDC properties. However, all of the rate limits
were effective only to a certain extent and the groups were regardless able to
grow substantially.
Any
successful movement also has organic growth that should not be discounted. A
third of the growth came from self joins, and while the plurality of the
inviting came from a handful of users, 82% of inviters invited fewer than 10
others. This combination of growth hacking with organic growth made exemplified
how complicated harmful network movements can be. In order to explore this
growth and the extent to which it was driven by amplification of the slogans,
we explored the way the Content flowed through the broader StS network, in
Groups and beyond.
Understanding
the Network
Using
Information Corridors, we were able to identify the larger community where StS
and election delegitimization was discussed most heavily. We started by
identifying users who posted the most using delegitimizing language, and who
used a wide variety of terms. These were our high StS engagers. We then fanned
out to everyone they interacted with, and identified those users who were also
using a lot of Stop the Steal language, or who had a high propensity for doing
so based on our classifiers. This network of high StS users was our Information
Corridor (IC). It identifies the part of the social network on platform where
the harmful content is circulating. For an overview demo of Stop the Steal
Information Corridors, more detail see here.
Out
of 6,450 high engagers, 4,025 (63%) of them were directly connected to one
another, meaning they interacted with one another’s content or messaged one
another. When using the full Information Corridor, 77% were connected to one
another. This suggests that the bulk of the Stop the Steal amplification was
happening as part of a cohesive movement.
[The
Facebook report includes a network diagram here, showing how "Information
Corridors allow us to identify the part of the network where harm
circulates."]
By
tracking these language networks, we can better capture the coordinated harm
that flows through the network. Members of the corridor produced 33% more hate,
31% more VNI compared to the broader community around the high engagers. Members
of an information corridor are vulnerable to the harmful message that is being
propagated because they are subject to, and most likely to engage with,
that harmful content. Amplifiers in the IC are users who are connected to many
other of these vulnerable users, so named because anything they say reaches a
larger audience. By looking at patterns in amplifier language, we can better
understand the harms that are being pushed through the IC. Amplifiers
posted 98% more VNI and 40% more hate. The core of that network had 85% more
VNI, and 45% more hate. We also identified the core of the IC — the
set of users who all tightly engage with one another, using k-core
decomposition.
[The
Facebook report includes a graphic here, showing relationships between
"closely connected users at the center of the network.”]
In
order to understand how the movement perpetuates harm, we also need to
understand the extent to which it persists beyond the coordinators and
amplifiers. We also want to understand the extent to which users who interacted
with coordinators and high engagers are also producing harm. To do this, we
looked at audience closeness around the inviters described above. Users who
engaged the most with inviters with at least 50 invitees. Those users who
interacted the most with those inviters produced 92% more VNI, and 49% more
hate. Relatedly, we also found that information corridors help link the users
in the core of the StS network to those in the periphery, helping spread the message
across the full network.
Overall,
we were able to show that where PP and StS signals were being amplified through
content and inviting, there was also higher levels of hate and violence,
suggesting that these movements were harmful and that the harm was perpetuated
through a network that we were able to define.
Growth
of Patriot Party (PP)
Stop
the Steal wasn’t the only movement that grew around the D14N theme. Patriot
Party was another movement that grew out of and eventually in competition with
StS, showing similar levels of harm. Many of the coordinators for PP expressed
a disappointment with the StS movement’s failure to do what it promised, and a
need to go further by bringing on systemic change through a new political
party. On the other site, StS admins and real-life leaders had a large amount
of celebrity and official-ness about them (the ones that weren’t banned from
platform — Trump, Roger Stone, Alex Jones...etc.) that they didn’t necessarily
want to be seen as deflecting from the traditional Republican Party to start a
scrappy, potentially angrier Patriot Party.
Admins
of PP attempted to recruit members from StS and Joe Biden is NOT my President
Groups. Popular posts and frequent posters on PP pages and groups often used
the Stop the Steal slogan, especially prior to inauguration. We also saw that
PP was able to grow within StS corridors: members of the StS IC were 6% more
likely to use the term “Patriot Party.” In the end, PP never grew as much as
StS, in large part because of the lessons we learned from StS and were able to
rapidly apply to PP.
[The
Facebook report includes a pair of graphics here, showing how “Information
Corridors” allowed the company to “track additional linguistic signals that
grow within the network.”]
The
leaders of PP had mixed success recruiting from StS supporters. As StS Groups
were disabled, we their members flock to PP Groups: 20% of the Groups that
members of disabled StS Groups joined were PP Groups. We were able to mitigate
this growth by feature limiting Groups that many users joined after being
disabled as an election Break The Glass measure. However, StS groups weren’t
the main source for PP Groups: only 6.5% of actioned PP Group members were part
of an actioned StS Group, and only 1.1% of actioned StS Group members joined an
actioned PP Group, with only 3 out of roughly 1000 shared admins. Moreover, we
saw that PP was primarily pushed by amplifiers within the StS IC who were not
fully successful: we did not see widespread use of the PP term by less engaged
members of the IC — learning from our previous work on StS, we were able to
stop PP before it was able to spread.
[The
Facebook report included a graphic here, showing “Group membership Jaccard
similarity.”]
Crisis
Response
Tracking
Evolving, Inter-Related Movements
One
of the most effective and compelling things we did was to look for overlaps in
the observed networks with militias and hate orgs. This worked because we were
in a context where we had these networks well mapped. During crises there are
likely to be multiple escalations ongoing at once, with different teams
focusing on different networks around DOI, misinfo, and other harms. By
combining these, we could better understand how the nature of the harm being
coordinated, and the myriad of tactics being used. As PP arose, showing the
connection between PP and StS helped us understand the harm being perpetuated
by PP in context, when the harm might have been less apparent alone.
We
were also able to add friction to the evolution of harmful movements and coordination
through Break the Glass measures (BTGs). We soft actioned Groups that users
joined en masse after a group was disabled for PP or StS, this allowed us to
inject friction at a critical moment to prevent growth of another alternative
after PP was designated, when speed was critical. We were also able to add
temporary feature limits to the actors engaging in coordinating behaviors, such
as the super posters and super-inviters in the Groups that were removed, to
prevent them from spreading the movement on other surfaces. These sets of
temporary feature limits allowed us to put the breaks on growth during a
critical moment, in order to slow the evolution of adversarial movements, and
the development of new ones. Our ongoing work through the disaggregating
networks taskforce will help us to make more nuanced calls about soft actions
in the future in order to apply friction to harmful movements.
Signals
of coordinating harm
In
addition to the network evolution tracking described above, several signals were
particularly useful for helping us identify coordinated harm. Specifically:
- Content signals: Aggregating
many signals from content within a complex entity helped us get to a
comprehensive view of what was happening within those entities in order to
understand harms broadly. We used HELLCAT tables to understand the
relationship between text signals related to a movement or escalation and
hate and violence. It also allowed us to compare signals from many ongoing
escalations and harms, such as looking for relationships between StS, StC,
and PP with Qanon and militias. We were also able to use the CIRD table to
quickly and easily search for complex entities within new text signals.
These tables also enabled us to quickly spin up D14N classifiers for both
content and complex entities once a designation did occur.
- Signals of rapid
growth and amplification: An important signal was the
rate of growth of Groups. This growth appeared to be through amplification
and coordination: the Groups share common admins and super inviters, with
these influential individuals participating across Groups.
- Rapid URL
sharing is another way that a movement can be amplified and spread
quickly, representing the movement through off-platform sources that are
harder to enforce on. Building off our lessons learned about
amplification, we’ve also built tools (Information Corridors) to help us
understand the growth of slogans and terms within a movement, in order to
understand how it’s being amplified and how quickly it’s growing.
- Branding: Not all
movements will have common branding, but when they do, it is a clear sign
of coordination. PP Groups and Pages used the same or similar logos to
identify official sources.
- Admin-only
Groups and formal organizational structure represented
on the platform: PP had admin-only Groups where formal coordination was
organized. This does not always occur on platform, especially with
adversarial networks, but is a clear signal when it does.
[The
Facebook report included three examples of Patriot Party logos here.]
Conclusion
Gaps:
There
are many lessons we can take away from our successes and challenges with
mitigating StS and PP which are critically valuable for understanding gaps in
detection, enforcement, and policy.
- Early focus on
individual violations made us miss the harm in the broader network.
- Designation
differences between STS and Storm the Capitol made it hard to enforce
because we couldn’t count strikes. The seams between policy areas
made it harder to have a unified effort to tackle the delegitimizing
harm as a whole, instead forcing us to target different parts of the
problem piecemeal, with the larger wave of the movement seeping through
the cracks.
- We were able to
observe growth of PP through StS, but this was a very manual process. It
made us question what we were missing, and would rise from the ashes once
we turned our attention away. Moreover, StS and PP were in competition
with one another, enforcing on StS maye have helped PP to grow. We need
tools and protocols for handling evolution of movements in the
future, and for quickly designating new movements around old harms that
arise when the field is cleared of competition.
- We have little
policy around coordinated authentic harm. While
some of the admins had VICN ties or were recidivist accounts, the majority
of the admins were “authentic.” StS and PP were not directly mobilizing
offline harm, nor were they directly promoting militarization. Instead,
they were amplifying and normalizing misinformation and violent hare in a
way that delegitimized a free and fair democratic election. The harm
existed at the network level: an individual’s speech is protected, but as
a movement, it normalized delegitimization and hate in a way that resulted
in offline harm and harm to the norms underpinning democracy.
- What do we do
when a movement is authentic, coordinated through grassroots or authentic
means, but is inherently harmful and violates the spirit of our policy?
What do we do when that authentic movement espouses hate or delegitimizes
free elections? These are some of the questions we’re trying to answer
through research and tool building in the Disaggregating Harmful Networks
Taskforce, and that we’re wrestling through in the Adversarial Harmful
Networks policy xfn.
- A policy of
coordinated authentic harm needs a broader definition of
coordination to handle network or movement level harms and the
interplay between organic and inorganic growth. It was hard to establish
coordination (outside of the same logo usage) across hundreds of
Groups/Pages due to the movement not being driven by a few actors, but
rather being “adopted” and “promoted” by authentic users.
- We need a range
of full spectrum interventions from hard action to soft
action in order to better handle the growth of organic harmful movements.
Our narrow definition of coordination is centered around hard punitive
actions. In order to slow the growth of movements, we should learn from
our BTGs and apply a range of count-interventions, friction, soft actions,
and hard actions in order to promote a healthier community beyond
targeting the worst of the violators.
- Enforcement
lacked a single source of truth. Bulk enforcement, continuous
enforcement, and adhoc enforcement had inconsistent labeling and case
attribution, which made recidivism analysis difficult, made it harder to
track the evolution of the movements, and made retrospectives and
follow-up research more difficult.
Next
Steps:
Luckily,
we’ve learned a lot from the US2020 IPOC and the StS and PP cases. Here are our
next steps.
- Building out the
tools we used:
·
- Query banks and
recipe books to replicate our ad hoc analyses quickly
- Add the steps
we took into formal protocols
- Improve and
integrate techniques into tooling such as C4, ENVI and ANP notebooks, and
integration into Centra.
- Building new
methods around network disaggregation and adding them to our tooling. Stay
tuned for future integration of core-periphery modeling and information
corridors into CORGI tooling.
- Teaching
investigators how to use the tools and techniques we’re developing through
communities of interest such as Network Tools for Investigators Group, the
Actor Investigation XFN, and improved documentation.
- Using these
cases, and tools, to help us understand organic coordinated harm, and harm
within networks for further policy development. Stay tuned to more notes
like this one as we continue to learn more!
- Test out these
new methods on ongoing investigation. For example, we are using our
disaggregating network techniques to identify users for counter-speech
interventions around Us hate groups. We’re also working on a set of cases
in Ethiopia and Myanmar to test the framework in action. We’re working
with BONJOVI to put together protocols for investigators.
Please
get in touch if you have a use case that would benefit from using these tools
and protocols! We’d love to work with you to help you track a broader harmful
network and understand coordination within it.
