Subversion of bootstrap-sass
David A. Wheeler
2019-05-02 (original 2019-04-10)
malicious backdoor has been found in the popular open source software
This subversion was done by someone who created an
unauthorized updated version of the software on
the RubyGems software hosting site.
The good news is that it was
quickly detected (within the day) and updated, and that
limited the impact of this subversion.
The backdoored version (220.127.116.11) was only downloaded 1,477 times.
For comparison, as of April 2019
the previous version in that branch (18.104.22.168) was
downloaded 1.2 million times, and the following version 22.214.171.124 (which
duplicated 126.96.36.199) was downloaded 1,700 times (that’s more than the
subverted version!). So it is likely that almost all subverted systems
have already been fixed.
That said, there's clearly room to improve.
I believe that every time a bad thing happens we should try to find out
how it happened, and then see if there are reasonable steps
that can be taken to reduce the risk.
This paper is part of my
essay suite Learning from Disaster,
which applies this approach to various past problems.
With that in mind, here are a few thoughts
on how to reduce the risk of a similar problem in the future:
- Use 2FA.
It appears that an attacker got the RubyGems password of
If you maintain a package on a package hosting site (like RubyGems,
PyPI, or node.js) or a source code hosting site (like GitHub),
use two-factor authentication (2FA) to authenticate yourself on them.
In the long term, perhaps some of them should require 2FA.
Using 2FA would greatly reduce this risk in general and might have
completely countered this particular attack.
Similarly, if you use such packages, encourage the package maintainers
to use 2FA.
I’ll note that the CII best practices
badge project (which I lead) requires 2FA at the gold level, see: require_2FA.
- Briefly delay non-security updates.
Don't update your dependencies in the same day they're released
unless the update fixes a vulnerability or something else vitally important
to your project.
Instead, wait a few days.
That gives other people (such as the authorized developers)
time to notice and respond to unexpected changes.
Unfortunately some package managers do not make this easy for indirect
dependencies, but you can at least do this easily for direct dependencies.
In the longer term, it'd be great if package managers could do this
automatically (including distinguishing between security updates and
- Require a reproducible build.
The attackers subverted the distributed package on RubyGems
without first posting the corresponding source code in its official source
code repository on GitHub. That’s a big red flag. I believe package
repositories should verify that code distributed can be reproducibly
regenerated from its putative source. Such an approach would have
prevented this attack, and also made the previous “event-stream”
incident far more visible than it was. They don't need to change their
inputs; accept a build, and check that the regenerating the build will
produce the same thing. Package distribution systems could make it
possible to enable this on a per-package basis, so this can be done
incrementally. It will take a while to make this happen, but the first
step is to agree to work towards it. For more about reproducibility,
see the reproducible-builds.org
No doubt there are many more ways to reduce risks like this.
This is, fundamentally, a software supply chain attack.
But since most software is reused, not custom-written,
software supply chain attacks are a real risk and
they need to be addressed.
If you enjoyed this paper, you might also enjoy the entire
suite of related papers in
my essay suite Learning from Disaster.
Feel free to see my home page at
You may also want to look at my paper
Why OSS/FS? Look at
the Numbers! and my book on
how to develop
(C) Copyright 2019 David A. Wheeler.
Attribution-ShareAlike version 3.0 or later