BlogExtraHop's engineering team had a dilemma. On the one hand, they wanted to focus on growing the team, building product features, and addressing technical debt - tasks which require domain knowledge and, often, experience in the codebase. On the other hand, the infamous JavaScript fatigue threatened to settle in, and it was primarily a result of dealing with lengthening build times. They found themselves asking questions such as, "Do I have enough time to get a cup of coffee while I wait for this build?"_
Our engineers came to the conclusion that outsourcing build improvements was the optimal path forward. Outsourcing build improvements requires relatively little ramp-up compared with outsourcing feature work. Further, they found that the folks who were available had qualifications beyond reproach. These consultants were core contributors to and owners of the build libraries we use. After speaking with a number of consultants we hired Tobias Koppers (webpack's founder) and John Reilly (ts-loader's primary contributor / tech lead). Hard to beat, right?
Even if the average build time was 45 seconds, that's definitely long enough where you'll stop whatever you're working on, switch over to Hacker News or Reddit or get up to refill your coffee.
ExtraHop: Give us a little background on the problem of build times.####
Alex Birmingham, Principal Software Engineer, ExtraHop
Alex Birmingham (Principal Software Engineer, ExtraHop): Prior to starting the consulting project with John and Tobias we were waiting 2 minutes, sometimes 3 minutes, for a build. Even if the average build time was 45 seconds, that's definitely long enough where you'll stop whatever you're working on, switch over to Hacker News or Reddit or get up to refill your coffee. It doesn't seem like that much time, but if you're in "the zone", or what we call a "flow state", 2-3 minutes is enough to disrupt your workflow.
Now, the amount of time wasted on disrupting an individual's workflow is difficult to quantify. But when you've got as many engineers as we do here at ExtraHop, that time spent waiting on builds starts to add up.
And how many builds does ExtraHop do on a given day?####
AB: A typical engineer rebuilds his/her code at least 50 times a day, sometimes even hundreds. In the past, when we had 10 or so engineers, that length of time didn't make as big of a dent in terms of engineering resources. But now that we have many more engineers on staff, it's far, far worse. So the value in trying to optimize for efficiency was really obvious. The overarching goal was to accelerate build times so that we didn't get sidetracked by something else. It's hard to quantify how much of a difference that makes, but in terms of engineering resources, if you can reduce a 34-second build time to 13 seconds, that shaves off 21 seconds. You multiply 21 seconds by the number of engineers, that's a lot of time had been spent each day waiting around.
But then you factor in the amount of time spent getting back into your "flow state"…####
It's much longer, because if you get sidetracked, you say "well, while this is building I'm going to look at my email." So your flow is totally disrupted and it takes even longer to get back into work mode. So we're talking about a number of minutes per interruption, really, not just seconds.
How did webpack's founder and ts-loader's primary contributor come on board?####
AB: Well, let me back up by saying it started when I approached our Director of Engineering and our CTO and co-founder. We had tried some solutions to this internally in the past and realized we were using up a ton of our own resources. I started sending out emails to members of the WebPack user community, one of which was to Juho Vepsäläinen who is the technical writer for WebPack proper. He teaches a course on how to get past JavaScript fatigue, because all of these tools have their limitations and users get exhausted. Performance is just one of them. I reached out to him and he's the one who directed me to Tobias Koppers, and Tobias is the one who directed me to John Reilly.
To me, this seemed like the obvious approach—what better way to speed up our builds than by reaching out to the contributors of these core communities that exist out there. These folks know more than we do about this technology. Better yet, we didn't have to spend time ramping them up on ExtraHop and our code because all they would be dealing with is the build. So there was very little friction.
John Reilly, primary contributor/tech lead, ts-loader
John Reilly: I had some ideas around how to make improvements around the TypeScript build, but I was less confident providing support with general webpack improvements. So I was very pleased Tobias would be on hand to provide expertise!
Typically when you're working on a TypeScript project you'll kick off a process that watches all of the files you're working on. When you make changes to your files, whatever build process you've got running in the background spins up, it recompiles your TypeScript and your JavaScript and it quite probably runs your unit tests as well. Depending on how you've got your build system setup, that can be a really really slow process. Certainly it can become that as the project scales.
Did you know immediately which tools you would use for this solution, or was it trial-and-error?####
JR: I already knew some of the general tips for speeding up builds. However, I had something particular in mind. It just so happened that shortly before starting work with ExtraHop, I'd heard about another project called fork-ts-checker-webpack-plugin. Before I talk about that I should preface this by talking about what ts-loader does. ts-loader has two responsibilities; the first is to take TypeScript source code and, using the TypeScript compiler, compile it into JavaScript and hand it back to webpack. The second thing ts-loader does is pick up all of the compilation errors that the TypeScript compiler has found and report them to webpack. fork-ts-checker-webpack-plugin's purpose is to take on that second responsibility from ts-loader. So basically, it says, "ts-loader, don't worry about the errors, I'm going to take care of them". The project is inspired by an approach awesome-typescript-loader has used; to run error checking in a different process to the transpilation process. In doing so, the plugin reduces the work that ts-loader needs to do and opens up some opportunities to speed up builds. I hadn't done much with the fork-ts-checker-webpack-plugin at that point, but it seemed very much worth a go.
Now, there's another project out there called HappyPack, which makes webpack builds faster by allowing you to transform multiple files in parallel. It splits up the compilations across various different processes, which consequently speeds up your builds. I was aware of someone raising a PR for ts-loader which added support for that. So, if I took HappyPack and fork-ts-webpack-checker-plugin—one of which allows you to split up ts-loader builds across multiple processes and the other which says "I can handle catching and reporting the errors"—I though that might be the answer for speeding up the builds. And it turned out to be very successful. Tobias was then able to take the principles which underpin HappyPack and create a further optimized approach in the form of thread-loader with cache-loader.
It's cool if the ExtraHop UI team reduces their build time by 33-percent, but it's a lot cooler if you take that and apply it to the worldwide JavaScript community.
And the results so far…?####
AB: In one case, we saw a common build process go from 45 seconds to 12 seconds. Overall, the reaction from our engineers has been largely positive. There have been some inconveniences introduced as a result of this work, but they have been more-than-offset by the improvements. What was really satisfying for me is there were a couple of points during the course of this project where there were fixes made to JavaScript community libraries that were potentially prompted by problems ExtraHop was having. For me, there was an immense amount of satisfaction in that because I wanted all along for the community to see benefits from our efforts. Sure, it's cool if the ExtraHop UI team reduces their build time by 33-percent, but it's a lot cooler if you take that and apply it to the worldwide JavaScript community. Ts-loader has been downloaded 740,168 times in the last month. Multiply that by the number of builds executed per download and seconds (or minutes) saved per build, and you've got an enormous net benefit to the community. It's extraordinary how much time will be saved worldwide.
Check out John Reilly's article "TypeScript + webpack: Super Pursuit Mode" on Medium.
If you're interested in joining an award-winning team committed to doing awesome stuff like this, check out our Careers page.
