Swatinem Blog Resume

Improving your JS Tooling

OR: how to speed things up considerably

— 10 min

So, I have recently given a talk at the local [viennajs] meetup, which was a huge success apparently. I was basically talking about and live demoing my improvements to the tsc compiler which I wrote about on my blog already. (Side note: the PR is still not merged sigh)

Anyway. Following my talk, Michael Bromley of vendure.io actually went ahead and tried some of my suggestions on his codebase, achieving massive wins! BTW, this is the best feedback anyone can give me! To see that I can have this positive effect on people and their projects makes me super happy :-)

I have then gone ahead and looked in more detail into their testing setup and suggested further improvements, which have also propagated to other projects. (As a side note, both Priscila and me have recently started at Sentry ;-)

So in writing this post, I want to give more details and explanations on why such simple things can have such profound impact, and also give further suggestions on how to speed things up. Because, to be honest, I am quite surprised how people can just put up with their workflows being so slow.

# Explaining skipLibCheck

The massive improvements mentioned above more or less boil down to using skipLibCheck. To understand why, we need to understand how typescript treats the files it loads. There are basically 4 groups of files, here is an excerpt from tsc --listFiles when run on my own rollup-plugin-dts:

  1. ./node_modules/typescript/lib/lib.es2020.d.ts: This and all other files in this directory are the type definitions of the JS built in types themselves. You can select the javascript edition you want write your project in via the tsconfig/lib setting.
  2. ./node_modules/rollup/dist/rollup.d.ts: This is an example of the type definitions of an external library. rollup ships with its own type definitions, but very often, you will consume definitions via @types/XXX, for example @types/node, which you will very likely use.
  3. ./.build/index.d.ts: You might also author .d.ts definition files manually, for example if you want to type external libraries that do not have any type definitions. Or if you want to augment global types, and for various reasons.
  4. ./src/index.ts: Well these are your source files. Nuff Said.

Now, typescript comes with two confusingly named settings, skipDefaultLibCheck and skipLibCheck.

By default, typescript will parse, and typecheck all of the files. When using skipDefaultLibCheck, it will only parse, but not typecheck files from category 1, the type definitions that are bundled with typescript itself. In 99.99% of the cases, you would want to use this setting in your project, since you will never ever touch those files yourself, or ever mess with definitions of js builtins. The typescript team is even thinking about making it the default.

The other setting skipLibCheck will skip all .d.ts files, so categories 1, 2 and 3 in the list above. This is fine to use if you don’t write .d.ts files yourself, which I would say is very likely. Most of the time, you can trust the authors of your external dependencies to have done a proper job. Or can you? :-D Anyway, it is very unlikely that you will either mess with, or conflict with any of the definitions that come from node_modules. And setting this flag can considerably speed up your type checking times.

# Optimizing for Iteration Speed

To shine some more light on the massive saving that Michael was able to achieve, we have to understand the tools that are very common in JS-land, and how they use typescript in the background.

And what everyone has to decide for themselves is what you want to optimize for. I personally want to optimize for development and iteration speed on the one hand, and for correctness and efficiency on the other. As always, it is a matter of tradeoffs.

But lets look at common tools and workflows first. I would argue there are more or less these tasks that you want to do:

Wow, this is a long list of tools, and unfortunately, the default settings they ship with are not optimal, as I think all of these tools run typescript in full typechecking mode by default.

Lets show this visually because it is easier to understand:

  typescript        eslint           jest            webpack
┌────────────┐ ┌──────┬───────┐ ┌──────┬───────┐ ┌──────┬────────┐
│typechecking│ │typeck│linting│ │typeck│testing│ │typeck│bundling│
└────────────┘ └──────┴───────┘ └──────┴───────┘ └──────┴────────┘

In my beautiful unicode-art, you see that it is very inefficient if each one of these tools does its own typechecking. Especially if the typechecking is super slow. I even had to truncate the label to not overflow the diagram :-D

Now, for eslint, doing a typecheck is actually mandatory, since a lot of rules actually use the typechecking information to function. But for jest and webpack, this is both wasted time, and a distraction.

In my opinion, iteration speed is the major selling point of js. I want to quickly try and validate some idea, and then polish it up later. jest and webpack are tools that help me iterate, while eslint will help me polish. I therefore have quite some beef with the default behavior of ts-jest and also with things like fork-ts-checker. They are just slowing me down.

And yes, I do use the default ts-jest behavior on two projects that I maintain, rollup-plugin-dts and intl-codegen, purely because of convenience and because the projects are small. But even then ts-jest is very frequently annoying me. I want to iterate over my tests quickly dammit, not having to perfect my typings! And for webpack, all the errors scrolling through the console are just distracting noise.

With eslint, it will most likely run together with my editor anyway, so I will get auto-fix on save and live highlights. But I know I can ignore those for now if all I want to do is iterate quickly.

So to summarize this point. My suggestion to other projects, especially big ones, is to optimize your workflow for quick iteration on the one hand, for example by disabling the slow, redundant and distracting typechecks in your test runner and bundler. And rather enforce strict typechecks when linting. Essentially configure ts-jest and whatever bundler you use to transpile only.

# Thinking in Compute Time

The above diagram showed very clearly the redundant and often very slow typechecking pass that certain tools do by default. Another thing that developers should be more aware of is Amdahl’s law. What this says is that there are limits to speedings things up when parallelizing.

Lets think of an example. Say your whole testsuite runs for 4 minutes, of which 1 minute is spent typechecking, which is not parallelizable. So no matter how much cores or machines you throw at it, it will never be faster than 1 minute. For example, we can split up our testsuite equally into 3 parts. But each of those runs a typecheck. We end up with a total runtime of 2 minutes, but in reality we actually used up 3 * 2 = 6 minutes of compute time. Meh!

Also, parallelism has some overhead itself. Especially in JS land! For the two small projects that I maintain, I use jests --runInBand flag, to prevent its default behavior of spawning multiple workers, since all that parallelism overhead actually slows things down!

Of course, this very much depends on the size of your project and testsuite, but give it a try!

# Bundling

Now to another topic which I am very opinionated about. Mostly because I am a big fan, advocate and contributor to rollup. This small digression is prompted by a project at my new job that I worked on this past week. Please note that this here is my personal opinion only, yadda yadda yadda.

So, my work assignment made me aware of metro bundler, which is yet another js bundler, as if we didn’t have enough of those already.

I was creating a very small testcase, similar to this:

// module.js
export function foo() {
  console.log("foo");
}
// index.js
import { foo } from "./module";
foo();

And I was expecting to get fairly simple code as a result. Like rollup gives you. You can check the repl for the output.

See, the native module syntax is static, which means it only has instructions for either the native js engine, or a bundler like rollup to know how certain modules interact with each other. All these statements do not actually execute any code at runtime. That is why rollup can actually remove all of. Even without any kind of dead code elimination, bundling should always make your bundle smaller than the sum of your source files, because it should be able to remove all of the boilerplate module code.

Well, at least that is what should happen. Or what I would expect to happen, living in the year 2020, 5 years after the module syntax was specified.

But, alas, running this example through metro gave me a 35k bundle! It was wrapping every module in a function scope, replacing static module syntax with require calls executed at runtime, and throwing in some polyfills for good measure.

WHYYYYYYY?¿?¿?¿?

Please tell me I have done something wrong! This can’t be true! It’s 2020 ffs!

Well, there is one code pattern however that rollup can’t really deal with that well:

if (process.env.NODE_ENV === "production") {
  module.exports = require("./production.min.js");
} else {
  module.exports = require("./development.js");
}

There are some tricks to make things like this work, but there are some things that you can’t handle with static synchronous module syntax. Sadface!

Long story short: Give rollup a try! It is amazing! Especially when publishing a library to npm, bundling via rollup is a must! And when you also ship type definitions, give rollup-plugin-dts a try as well ;-)

# Conclusion

Well, this got a bit longer than expected.

What I wanted to do is raise awareness about how your favorite tools work, and how you can configure them in a way that might better suit your workflows and save you time and money. And to also advocate to bundle your libraries and types ;-)