CommonJS require(..) API is a poor fit for client side JS
This post is in response to David Flanagan's client side JS implementation of CommonJS's module pattern/API.
There are several performance considerations with script/module/dependency loading.
- script tags in markup block the page progression
- synchronous XHR blocks the js thread and therefore the UI responsiveness
- script tags added to the DOM via js do not block page progression, but their execution order must be managed
- there are severe performance penalties for numerous http requests (http meta/overhead, time, and browser's concurrent resource request limits)
- "discovered" dependency resolution == chained http requests (sync or async), so deep deps trees could take a long time to resolve. A reqs B reqs C could result in load A, then load B, then load C, then execute C, then B, then A, then execute requiring code (see #2 and #4).
The CommonJS API has great qualities for environments that don't suffer from the sorts of IO costs and conditions that make sync behavior a Bad Idea(tm) in client side scripting. The implementation as one-expression-per-requirement is especially bad because it is functionally equivalent to multiple blocking script nodes locking up the page progress or rendered UI, which is somewhere between bad and horrible. To avoid this, require might be a registration function of an async loader to bundle multiple require(..) calls from the same thread, but that breaks the require-then-use contract.
The YUI 3 module and use(..) arch is basically an async implementation of key parts of this API. This is especially obvious in the YUI.add(moduleName, function (Y) { /* add stuff to Y */ }, ver, meta); form of registering modules where you could easily substitute export for Y.
The big differences are that Y.use(..) accepts multiple modules/requirements to avoid the one-expression-per-requirement, it does pre-fetch dependency resolution based on a managed dependency metadata tree, and the modules are loaded asynchronously using as few requests as possible (the Yahoo! CDN combo service makes 1 request the common case), executing a callback when complete.
The less than optimal part of this is the dependency resolution tactic, especially for modules from third parties. Delaying dependency calculation for a module until that module is loaded would kill performance, so all dependency metadata for library modules is stored in a meta module for pre-fetch resolution. We already know this won't scale in a non-impactful way, and of course this doesn't address third party modules. For third party modules, we require dependency metadata in the configuration for requiring code, which is less than ideal. For optimal performance and client side footprint, dependency resolution basically has to happen on the server side or be integrated into a pre-deployment build step.
All that said, my main point is that as exciting as the recent server-side JS movement has been, the synchronous require(..) pattern is a poor fit on the client side for the high performance demands of today's sites and RIAs.
4 Comments
November 25, 2009 12:56 AM Isaac Z. Schlueter
You forgot to mention that XHR is limited by the same domain restriction, meaning that code cannot be as effectively edge-cached or dns-balanced.
While the require/module/exports thing is a pattern with a lot going for it, IMO, it's indeed too tied to synchronicity to be ideal, even outside the browser.
In a nodejs environment, where all IO is asynchronous (whether its touching the disk, or a tcp connection of some kind), it stands out to me as strange. In the code, it's just a promise with a .wait() attached, which is a bit of a bad smell.
What that means is that, even outside the browser, CommonJS programs are less than ideally suited to implementations that may want to make use of strange environments, like loading code from different servers, or distributed execution.
The isolated script env with an exports object is a great step away from the standard JS pattern of everyone pissing on the same global, but require() itself leaves a lot to be desired. JavaScript is a language which is very well suited to asynchronicity, but there's a cultural shift that needs to occur to get our specs there.
November 25, 2009 1:00 AM Kris Kowal
Thanks for this analysis. David Flanagan's CommonJS loader implementation is a bit of a red herring; it is not a viable *implementation* of a CommonJS client-side loader. In any production system, it would be necessary to wrap CommonJS modules in a "Module Transport Format" very similar to the YUI.use pattern. Caja supports CommonJS modules by compiling modules to a similar script-injectable format. Furthermore, production systems would need to inject a mini-loader like labjs or the YUI scaffolding and then would have an *opportunity*, but not the necessity, to optimize the downloading of module factory functions by initiating a preload of the transitive dependencies, possibly in bundles, possibly in an optimal packing order (large to small is a good heuristic). Narwhal does all of these things with a Jack JSGI module on the server. I'm presently using a similar deployment system built on Django at FastSoft. I'm working on a tool for Narwhal that will prepare modules for transport over a CDN by converting them to transport format, and also a tool for generating the preloader.
Anyone is welcome to join the discussion for convergence on a module transport system, on CommonJS http://wiki.commonjs.org/wiki/Modules/Transport. I have a specification proposal in the works.
November 25, 2009 6:36 PM Tom Robinson
CommonJS actually specifies that the argument passed to the "require" function must be a string literal, which allows for simple static analysis to determine the dependencies of each module before evaluating it, and *asynchronously* load them ahead of time. This is similar to how the loader in Objective-J (and several other JavaScript frameworks) works.
If a module needs to compute a module id at runtime then it should use the require.async() form, which returns a promise.
November 29, 2009 11:16 AM Luke Smith
@Kris, Tom
Thanks for the great info, guys!