TipsAndTricks/NPM

Background

JavaScript is becoming a leading programming language for IoT due to the popularity of Node.js [1] [2] [3]. However Node.js application packages (or modules as they are typically known) tend to have many dependencies and often are not very descriptive of what versions of these dependencies they require. Node.js modules are managed by a tool called Node Package Manager (NPM) which accesses a module registry to install dependencies. In previous versions of Yocto Node.js module recipes created the package by running npm in the do_compile task that would look something like this

SRC_URI = "git://github.com/enableiot/iotkit-agent.git;protocol=git"

do_compile() {
    # changing the home directory to the working directory, the .npmrc will be created in this directory
    export HOME=${WORKDIR}

    # configure cache to be in working directory
    npm set cache ${WORKDIR}/npm_cache

    # clear local cache prior to each compile
    npm cache clear

    # compile and install node modules in source directory
    npm --arch=${TARGET_ARCH} --verbose install
}

The problem with this approach is that the npm install command triggers download of dependent modules. As web operations are not expected in the do_compile task, proxy variables are not propagated so recipes must be extended to add configuration for correct operation behind corporate firewalls. As you can see, this is about as complex are the bare recipe.

SRC_URI = "git://github.com/enableiot/iotkit-agent.git;protocol=git"

do_compile() {
    # changing the home directory to the working directory, the .npmrc will be created in this directory
    export HOME=${WORKDIR}

    # configure cache to be in working directory
    npm set cache ${WORKDIR}/npm_cache

    # clear local cache prior to each compile
    npm cache clear

    # configure proxy
    if [ -n "${http_proxy}" ]; then
        npm config set proxy ${http_proxy}
    fi
    if [ -n "${HTTP_PROXY}" ]; then
        npm config set proxy ${HTTP_PROXY}
    fi
    if [ -n "${https_proxy}" ]; then
        npm config set https-proxy ${https_proxy}
    fi
    if [ -n "${HTTPS_PROXY}" ]; then
        npm config set https-proxy ${HTTPS_PROXY}
    fi
    export ALL_PROXY=${ALL_PROXY}
    export GIT_PROXY_COMMAND=${GIT_PROXY_COMMAND}

    # compile and install node modules in source directory
    npm --arch=${TARGET_ARCH} --verbose install
}

In Yocto 2.1 an NPM fetcher was added to greatly simplify the packaging of Node.js modules as well as helping you check your licensing requirements.

New Npm Fetcher

The new npm fetcher uses the 'npm scheme, must have the registry as the path (registry.npmjs.org) and requires a name parameter to specify the module name. The above recipe snippet could be replaced with the following

SRC_URI = "npm://registry.npmjs.org;name=${PN};version=${PV}"

Note that the using the fetcher is not yet documented in the bitbake manual [4], so this article will help you get the best out of it.

Recipetool

Recipetool now allows an npm URL to be given like this:

 recipetool create "npm://registry.npmjs.org;name=grunt-cli;version=1.1.0"

Note that registry.npmjs.org is the default NPM registry but any registry URL can be used. The name and version tags should be self documenting. Behind the scene, recipetool will download each dependency and write a recipe file. The recipe file is fairly simple but will contain every license that recipetool has found and include it in the LIC_FILES_CHKSUM. Note alot of node modules have unclear licensing so "unknown" happens alot in the LICENSE field, have a look at the modules not listed.

Integrity of your package

Recipetool will also create a shrinkwrap and lockdown file for your recipe. Shrinkwrap files in npm are used to make sure that the full dependency chain of a node module is the same as the user expected. Alot of packages don't provide this so we create one on the fly, you can replace it with your own. Lockdown checks that the files recipetool downloaded are the same as the ones your users will download when using your recipe. This simply checks dependencies have not been changed and that your NPM registry is still handing out the same file.

Building & dependencies

Some stuff here