TipsAndTricks/NPM: Difference between revisions

From Yocto Project
Jump to navigationJump to search
Line 80: Line 80:
* Download module from https://github.com/otcshare/iotivity-node/archive/1.1.1-0.tar.gz
* Download module from https://github.com/otcshare/iotivity-node/archive/1.1.1-0.tar.gz
* Unpack it
* Unpack it
* Install it. This will create dependencies in the node_modules folder that we will package inot oa tarball
* Install it.  
  cd iotivity-node-master
  cd iotivity-node-master
  npm install --production
  npm install --production
  tar cvzf node_modules.tar.gz node_modules/*
  tar cvzf node_modules.tar.gz node_modules/*
* This will create dependencies in the node_modules folder that we will package into a tarball. This obviates the need for shrinkwrap and lockdwon as the dependencies are define by what is the tarball which has a checksum set in the recipe.
* Now create recipe. The following shows a recipe snippet that does not include package meta-data, licence or tarball checksum details.  
* Now create recipe. The following shows a recipe snippet that does not include package meta-data, licence or tarball checksum details.  
** The node_modules tarball you just created must go in the files sub-directory.  
** The node_modules tarball you just created must go in the files sub-directory.  
** Add RDEPENDS_${PN} = "nodejs" (this is an omission from the npm class)
** Add <tt>RDEPENDS_${PN} = "nodejs"</tt> (this is an omission from the npm class)
** SRC_URI includes the module source code (https://github.com/otcshare/iotivity-node/archive/1.1.1-0.tar.gz) and the node_module tarball (file://node_modules.tar.gz)
** <tt>SRC_URI</tt> includes the module source code (https://github.com/otcshare/iotivity-node/archive/1.1.1-0.tar.gz) and the node_module tarball (file://node_modules.tar.gz)
** inherit the npm class to do the hard work.  
** inherit the npm class to do the hard work.  
** As npm class sets ${S}, we need to override it to pick up our unpacked node module. This has to be done after 'inherit npm' directive
** As npm class sets <tt>${S}</tt>, we need to override it to pick up our unpacked node module. This has to be done after <tt>inherit npm</tt> directive
** As npm fetcher has not been used you must manually add licence information from dependent modules
** As npm fetcher has not been used you must manually add licence information from dependent modules
<pre>
<pre>
RDEPENDS_${PN} += "nodejs"
RDEPENDS_${PN} += "nodejs"


SRC_URI = "http://https://github.com/otcshare/iotivity-node/archive/1.1.1-0.tar.gz \
SRC_URI = "https://github.com/otcshare/iotivity-node/archive/1.1.1-0.tar.gz;name=iotivity_node \
           file://node_modules.tar.gz"
           file://node_modules.tar.gz;name=node_modules"


inherit npm
inherit npm

Revision as of 01:09, 19 August 2016

Background

JavaScript is becoming a leading programming language for IoT due to the popularity of Node.js [1] [2] [3]. However Node.js application packages (or modules as they are typically known) tend to have many dependencies and often are not very descriptive of what versions of these dependencies they require. Node.js modules are managed by a tool called Node Package Manager (NPM) which accesses a module registry to install dependencies. In previous versions of Yocto Node.js module recipes created the package by running npm in the do_compile task that would look something like this

SRC_URI = "https://github.com/gruntjs/grunt-cli.git"

do_compile() {
    # changing the home directory to the working directory, the .npmrc will be created in this directory
    export HOME=${WORKDIR}

    # configure cache to be in working directory
    npm set cache ${WORKDIR}/npm_cache

    # clear local cache prior to each compile
    npm cache clear

    # compile and install node modules in source directory
    npm --arch=${TARGET_ARCH} --verbose install
}

The problem with this approach is that the npm install command triggers download of dependent modules. You have to manually track down the licences of all these component and add them to the recipe. Also due the loose specification of of dependent package versioning, building with the same recipe at a later date may result in different package contents. Furthermore, as web operations are not expected in the do_compile task, proxy variables are not propagated so recipes must be extended to add configuration for handling corporate firewalls. so although this approach is functional, it's far from ideal.

In Yocto 2.1 an NPM fetcher was added to simplify the lock down the packaging of Node.js modules as well as helping you check your licensing requirements.

The fetcher is not yet documented in the bitbake manual [4], so this article will help you get the best out of it.

Creating NPM Recipes

Fetcher Syntax

The new npm fetcher uses the npm scheme, must have the registry as the path (usually registry.npmjs.org, but any registry can be used) and requires a name parameter to specify the module. Assuming recipe name and version match the module, the above recipe snippet could be replaced with the following. Note use of npm class. In future releases the REDPENDS entry will be moved into the npm class for the time being must be manually added to the recipe.

SRC_URI = "npm://registry.npmjs.org;name=${PN};version=${PV}"
inherit npm
RDEPENDS_${PN} += "nodejs"

Dependent Layers

The npm fetcher needs the native nodejs-npm package which is part of meta-oe. You must get the meta-openembdeed layer from git://git.openembedded.org/meta-openembedded and add /path/to/meta-openembedded/meta-oe to conf/bblayers.conf

Using devtool

Although npm recipes can be created manually, using devtool make the job much easier. Just run the follow for module grunt-cli, version 1.10.

devtool add "npm://registry.npmjs.org;name=grunt-cli;version=1.1.0"

Under the hood devtool runs recipetool create with the same fetch uri. Recipetool downloads each dependency, capturing licence details where possible and generates a recipe file. The recipe file is fairly simple but will contain every license that recipetool has found and include it in the LIC_FILES_CHKSUM. Many node modules have unclear licensing so you'll see "unknown" in the LICENSE field. Have a look at the modules not listed.

Recipetool will also create shrinkwrap and lockdown files for your recipe. Shrinkwrap files capture the version of all dependent modules. Many packages don't provide this so we create one on the fly. You can replace it with your own file by setting the NPM_SHRINKWRAP variable. Lockdown files contain the checksum for each module to check your users will download the same files when building with your recipe. This ensure that dependencies have not been changed and that your NPM registry is still handing out the same file.

Result will look something like this. Key entries are SRC_URI and inherit npm directive. You will have manually track down the unknown and missing licences, but apart form that recipe should be functional.

LICENSE = "ISC & Unknown & MIT"
LIC_FILES_CHKSUM = "file://node_modules/nopt/LICENSE;md5=82703a69f6d7411dde679954c2fd9dca \
                    file://node_modules/nopt/node_modules/abbrev/LICENSE;md5=82703a69f6d7411dde679954c2fd9dca \
                    ...
                    file://node_modules/findup-sync/node_modules/glob/nodeLICENSE_${PN}-findup-sync-glob-inflight-wrappy = "ISC"

SRC_URI = "npm://registry.npmjs.org;name=grunt-cli;version=${PV}"

S = "${WORKDIR}/npmpkg"

SUMMARY = "The grunt command line interface"
NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"

inherit npm

LICENSE_${PN}-findup-sync-glob-inflight-once-wrappy = "ISC"
LICENSE_${PN}-findup-sync-glob-inflight-once = "ISC"
...
LICENSE_${PN} = "MIT"

Dealing With NPM Modules Not In Registry

The npm fetcher only supports node modules that already exist in the registry, so cannot deal with modules not yet published. This feature is scheduled for development and in the meantime we have a work around.

Let's use iotivity-node as an example.

cd iotivity-node-master
npm install --production
tar cvzf node_modules.tar.gz node_modules/*
  • This will create dependencies in the node_modules folder that we will package into a tarball. This obviates the need for shrinkwrap and lockdwon as the dependencies are define by what is the tarball which has a checksum set in the recipe.
  • Now create recipe. The following shows a recipe snippet that does not include package meta-data, licence or tarball checksum details.
    • The node_modules tarball you just created must go in the files sub-directory.
    • Add RDEPENDS_${PN} = "nodejs" (this is an omission from the npm class)
    • SRC_URI includes the module source code (https://github.com/otcshare/iotivity-node/archive/1.1.1-0.tar.gz) and the node_module tarball (file://node_modules.tar.gz)
    • inherit the npm class to do the hard work.
    • As npm class sets ${S}, we need to override it to pick up our unpacked node module. This has to be done after inherit npm directive
    • As npm fetcher has not been used you must manually add licence information from dependent modules
RDEPENDS_${PN} += "nodejs"

SRC_URI = "https://github.com/otcshare/iotivity-node/archive/1.1.1-0.tar.gz;name=iotivity_node \
           file://node_modules.tar.gz;name=node_modules"

inherit npm

# npm class sets ${S} so we need to override it. This has to be done after 'inherit npm' directive
S = "${WORKDIR}/${PN}-${PV}"

Building & dependencies

Some stuff here

Examples