TipsAndTricks/NPM: Difference between revisions

From Yocto Project
Jump to navigationJump to search
(→‎Yocto 2.1.1 (krogoth) Restrictions: Remove this obsolete section)
 
(15 intermediate revisions by 4 users not shown)
Line 25: Line 25:
The problem with this approach is that the '''npm install''' command triggers download of dependent modules. You have to manually track down the licences of all these component and add them to the recipe. Also due the loose specification of dependent package versioning, building with the same recipe at a later date may result in different package contents.  Furthermore, as web operations are not expected in the do_compile task, proxy variables are not propagated so recipes must be extended to add configuration for handling corporate firewalls. In summary although this approach is functional, it's far from ideal.
The problem with this approach is that the '''npm install''' command triggers download of dependent modules. You have to manually track down the licences of all these component and add them to the recipe. Also due the loose specification of dependent package versioning, building with the same recipe at a later date may result in different package contents.  Furthermore, as web operations are not expected in the do_compile task, proxy variables are not propagated so recipes must be extended to add configuration for handling corporate firewalls. In summary although this approach is functional, it's far from ideal.


NPM fetcher and recipetool support was added was added in Yocto 2.1 and improved in 2.2. This simplifies the locking down the packaging of Node.js modules as well as helping you check your licensing requirements.
NPM fetcher and recipetool support was added in Yocto 2.1 and improved in 2.2. This simplifies the locking down the packaging of Node.js modules as well as helping you check your licensing requirements.


Aspects of these features are not yet documented in the bitbake manual [https://bugzilla.yoctoproject.org/show_bug.cgi?id=10098], so this article will help you get the best out of it.
Aspects of these features are not yet documented in the bitbake manual [https://bugzilla.yoctoproject.org/show_bug.cgi?id=10098], so this article will help you get the best out of it.


== Creating NPM Packages ==
== Creating NPM Packages ==
=== Introduction and Caveats ===
Although npm recipes can be created manually, using [http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#using-devtool-in-your-workflow '''devtool'''] make the job much easier. There are two options for creating recipe, using the NPM registry and NPM project source code. The registry approach is slightly simpler, but you're more likely to use the project approach as you don't have to publish your module in the registry.


There are a few requirements and caveats
Documented on [https://docs.yoctoproject.org/dev-manual/common-tasks.html#creating-node-package-manager-npm-packages the Yocto Project manual]
* You need to be familiar with [http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#using-devtool-in-your-workflow devtool] to get the best out of this article
* The npm host tools need the native nodejs-npm package which is part of meta-oe. You must get the meta-openembedded layer from git://git.openembedded.org/meta-openembedded and add /path/to/meta-openembedded/meta-oe to conf/bblayers.conf.
* Devtool cannot detect native libraries in module dependencies, you you'll need to manually add packages to recipe
* At deployment time devtool cannot determine which dependent packages are missing on the target (e.g. the node runtime, <tt>nodejs</tt>), so you have to do that manually.
* Although npm may not be needed to run your node package, it is useful to have on the target. Package name is <tt>nodejs-npm</tt>
 
=== Registry Modules ===
In this example, we'll use the cute-files module that is file browser web app. First we'll use the registry approach. Note that you need to know the module version.
devtool add "npm://registry.npmjs.org;name=cute-files;version=1.0.2"
Under the hood devtool runs '''recipetool create''' with the same fetch uri. Recipetool downloads each dependency, capturing licence details where possible and generates a recipe file. The recipe file is fairly simple but will contain every license that recipetool has found and include it in the <tt>LIC_FILES_CHKSUM</tt>. Many node modules have unclear licensing so you'll see "unknown" in the LICENSE field. Have a look at the modules not listed.
 
Recipetool will also create ''shrinkwrap'' and ''lockdown'' files for your recipe. Shrinkwrap files capture the version of all dependent modules. Many packages don't provide this so we create one on the fly. You can replace it with your own file by setting the <tt>NPM_SHRINKWRAP</tt> variable. Lockdown files contain the checksum for each module to check your users will download the same files when building with your recipe. This ensure that dependencies have not been changed and that your NPM registry is still handing out the same file.
 
You can tool a look at the generate recipe with
devtool edit-recipe cute-files
There are three key points to note
* SRC_URI uses the <tt>npm</tt> scheme so that the npm fetcher is used
* Recipetool has collected all the licence info (if it can't get a licence for specific sub-modules, you see their names listed in the comments)
* <tt>inherit npm</tt> causes the npm class package up all the modules
<pre>
SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
LICENSE = "BSD-3-Clause & Unknown & MIT & ISC"
LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
                    file://node_modules/content-disposition/LICENSE;md5=c6e0ce1e688c5ff16db06b7259e9cd20 \
                    file://node_modules/express/LICENSE;md5=5513c00a5c36cd361da863dd9aa8875d \
                    ...
 
SRC_URI = "npm://registry.npmjs.org;name=cute-files;version=${PV}"
NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"
inherit npm
# Must be set after inherit npm since that itself sets S
S = "${WORKDIR}/npmpkg"
 
LICENSE_${PN}-content-disposition = "MIT"
...
LICENSE_${PN}-express = "MIT"
LICENSE_${PN} = "MIT"
</pre>
You can run run <tt>devtool build cute-files</tt> to build package. Remember that <tt>nodejs</tt> must be installed on the target before deploying.
 
Once installed on target, test app as follows (note that you can't simply run cute-files as if you'd done an 'npm install'. This is a [https://bugzilla.yoctoproject.org/show_bug.cgi?id=10460 known issue]).
cd /usr/lib/node_modules/cute-files
node cute-files.js
Then on a browser go to http://target-ip:3000 and you'll see the following
:[[File:npm-cute-files-screenshot.png|640px]]


=== NPM Projects in Development ===
=== NPM Projects in Development ===
Although it is useful to package modules already in the registry, adding node.js projects in development is a more common developer use case. The is very similar to he "registry" approach but instead we give devtool a URL to the source code. To create the cute-files example above from source code would we done as follows
Although it is useful to package modules already in the registry, adding node.js projects in development is a more common developer use case. The is very similar to the "registry" approach, but instead we give devtool a URL to the source code. To create the cute-files example above from source code we would have done as follows:
  devtool add https://github.com/martinaglv/cute-files.git
  devtool add https://github.com/martinaglv/cute-files.git
If you look at the genenrated recipe, it will be very similar, but the SRC_URI will look like this
If you look at the generated recipe, it will be very similar, but the SRC_URI will look like this:
  SRC_URI = "git://github.com/martinaglv/cute-files.git;protocol=https \
  SRC_URI = "git://github.com/martinaglv/cute-files.git;protocol=https \
           npm://registry.npmjs.org;name=commander;version=2.9.0;subdir=node_modules/commander \
           npm://registry.npmjs.org;name=commander;version=2.9.0;subdir=node_modules/commander \
Line 92: Line 44:
You can see that the main module is taken from git repo and dependents obtained from registry. Other than that, recipe is fairly similar and can be built and deployed as above.
You can see that the main module is taken from git repo and dependents obtained from registry. Other than that, recipe is fairly similar and can be built and deployed as above.


== Yocto 2.1.1 (krogoth) Restrictions ==
== Yocto 3.1 (dunfell) update ==
* The npm fetcher only supports node modules that are in the registry, so cannot get modules not yet published.  
 
* The npm class does not add node as a runtime dependency so you have to add this to the recipe (will be fixed in 2.1.2)
The [https://docs.yoctoproject.org/3.2.3/ref-manual/migration-3.1.html#npm-fetcher-changes npm fetch system] has been [https://blog.savoirfairelinux.com/en-ca/2020/dealing-with-angular-npm-dependencies-on-embedded-systems/ refactored] for the [https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#creating-node-package-manager-npm-packages dunfell release].
 
Importing a recipe that uses npm from zeus to dunfell it is not an straightforward process. Syntax and fetch has changed, so you might encounter errors like the following:


=== Non-Registry Workaround ===
Here is a workaround for packaginfg node projects not in the registry. We'll use cute-files again as the example.
* Make sure you in Yocto build environment and run the following to get into a shell that has the correct version of npm
bitbake nodejs-native -c devshell
In the devshell, run the following
cd /path/to/tmp/folder
wget https://github.com/martinaglv/cute-files/archive/master.zip
unzip master.zip
cd cute-files-master
npm install --production
tar cvzf node_modules.tar.gz node_modules/*
* This will create dependencies in the node_modules folder that we will package into a tarball. This obviates the need for shrinkwrap and lockdown as the dependencies are defined by what is in the tarball. And the tarball has a checksum set in the recipe.
* Now create recipe. The following shows a recipe snippet that does not include package meta-data, licence or tarball checksum details.
** The node_modules tarball you just created must go in the files sub-directory.
** Add <tt>RDEPENDS_${PN} = "nodejs"</tt> (this is an omission from the npm class)
** <tt>SRC_URI</tt> includes the module source code and the node_module tarball you just created (file://node_modules.tar.gz)
** inherit the npm class to do the hard work.
** As npm class sets <tt>${S}</tt>, we need to override it to pick up our unpacked node module. This has to be done after <tt>inherit npm</tt> directive
** As npm fetcher has not been used you must manually add licence information from dependent modules
<pre>
<pre>
SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network"
ERROR: cute-files-1.0.2-r0 do_fetch: URL: 'npm://registry.npmjs.org/;name=cute-files;version=1.0.2' is missing the required parameter 'Parameter 'package' required'
</pre>


LICENSE = "MIT"
In this case the error is related to the registry syntax. When using the registry url (from devtool or in your recipe) from dunfell release, the package name is specified by the "package" parameter instead of "name". Other errors might occur, so importing npm recipes to dunfell release might require a migration process.
LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca"


# Add missing RDEPENDS (should be implemented by npm class)
=== Migrating NPM recipes ===
RDEPENDS_${PN} += "nodejs"


SRC_URI = "git://github.com/martinaglv/cute-files.git;protocol=https \
1. Regenerate the shrinkwrap file
          file://node_modules.tar.gz;name=node_modules"
SRC_URI[md5sum] = "e1c2e3a31fdd5fc737dd7329e27e8d9a"
SRC_URI[sha256sum] = "cc3bdb2a0c2cc313c0d02141a02ac828d24de37fae35906295880be5ce8c2742"
SRCREV = "98fe76448b8367adf206de6809b4adb7189b05ee"


inherit npm
As described above, the ''devtool add'' offers the possibility to create recipes and shrinkwrap files for npm applications (either from registry, git, local sources...).


# npm class sets ${S} so we need to override it. This has to be done after 'inherit npm' directive
You can create an updated shrinkwrap for your application with:
S = "${WORKDIR}/git"


# Copy dependent modules into ${S} so they will be found during compile task.
<pre>
do_compile_prepend () {
devtool add <app>
    cp -a ${WORKDIR}/node_modules ${S}/
</pre>
}


# You must go through dependent modules extract licences and add them here.
where ''<app>'' refers to the npm registry, git repository or local sources to your application.
# Example below is from 'freeport' module
 
#LIC_FILES_CHKSUM += "file://node-modules/freeport/license;md5=ee9ea0a30e576793053e192d4608daba"
Once the command has been successfully executed you can overwrite your previous shrinkwrap with the content of ''build/workspace/recipes/<app>/<app>/npm-shrinkwrap.json''.
#LICENCE_${PN}-freeport = "Aapche-2.0"
 
2. Clean workspace
 
In order to avoid duplicated recipes you should remove the newly generated one:
 
<pre>
rm -rf build/workspace/recipes/<app> build/workspace/appends/<app> build/workspace/sources/<app>
</pre>
</pre>


3. Adapt old recipe
* Update ''SRC_URI'' variable: A common use case for npm developers is to integrate a non-registered application that has dependencies to fetch from the registry. As described on the [https://www.yoctoproject.org/docs/3.0.4/mega-manual/mega-manual.html#creating-node-package-manager-npm-packages documentation], until zeus release, this recipe would describe the sources to fetch with the following syntax:
<pre>
SRC_URI = "git://github.com/martinaglv/cute-files.git;protocol=https \
              npm://registry.npmjs.org;name=commander;version=2.9.0;subdir=node_modules/commander \
              npm://registry.npmjs.org;name=express;version=4.14.0;subdir=node_modules/express \
              npm://registry.npmjs.org;name=content-disposition;version=0.3.0;subdir=node_modules/content-disposition \
              "
</pre>
where the first line represents the repository for the main package and the other three the registry dependencies.
However, with the refactoring made on the [https://docs.yoctoproject.org/3.2.3/ref-manual/migration-3.1.html#npm-fetcher-changes dunfell release], the fetcher is capable of getting all the dependencies from a single shrinkwrap file (which is very convenient for large applications). So the ''SRC_URI'' of your recipe can be left as:
<pre>
SRC_URI = " \
    git://github.com/tutorialzine/cute-files.git;protocol=https \
    npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
    "
</pre>
Note that the syntax uses ''npmsw://'' instead of ''npm://''.
* Remove obsolete variables: Another change that might be necessary is to remove the obsolete variables ''NPM_SHRINKWRAP'' and ''NPM_LOCKDOWN'' from your recipe.
<pre>
NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
NPM_LOCKDOWN := "${THISDIR}/${PN}/package-lock.json"
</pre>


Set NODE_PATH
* Remove ''package-lock.json'': The file ''package-lock.json'' can also be removed from your recipe folder.

Latest revision as of 13:27, 22 June 2022

Background

JavaScript is becoming a leading programming language for IoT due to the popularity of Node.js [1] [2] [3]. However Node.js application packages (or modules as they are typically known) tend to have many dependencies and often are not very descriptive of what versions of these dependencies they require. Node.js modules are managed by a tool called Node Package Manager (NPM) which accesses a module registry to install dependencies. In previous versions of Yocto Node.js module recipes created the package by running npm in the do_compile task that would look something like this

SRC_URI = "https://github.com/gruntjs/grunt-cli.git"

do_compile() {
    # changing the home directory to the working directory, the .npmrc will be created in this directory
    export HOME=${WORKDIR}

    # configure cache to be in working directory
    npm set cache ${WORKDIR}/npm_cache

    # clear local cache prior to each compile
    npm cache clear

    # compile and install node modules in source directory
    npm --arch=${TARGET_ARCH} --verbose install
}

The problem with this approach is that the npm install command triggers download of dependent modules. You have to manually track down the licences of all these component and add them to the recipe. Also due the loose specification of dependent package versioning, building with the same recipe at a later date may result in different package contents. Furthermore, as web operations are not expected in the do_compile task, proxy variables are not propagated so recipes must be extended to add configuration for handling corporate firewalls. In summary although this approach is functional, it's far from ideal.

NPM fetcher and recipetool support was added in Yocto 2.1 and improved in 2.2. This simplifies the locking down the packaging of Node.js modules as well as helping you check your licensing requirements.

Aspects of these features are not yet documented in the bitbake manual [4], so this article will help you get the best out of it.

Creating NPM Packages

Documented on the Yocto Project manual

NPM Projects in Development

Although it is useful to package modules already in the registry, adding node.js projects in development is a more common developer use case. The is very similar to the "registry" approach, but instead we give devtool a URL to the source code. To create the cute-files example above from source code we would have done as follows:

devtool add https://github.com/martinaglv/cute-files.git

If you look at the generated recipe, it will be very similar, but the SRC_URI will look like this:

SRC_URI = "git://github.com/martinaglv/cute-files.git;protocol=https \
          npm://registry.npmjs.org;name=commander;version=2.9.0;subdir=node_modules/commander \
          npm://registry.npmjs.org;name=express;version=4.14.0;subdir=node_modules/express \
          npm://registry.npmjs.org;name=content-disposition;version=0.3.0;subdir=node_modules/content-disposition \
          "

You can see that the main module is taken from git repo and dependents obtained from registry. Other than that, recipe is fairly similar and can be built and deployed as above.

Yocto 3.1 (dunfell) update

The npm fetch system has been refactored for the dunfell release.

Importing a recipe that uses npm from zeus to dunfell it is not an straightforward process. Syntax and fetch has changed, so you might encounter errors like the following:

ERROR: cute-files-1.0.2-r0 do_fetch: URL: 'npm://registry.npmjs.org/;name=cute-files;version=1.0.2' is missing the required parameter 'Parameter 'package' required'

In this case the error is related to the registry syntax. When using the registry url (from devtool or in your recipe) from dunfell release, the package name is specified by the "package" parameter instead of "name". Other errors might occur, so importing npm recipes to dunfell release might require a migration process.

Migrating NPM recipes

1. Regenerate the shrinkwrap file

As described above, the devtool add offers the possibility to create recipes and shrinkwrap files for npm applications (either from registry, git, local sources...).

You can create an updated shrinkwrap for your application with:

devtool add <app>

where <app> refers to the npm registry, git repository or local sources to your application.

Once the command has been successfully executed you can overwrite your previous shrinkwrap with the content of build/workspace/recipes/<app>/<app>/npm-shrinkwrap.json.

2. Clean workspace

In order to avoid duplicated recipes you should remove the newly generated one:

rm -rf build/workspace/recipes/<app> build/workspace/appends/<app> build/workspace/sources/<app>

3. Adapt old recipe

  • Update SRC_URI variable: A common use case for npm developers is to integrate a non-registered application that has dependencies to fetch from the registry. As described on the documentation, until zeus release, this recipe would describe the sources to fetch with the following syntax:
 SRC_URI = "git://github.com/martinaglv/cute-files.git;protocol=https \
               npm://registry.npmjs.org;name=commander;version=2.9.0;subdir=node_modules/commander \
               npm://registry.npmjs.org;name=express;version=4.14.0;subdir=node_modules/express \
               npm://registry.npmjs.org;name=content-disposition;version=0.3.0;subdir=node_modules/content-disposition \
               "

where the first line represents the repository for the main package and the other three the registry dependencies.

However, with the refactoring made on the dunfell release, the fetcher is capable of getting all the dependencies from a single shrinkwrap file (which is very convenient for large applications). So the SRC_URI of your recipe can be left as:

SRC_URI = " \
    git://github.com/tutorialzine/cute-files.git;protocol=https \
    npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
    " 

Note that the syntax uses npmsw:// instead of npm://.

  • Remove obsolete variables: Another change that might be necessary is to remove the obsolete variables NPM_SHRINKWRAP and NPM_LOCKDOWN from your recipe.
NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
NPM_LOCKDOWN := "${THISDIR}/${PN}/package-lock.json"
  • Remove package-lock.json: The file package-lock.json can also be removed from your recipe folder.