How can I require node modules across disk partitions?

Question

Node.js Version: 14.15.0
OS: Raspbian
Scope (install, code, runtime, meta, other?): require

I have a Node program that I run on Raspberry Pi 4. I've recently started using a OTA deployment system called Mender to push updates to my code on remote RPis. Mender creates a partition system that uses two 3.5GB partitions, one as the main and the other as a rollback in the event of a failed deployment. And it has a 3rd partition /data, that is around 20GB in my case, for things that need to be persisted between updates.

I was unable to get my entire application and all of it's node module dependencies into the 3.5GB partition. So I moved the node_modules directory to the /data partition and created a symlink that points back to my project directory(home/pi/myProject). This works for module installs but when I try to require an installed module from within my project an error is thrown;

internal/modules/cjs/loader.js:883
  throw err;
  ^

Error: Cannot find module '@google-cloud/pubsub'
Require stack:
- /home/pi/myProject/pwrMngmnt.js
- /home/pi/myProject/[eval]
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
    at Function.Module._load (internal/modules/cjs/loader.js:725:27)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (/myProject/pwrMngmnt.js:3:20)
    at Module._compile (internal/modules/cjs/loader.js:1063:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Module.require (internal/modules/cjs/loader.js:952:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/pi/myProject/pwrMngmnt.js',
    '/home/pi/myProject/[eval]'
  ]
}

Is there a configuration I need to set to make this work?

score 4 · Accepted Answer · answered Dec 27 '20 at 18:09

4

Instead of doing a symlink, you could specify the NODE_PATH environment variable. Quoting the NodeJS documentation : "NODE_PATH was originally created to support loading modules from varying paths before the current module resolution algorithm was defined."

NODE_PATH is still supported and could perfectly fit your use-case IMO. Don't forget to fix file permissions if needed (using chmod and chown).

For instance:

export NODE_PATH="/data/node_modules"
node <your script>

Furthermore, you can ask npm or yarn to install modules in this directory.

With yarn:

yarn install --modules-folder /data/node_modules

With npm:

mkdir -p /data/node_modules
npm install --prefix /data

Possible related questions:

answered Dec 27 '20 at 18:09

Raphael Medaer

2,528
12
18

I tried this earlier but it didn't change the error. Do you know does `NODE_PATH` need to be set relative to my node application's directory in `home/pi/myProject`, where `home` and`data` are sibling directories? Or is setting an absolute path as you have shown correct? – Brad W Dec 27 '20 at 20:07
I did a quick test with an absolute path and it works well. I tested with "npm install --prefix /tmp/node_test" and "NODE_PATH=/tmp/node_test/node_modules". Without the NODE_PATH I got a "MODULE_NOT_FOUND" error. With the NODE_PATH, I don't have any issue. – Raphael Medaer Dec 27 '20 at 20:17
Cool. I was just reading [this](https://raspberrypi.stackexchange.com/questions/37771/setting-system-wide-path-not-working-in-etc-environment) about how to set global environment variables on Raspbian. The answer accepted recommends adding a file to `profile.d` that sets the variable. Do you know if it matters from either a Raspbian OR Node perspective what you name the file? – Brad W Dec 27 '20 at 20:20
Well, your question is not clear for me. It depends how you start your node script/daemon. Do you start it from your tty ? From a init script (e.g. systemd) ? From another tool ? I would recommend to read the following documentation: https://help.ubuntu.com/community/EnvironmentVariables Pay attention that it's an Ubuntu documentation. Some mechaisms could differ on Raspbian. I also saw this Debian wiki page: https://wiki.debian.org/EnvironmentVariables ... but it looks pretty "old"/"outdated". – Raphael Medaer Dec 27 '20 at 20:27
I need the program to run at boot/restart so I was planning on setting it up using `init.d` as described [here](https://www.instructables.com/Nodejs-App-As-a-RPI-Service-boot-at-Startup/) – Brad W Dec 27 '20 at 20:37
This tuto looks outdated, I would recommend you to use systemd as [documented on RaspberryPi website](https://www.raspberrypi.org/documentation/linux/usage/systemd.md) Please have a look in [Systemd documentation](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Environment) (or [on stack](https://serverfault.com/a/413408/225643)) to know how to set an Environment variable for a systemd service. – Raphael Medaer Dec 27 '20 at 20:42

phoenixstudio · Answer 2 · 2021-01-02T13:59:18.307

Here is the basic structure

-|- data
 |- myProject

Create two folders node_modules, one in the source and one in the destination

mkdir myProject/node_modules
mkdir data/node_modules
sudo mount --bind data/node_modules/ myProject/node_modules/

You should have now something like this

-|- data -|- node_modules 
 |- myProject -|- node_modules
               |- index.js
               |- package.json

Now myProject/node_modules is a mirror of data/node_modules This is extremely handy since you can even mount files from your network, if you want this to be permanant over restarts you can add an entry in /etc/fstab.

How can I require node modules across disk partitions?

2 Answers2

Linked