I’m a front-end developer, and I keep my front-end projects in OneDrive. Due to how Node.js organizes dependencies, each project has a node_modules directory with tens of megabytes of tiny files which take forever to sync.
What can I do to avoid synchronizing node_modules directories while keeping the projects in OneDrive?
I know this is an older question, but I ran into the same issue. The solution I came across was to first create an empty node_modules folder and then sync that with the cloud.
Once the empty folder was synced, I then went into
OneDrive settings → Choose folders
and deselected the node_modules folder.
Then when you run npm install in the project root folder, OneDrive will detect a conflict with the contents of the folder in the cloud. It will not sync.
The downside of this approach is that OneDrive produces sync errors, but it seemed to work well for my needs.
OneDrive doesn’t allow to exclude directories by name, and that’s something that won’t change soon:
[OneDrive’s representative response on a UserVoice request:] Not right now
Some people recommend to uncheck directories you don’t want to sync in OneDrive settings, but that became impossible with OneDrive Files on-demand.
However, you can solve this by tweaking things on the Node.js’ side. Make the node_modules directory a file symlink to a different place:
# Open any place outside OneDrive
cd D:\node_dependencies
# Make a symlink target. After linking, node_modules for your project
# will be installed here
mkdir node_modules_for_my_project
# Open the project directory
cd <the project directory>
# Make a *file* link from node_modules to the newly created directory.
# You will need to delete existing node_modules directory if it exists
cmd /C "mklink node_modules D:\node_dependencies\node_modules_for_my_project"
The important piece here is that you create a file symlink, not directory one. OneDrive won’t recognize and sync this symlink, whereas Node.js will work with it as intended:
Tested with OneDrive v17.3.7101.1018 and OneDrive Files on-demand enabled.
This is not a universal solution. A drawback of this is that Explorer, Powershell and other tools won’t recognize node_modules as a directory:
However, Node.js-based code editors will read it just fine:
I've found that the best solution is creating a 'local' folder outside the OneDrive folder where the project will be saved and any modules will be installed. This is the actual project.
To sync it to OneDrive I manually create a 'parallel' folder and copy only the content I want using 'RoboCopy'. You can then schedule this task.
The files & folders names/wildcards to ignore were copied from the .gitignore file
The solution is based on a few articles:
Schedule RoboCopy task and simple examples
RoboCopy documentation by Microsoft
you can also use GUI to create the .bat task file to be scheduled. GUI by microsoft & ChoEazyCopy GUI which is much better
For Typescript users out there, add node_modules to your path to avoid module resolution issues.
In tsconfig.json add :
"paths": {
[... some paths ...],
"*": [
"node_modules/*"
]
}
Edit: Still have some issues. I can build the project but ts-node and nodemon fire Cannot find module errors. Well, this is still an issue then :(
Since OneDrive lacks folder filtering (and probably always will) existing solutions become workarounds where you end up creating duplicate folders and then have to manage filtered syncing between these local folders. This just wastes your local disk space and complicates the solution. Other hacks using symlinks and junctions won't scale well when you have hundreds of NodeJS projects or subprojects (as I do) and just make for more potential sources of errors and time waste. For backups, you need reliability and thus simplicity so IMHO, without folder filtering OneDrive is just the wrong tool for the job.
An alternative is to use a smarter cloud backup tool like rclone to perform your cloud file synchronization. It's open source, MIT licensed and totally free. I have NO affiliation with the rclone project.
rclone supports over 70 different storage providers including OneDrive with rich file and folder filtering rules so setting up a filter to exclude node_module folders (or any other file/folder pattern) is fairly straight forward.
OneDrive's background syncing can ruin your day as a software developer since you're running compilers or other code gen tools that compete with OneDrive that wants to lock files and folders. With a tool like rclone you're in control of when your files get synced so you can safely avoid these file and folder locking issues. I prefer to run a sync once I've reached a logical checkpoint in my project work so my backups represent a semi-stable state.
You manage your own encryption keys and can enable many other features such as file compression. You can even union together multiple local and/or cloud file systems to present as one. Make sure you've backed up your encryption keys!
rclone is supported on Windows, macOS, linux and FreeBSD although be aware of the file name limitations for OneDrive.
The rclone community has a support forum if you have any related questions.
After reading these proposed options, I ended up deciding to put everything outside of OneDrive and create a script with Robocopy to backup to OneDrive whenever there was a change in the projects folder.
Something like:
robocopy SOURCE DEST /mir /xd node_modules
I use a development folder that OneDrive doesn't back up and a backup folder OneDrive does back up and copy the development folder to the backup folder with a batch file. The following is essentially an example / step-by-step of @RanST's answer.
@ECHO ON
ECHO Copy started.
robocopy "{{SOURCE}}\PROJECTS" "{{DESTINATION}}\PROJECTS" /xd node_modules .serverless /s /z /r:3 /w:10 /tbd /np /ts
ECHO Copy completed!
PAUSE
/xd = exclude. Simply add the folder names to exclude.
/s = include subfolders exclude empty directories.
/z = retryable. /r:3 = 3 retries. /w:10 = wait 10 secs between retries.
/tbd = wait for a share names if they caused a retry error
/np = don't display progress
/ts = add a timestamp to the log
The Microsoft Documentation explains all the options.
I wanted a backup that didn't need to be updated every new project, so I chose to back up a parent folder and maintain the file structure when I created the development specific folders.
Main synced folder
/MY_WHOLE_BACKUP/.../PROJECTS/clients/client1/parentProject/subProject/root/myApp/app.js
DON'T simplify the path
/MY_LOCAL_FILES/.../PROJECTS/client1/myApp/app.js
DO keep it all
/MY_LOCAL_FILES/.../PROJECTS/clients/client1/parentProject/subProject/root/myApp/app.js
Search for and open the Task Scheduler in Windows.
Click Create Task....
Name the task.
Click the Triggers tab.
Click the New... button.
Adjust the schedule to your preference.
Click OK.
Click the Actions tab.
Click the New... button.
Browse for your batch file.
Click OK.
BOOM!
Test and configure as needed. I hope this helps.
In 2024, if you're still encountering issues and previous solutions haven't worked, here's an alternative method:
Move your source files to a local folder that syncs with the cloud, such as C:\onedrive\mynodeproject.
Select a different folder for storing your node_modules. For example, you could use C:\temp\myproject.
Open a command prompt as an administrator. (Refer to additional notes if you lack admin privileges.)
Create the new storage folder by executing:
mkdir C:\temp\myproject
Establish a junction point named "src" at this location, pointing to your source files:
mklink /J C:\temp\myproject\src C:\onedrive\mynodeproject
Create a symbolic link for package.json to ensure it points to your original package.json in the source directory:
mklink C:\temp\myproject\package.json C:\onedrive\mynodeproject\package.json
If a node_modules directory already exists, relocate its contents to C:\temp\myproject\node_modules.
Attempt to install your modules using npm. For instance:
npm install SomeLib
To start your application, instead of using the current directory (.), specify the path to your source files. For example:
node src/index.js
or simply:
node src
Please ensure to replace placeholder values like SomeLib with the actual names of the libraries you intend to install.
Note on Administrative Privileges:
Important Consideration Regarding package.json: If you choose not to sync your package.json from the source by omitting the symbolic link creation, be aware that npm will generate a new package.json file in your temporary local folder (e.g., C:\temp\myproject). This may lead to discrepancies between your project's dependencies and scripts in your source directory and those in the temporary directory. It's crucial to ensure consistency in your project setup to avoid potential issues during development and deployment.
Note on Syncing package.json:
If you prefer not to have your package.json file automatically sync with every change, you can opt to only create a junction point for your project directory. This approach bypasses the need for a symbolic link to package.json, thus avoiding the necessity for administrative privileges if Developer Mode is enabled and configured correctly.
Adapting to Path Changes and Invocation Methods: Utilizing this method may necessitate adjustments to your project's paths and the way certain calls are invoked. For instance, if you are using Electron or similar frameworks, you might need to modify the package.json to correct paths or establish new commands to accommodate the altered directory structure. This approach is not a one-size-fits-all solution but rather an alternative strategy to manage your project files in the cloud. It serves as a workaround until cloud service providers offer more flexible options for excluding specific files and folders from synchronization according to user preferences.
I have found that using dev containers can fix this issue if you can put the time into creating your environment.
Just delete the files or folders in OneDrive that you don't want to sync , they're still there in your local disk.
