Your question is actually quite broad - however, I'll try to list as much as possible.
Lets say you're building a project from scratch and needed to include vuejs, jquery, fontawesome but only need to include a couple of files.
The issue you're hitting here is module dependency with respect to npm modules. (and there are many different tools that you can use to manage versions on your library dependencies as well as ensuring they are included into your project, i.e. package managers).
Ok - now from here, you may say to yourself 
but I only need say, one icon from fontawesome in your final build (dist) and I don't want to commit all my modules into source control
Again, this is where you omit node_modules and other dependent libraries from source control (i.e. include node_modules your .gitignore)
To reiterate
- You can install the required library, 
- add node_modulesto .gitignore ,
- bundle those libraries into a vendor single file to be consumed by your users (can be via browserify/webpack/rollup/gulp/grunt/yarnetc).
- generate bundle within npm script
Now you may ask further - 
Why would I use any of those tools? - they're distracting me from simply copy/pasting vendor libaries into my source control.
Build tools were created to
- streamline the developer pipeline so that you DONT have to copy/paste vendor libaries into a vendor folder.
- ensures that all files are bundled to the client automatically 
- allows you to keep track/restrict library version updates/ when required via package.json
- allows you to add other build steps (such as minification, md5hash versioning, compression, code splitting, asset management to name a few).
Now lets break down the original question here:
- How to ensure other developers get everything they need when cloning the repository
- how do I ensure I can provide only the necessary files to the end user (if I only use specific parts of vendor libaries?)
1. How to ensure developers get what they need
Again, to reiterate above, adding devDependancies and .gitignoring allows you to only add the necessary files to your project.
2. How can I ensure clients get what they need without bloating request files?
This is where build tools such as webpack, browserify, gulp, grunt, rollup, attempt to achieve. This is because with some libraries that exceed in file size of 200kb minified, you may need to separate these files into different client requests (and as such, not demand the user to request one large file, which was symtomatic of browserify projects).
The second technique you will need to be aware of, is with specific libraries, you can use import mdn link where you can require one function/class from a dependant library (which further reduces file size).
Another technique is using less import statements (which can extract less functions/styles similar to above, but this isn't currently supported in SCSS). for SCSS, you're basically left with copy/pasting the necessary styles into your base scss and that'll save you space as well.
EDIT
How to create a bundle from npm install libaries
From the comments you've mentioned above (about not wanting to include a tool into your workflow, there's no point outlining any one particular strategy - you can find answers/tutorials online about how to setup gulp/browserify/webpack for your particular needs).
However, As you are currently using yarn - I'll go into details about that.
Firstly, yarn is a package manager (like npm). All it does with the --modules-folder is install the package into the specified folder, that's all. So we don't really care about that (since it's doing the same thing as npm). (i.e. your vendor folder is the same as node_modules in many respects).
We could use 
- webpack
- gulp
- grunt
- rollup
- browserify
- brunch
(All build tools that essentially allow you to bundle all those packages into a single entry point to be packaged to the client).
I won't go into how, because that is a process you can find online, and from the above comments, I can tell you don't particularly care either.
What you are looking for is a zero-config javascript build tool. (Extremely out of the scope of your original question, and i'll only answer that in a separate Q&A).