My problem was that this shared module I was trying to import was created with ng generate module. In doing so, angular automatically imports NgModule, CommonModule and adds an @NgModule with declarations and imports to the exported class. These give you access to the DOM, and web workers cannot have access to the DOM. Which is what the compile error I was getting was talking about. Even tho I wasn't really using these imports and none of these were the reason why I was importing this module into the webworker. They were there anyway and they were causing the error.
I fixed it by simply removing the 2 import statements for NgModule, CommonModule and the whole @NgModule from my shared module. Then it imported fine without any errors into the web worker.