The other solution did not work for me. Following the top voted answer on how to remove submodules, I used the following command, which is more flexible, as you can specify submodules in multiple folders:
git filter-branch -f --prune-empty --tree-filter '
    for SUBMODULE_DIR in submodule-dir-1 lib/submodule-dir-2 lib/submodule-dir-3
    do
        if [ -d $SUBMODULE_DIR ]; then
            git submodule deinit -f $SUBMODULE_DIR
            git rm -rf .git/modules/$SUBMODULE_DIR
            git rm -f $SUBMODULE_DIR
        fi
    done
    git rm -f --ignore-unmatch .gitmodules' HEAD
Effectively we are checking the list of submodule folders for each commit. If the submodule folder exists, we completely remove the submodule. Technically that alone is sufficient, be we also want to get rid of the .gitmodules file. If you only want to delete specific submodules, that line might need some additional work.