I assume that you know what you are doing, that you are familiar with git concepts and that you are forced by circumstances (upper management or whatever) to do this. It is very against git ways.
That said...
Answer
src in master and src in testbranch contains different content.
Then you should not have them in the same directory.
Make your directory structure like this:
core
src_master
src_testbranch
src --> src_master
src is then a symlink pointing to either src_master or src_testbranch depending on which branch you are in.
Now everything will work fine. The two src_... directories will show up in both branches, but you can just ignore that. You will still just work with src.
It is up to you whether you put the src symlink in .gitignore or not (depends on what you actually want to achieve with all this). Either way it does not matter much.
N.B.: you do not need to call the src_... directories after their branch names, rather name them by whatever is actually in there. You can also put them wherever else, like dist/... or vendor/....