Delete your remote tracking branches
Individually run git branch -rd origin/branchIDontWant for each branch, or if there are many you can blow them all away with git for-each-ref --format "%(refname)" refs/remotes/origin | xargs -I% git update-ref -d "%".
Fetch desired branches
You could only fetch (or pull) the branches that you want (git fetch origin branchIWant) going forward, or configure git fetch 
to only fetch desired branches from the remote. 
To configure git fetch to only pull branches that you desire, you'll need to update your configs for the remote you're pulling from. This is described in more detail here: https://git-scm.com/book/en/v2/Git-Internals-The-Refspec. You will want to remove the default refspec fetch = +refs/heads/*:refs/remotes/origin/* from your .git/config for the repository. Then, you'll add a refspec fetch = +refs/heads/branchIWant:refs/remotes/origin/branchIWant for each branch you want fetched (branchIWant).
Keep in mind I'm assuming you're only using a single remote called origin, repeat above steps for more remotes.