I am trying to build my own load balancer on C++. But not sure whether all packets(both requests and responses) between client and server have to go through load balancer.
This depends on the setup in most cases all traffic would go through the load balancer in both directions as this is the "safest" and easiest configuration. There are also advantages from a network security POV as you can better isolate servers from end users.
For a tcp session to work the source and Target IP addresses and ports need to be consistent per tcp stream. While it is possible to have multiple systems respond on the same IP, you are asking for pain and difficulty when things go wrong.
If you want to do this without having the load balancer in the critical path you may want to do this at a DNS level rather then a load balancer level.
Casting my mind back to the distant past I recall LVS (http://www.linuxvirtualserver.org/how.html) which has 3 different mechanisms for load balancing and would be useful reading. I have not used this for a very long time, and I am not convinced it's better then Apache or equivalent load balancing solutions though. [The Internet has moved on and this document is 30 years old. Egress filtering and security in depth we're not really a thing then, and computers were much, much slower. Also, you can't use it to strip https -> http, very useful today]
in case of normal http response it would be logical that all traffic (may) go north bound via load balancer,
but , if its a stream of data like video, its logical that the header is only passed and the rest of data is directed straight to client.
would be glad to know how it happens in HAproxy or Nginx. The wowza load balancer works in later format ( has listerners and traffic directors)
