If you want the best explanation on how the ViewHolder works, check out Romain Guy's Google I/O 2009 talk in youtube , specially the first 15 minutes.
In short, the Adapter functions as a link between the underlying data and the ViewGroup. It will render as many Views as required to fill the screen. Upon scrolling or any other event that pushes a View is out of the screen, the Adapter will reuse that View, filled with the correct data, to be rendered at the screen. 
The getView(int pos, View view, ViewGroup parent) method will use the right View at any time, regardless of your layout. I do not know the internals of this, but I'm sure you can browse the source code for any adapter (such as ArrayAdapter.java) if you're interested.
The ViewHolder just keeps a pointer to the Views as obtained by view.findViewById(int id). It is the Adapter responsibility to return the right data corresponding to any position. 
Slides 11 to 13 of Romain's presentation will make it a lot more clear than anything I can write.