The specific reason that you have to pay close attention to byte-order when handling IPv4 address and port numbers, is that the structures sockaddr_in and in_addr have data members with integer types larger than char, and whose contents are required to be in network byte order.
Even with IPv4 you can avoid worrying about the byte-order of the address part of that -- use inet_aton or inet_pton to fill the in_addr directly from a string. The latter function does IPv6 addresses too (filling an in6_addr).
When using IPv6 you still need htons for the port number, and you will need host/network conversion if you choose to access an in6_addr in chunks bigger than a byte.
As you say, if your platform doesn't have a 128 bit type then you can't access an in6_addr as a single 128-bit chunk in the way that you might have accessed in_addr as a single 32-bit chunk. But if some networking interface/implementation with a 128 bit type ever does decide to expose a 128-bit view of it, then hopefully it will also provide matching ntohX and htonX functions.