There are a number of problems here
1) In helper buf L1 L2 = buf, the pattern buf L1 L2 would match all possible inputs, rendering your next clause (once debugged) redundant. In context, I think that you meant helper buf [] [] = buf, but then you would run into problems of non-exhaustive matching in the case of lists of unequal sizes. The simplest fix would be to move the second clause (the one with x::rest1) into the top line and then have a second pattern to catch the cases in which at least one of the lists are empty.
2) [xs::rest] is a pattern which matches a list of 1 item where the item is a nonempty list. That isn't your attention. You need to use (,) rather than [,].
3) reverse should be rev.
Making these changes, your definition becomes:
fun zipTail L1 L2 = 
let 
    fun helper buf (x::rest1) (y::rest2) = helper ((x,y)::buf) rest1 rest2
      | helper buf rest1 rest2 = buf
in
    rev (helper [] L1 L2)
end;
Which works as intended.
The error message itself is a bit hard to understand, but you can think of it like this. In
helper buf [x::rest1] [y::rest2] = helper ((x,y)::buf) rest1 rest2
the things in the brackets on the left hand side are lists of lists. So their type would be 'a list list where 'a is the type of x. In x::rest1 the type of rest1 would have to be 'a list Since rest1 also appears on the other side of the equals sign in the same position as [x::rest1] then the type of rest1 would have to be the same as the type of [x::rest1], which is 'a list list. Thus rest1 must be both 'a list and 'a list list, which is impossible.
The circularity comes from if you attempt to make sense of 'a list list = 'a list, you would need a type 'a with 'a = 'a list. This would be a type whose values consists of a list of values of the same type, and the values of the items in that list would have to themselves be lists of elements of the same type ... It is a viscous circle which never ends.