You removed one pair of parentheses and added another two.
The parentheses in (f a) :: (map_alternate g f rest) make no difference. They can be omitted.
The parentheses in
fun map_alternate (f, g, lst) =
case lst of
[] => []
| a::b => f a :: map_alternate (g, f, lst)
actually carries the a meaning: This function takes one argument, a 3-tuple, pattern matches each three components and eventually calls itself with a modification of that 3-tuple (with the 1st and 2nd elements interchanged).
In contrast, the function
fun map_alternate f g lst =
case lst of
[] => []
| a::b => f a :: map_alternate g f rest
carries a different meaning. It is equivalent to
val rec map_alternate = fn f => fn g => fn lst =>
case lst of
[] => []
| a::b => f a :: map_alternate g f rest
and is a function that takes an argument f and returns a function that takes an argument g and returns a function that takes an argument lst and returns case lst of .... That is, a curried version of a three-argument function rather than a tupled. Functions returning functions involve more ->s in the type signature.