I think the best way to understand this is to consider what these things are like as set-theoretic sets, as opposed to as Agda Set. suppose you have A = {a,b,c}. An example of a function f : A → A is a set of pairs, let's say f = { (a,a) , (b,b) , (c,c) } that satisfies some properties that don't matter for this discussion. That is to say, f's elements are the same sort of thing that A's elements are -- they're just values, or pairs of values, nothing too "big".
Now consider the a function F : A → Set. It too is a set of pairs, but its pairs look different: F = { (a,A) , (b,Nat) , (c,Bool) } lets say. The first element of each pair is just an element of A, so it's pretty simple, but the second element of each pair is a Set! That is, the second element is itself a "big" sort of thing. So A → Set couldn't possibly be set, because if it were, then we should be able to have some G : A → Set that looks like G = { (a,G) , ... }. As soon as we can have this, we can get Russell's paradox. So we say A → Set : Set1 instead.
This also addresses the issue of whether or not Set → A should also be in Set1 instead of Set, because the functions in Set → A are just like the functions in A → Set, except that the As are on the right, and the Sets are on the left, of the pairs.