Historically, there was not much thought on name resolution, it seems. std::swap was designed as a customization point, but also wound up as the function that one would call in generic code to swap things. Hence, if std::swap didn't work, or was too slow, then one might have had to overload it, even if there already was a perfectly good swap to be found with ADL. This design cannot be changed without breaking or changing the meaning of existing code. Now there have been cases where the committee gladly decided to change the meaning of existing code for the sake of performance, such as implicit move semantics (e.g. when passing temporaries by value, or RVO where elision is not implemented). So this is not entirely consistent. Nonetheless, changing std::swap from a customization point to a name resolution wrapper with all the existing std::swap overloads in existence is dubious. This could absolutely  cause catastrophic bugs to be triggered in poorly written legacy code.
Another important reason is, IMO, that you cannot move this idiom to the std namespace, while maintaining its generality. E.g.:
namespace A { struct X{}; }
namespace B {
   using std::swap;
   void swap(A::X&, A::X&);
   template<typename T>
   void reverse(T (&ts)[4])
   {
       swap(ts[0], ts[3]);
       swap(ts[1], ts[2]);
   }
   void silly(A::X (&xs)[4])
   {
       reverse(xs);
   }
}
Here, silly winds up using B::swap. The reason for this is that std::swap (via using) and B::swap are both visible with the same precedence, but the latter is a better match. Now, you may argue that this is a silly example, so here's another that is a bit less contrived:
namespace types { /*...*/ }
namespace algorithms { /*including some swap implementations for types from above...*/ }
template<typename T>
void reverse(T (&ts)[4])
{
    using std::swap;
    using algorithms::swap;
    swap(ts[0], ts[3]);
    swap(ts[1], ts[2]);
}
This will use a swap function from algorithms if it is a better match than any std::swap overload, but algorithms::swap will not be found by ADL, therefore the lookup cannot happen inside std::swap, generically. An arbitrary number of namespaces could be involved here.