The contains method is not necessarily very heavy (time consuming). For example if you use a HashSet it is rather fast. For a HashSet it will calculate the hashCode of the object, and only go over the objects in the corresponding 'bucket'. It will certainly not go over all elements (or you would have a very poor implementation of the hashCode method).
This can be seen in the source code of the HashSet#contains method, which ultimately calls the following code in HashMap:
final Entry<K,V> getEntry(Object key) {
    int hash = (key == null) ? 0 : hash(key.hashCode());
    for (Entry<K,V> e = table[indexFor(hash, table.length)];
         e != null;
         e = e.next) {
        Object k;
        if (e.hash == hash &&
            ((k = e.key) == key || (key != null && key.equals(k))))
            return e;
    }
    return null;
}
This clearly shows it loops only over a limited set of objects.
See also this question for more information about a HashMap (which is used internally by a HashSet)
Last piece of advise: if you have performance problems, use a profiler to see where the actual bottleneck is located. I doubt it will be in the contains call.