I did a little work to check that out for you
Random rng = new Random();
ArrayList<Integer> numbers = new ArrayList<>();
for (int i = 0; i < 100000; ++i)
    numbers.add(rng.nextInt());
    
HashSet<Integer> set = new HashSet<>();
long beginNoCheck = System.nanoTime();
for (int i : numbers)
{
    set.add(i);
}
long endNoCheck = System.nanoTime();
   
set = new HashSet<>();
long beginCheck = System.nanoTime();
for (int i : numbers)
{
    if (!set.contains(i))
        set.add(i);
}
long endCheck = System.nanoTime();
    
System.out.println("Without check: " + (endNoCheck - beginNoCheck));
System.out.println("With check: " + (endCheck - beginCheck));
And it seems, that checking if HashSet contains an element before adding it is a bit faster. I am getting results like
Without check: 66060748
With check: 46209808
Without check: 38249163
With check: 32058546
Without check: 58362677
With check: 34429848
Without check: 52095512
With check: 39612073
Without check: 34892990
With check: 28945278
Without check: 42090287
With check: 38426209