Your classes have to override equals and hashCode 
From the Set Documentation
A collection that contains no duplicate elements. More formally, sets
  contain no pair of elements e1 and e2 such that e1.equals(e2), and at
  most one null element. As implied by its name, this interface models
  the mathematical set abstraction.
The Set interface places additional stipulations, beyond those
  inherited from the Collection interface, on the contracts of all
  constructors and on the contracts of the add, equals and hashCode
  methods.
to override these methods you can do this:
public class Person {
    private String name;
    private int age;
    // ...
    @Override
    public int hashCode() {
        return new HashCodeBuilder(17, 31). // two randomly chosen prime numbers
            // if deriving: appendSuper(super.hashCode()).
            append(name).
            append(age).
            toHashCode();
    }
    @Override
    public boolean equals(Object obj) {
       if (!(obj instanceof Person))
            return false;
        if (obj == this)
            return true;
        Person rhs = (Person) obj;
        return new EqualsBuilder().
            // if deriving: appendSuper(super.equals(obj)).
            append(name, rhs.name).
            append(age, rhs.age).
            isEquals();
    }
}
https://stackoverflow.com/a/27609/1754020