vector properly destroys objects stored IN the vector.  The destructor will be called.  If you have a vector of pointers, then this means the pointer's own destructor (and not what it points to).
The destructor for a raw pointer does nothing.  This is what you want if you have a non-owning pointer to an object that another part of the program will destroy.
The destructor for a smart pointer does whatever is necessary to make sure the object gets freed at the right time.  For unique_ptr, that's right now.  For shared_ptr, it's whenever the reference count hits zero.
Use the right kind of pointer, and trust vector to trigger the behavior associated with that pointer when the element is erased.