Assuming you are talking about shared_ptr here...
Is this simply because of using the
  reference of the smart pointer after
  its been destroyed?
This is a good answer. You may not know absolutely the lifetime of the pointer your reference refers too. 
To get around this, you'd want to look into boost::weak_ptr. It doesn't participate in reference counting. When you need to use it, it gives you a shared_ptr which goes away once your done with it. It will also let you know when the refered to pointer has been collected. 
From the weak_ptr documentation
The weak_ptr class template stores a
  "weak reference" to an object that's
  already managed by a shared_ptr. To
  access the object, a weak_ptr can be
  converted to a shared_ptr using the
  shared_ptr constructor or the member
  function lock. When the last
  shared_ptr to the object goes away and
  the object is deleted, the attempt to
  obtain a shared_ptr from the weak_ptr
  instances that refer to the deleted
  object will fail: the constructor will
  throw an exception of type
  boost::bad_weak_ptr, and
  weak_ptr::lock will return an empty
  shared_ptr.
Note the method expired() will also tell you if your ptr is still around.