Basically they just describe what the individual components are:
CV_8U: 1-byte unsigned integer (unsigned char).
CV_32S: 4-byte signed integer (int).
CV_32F: 4-byte floating point (float).
What you always have to keep in mind is that you cannot just cast them from one into the other (or it probably won't do what you want), especially between differently sized types.
So always make sure you use real conversion functions for converting between them, like cv::convert or cv::Mat::convertTo. Don't just try to access the elements of e.g. a cv::Mat of CV_8U type using e.g. cv::Mat::at<float> or cv::Mat_<float>.
Or if you just want to convert individual elements and don't want to create a new matrix of the other type, access the elements using the appropriate function (in the example cv::Mat::at<unsigned char>) and convert the result to float.
Likewise is there also a difference between the number of components and a cv::Mat of CV_8UC3 type is different from an image of CV_8UC1 type and should (usually) not be accessed by cv::Mat::at<unsigned char>, but by cv::Mat::at<cv::Vec3b>.
EDIT: Seeing Martin's answer it may be that you are aware of this all and his explanations are more what you have been looking for.