I gave another try on trying to understand the content of an objects.inv file and hopefully this time I inspected numpy and h5py instead of only OpenCV's one.
How to read an intersphinx inventory file
Despite the fact that I couldn't find anything useful about reading the content of an object.inv file, it is actually very simple with the intersphinx module.
from sphinx.ext import intersphinx
import warnings
def fetch_inventory(uri):
    """Read a Sphinx inventory file into a dictionary."""
    class MockConfig(object):
        intersphinx_timeout = None  # type: int
        tls_verify = False
    class MockApp(object):
        srcdir = ''
        config = MockConfig()
        def warn(self, msg):
            warnings.warn(msg)
    return intersphinx.fetch_inventory(MockApp(), '', uri)
uri = 'http://docs.python.org/2.7/objects.inv'
# Read inventory into a dictionary
inv = fetch_inventory(uri)
# Or just print it
intersphinx.debug(['', uri])
File structure (numpy)
After inspecting numpy's one, you can see that keys are domains:
[u'np-c:function',
 u'std:label',
 u'c:member',
 u'np:classmethod',
 u'np:data',
 u'py:class',
 u'np-c:member',
 u'c:var',
 u'np:class',
 u'np:function',
 u'py:module',
 u'np-c:macro',
 u'np:exception',
 u'py:method',
 u'np:method',
 u'np-c:var',
 u'py:exception',
 u'np:staticmethod',
 u'py:staticmethod',
 u'c:type',
 u'np-c:type',
 u'c:macro',
 u'c:function',
 u'np:module',
 u'py:data',
 u'np:attribute',
 u'std:term',
 u'py:function',
 u'py:classmethod',
 u'py:attribute']
You can see how you can write your cross-reference when you look at the content of a specific domain. For example, py:class:
{u'numpy.DataSource': (u'NumPy',
  u'1.9',
  u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.DataSource.html#numpy.DataSource',
  u'-'),
 u'numpy.MachAr': (u'NumPy',
  u'1.9',
  u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.MachAr.html#numpy.MachAr',
  u'-'),
 u'numpy.broadcast': (u'NumPy',
  u'1.9',
  u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast.html#numpy.broadcast',
  u'-'),
  ...}
So here, :class:`numpy.DataSource` will work as expected.
h5py
In the case of h5py, the domains are:
[u'py:attribute', u'std:label', u'py:method', u'py:function', u'py:class']
and if you look at the py:class domain:
{u'AttributeManager': (u'h5py',
  u'2.5',
  u'http://docs.h5py.org/en/latest/high/attr.html#AttributeManager',
  u'-'),
 u'Dataset': (u'h5py',
  u'2.5',
  u'http://docs.h5py.org/en/latest/high/dataset.html#Dataset',
  u'-'),
 u'ExternalLink': (u'h5py',
  u'2.5',
  u'http://docs.h5py.org/en/latest/high/group.html#ExternalLink',
  u'-'),
 ...}
That's why I couldn't make it work as numpy references. So a good way to format them would be :class:`h5py:Dataset`.
OpenCV
OpenCV's inventory object seems malformed. Where I would expect to find domains there is actually 902 function signatures:
[u':',
 u'AdjusterAdapter::create(const',
 u'AdjusterAdapter::good()',
 u'AdjusterAdapter::tooFew(int',
 u'AdjusterAdapter::tooMany(int',
 u'Algorithm::create(const',
 u'Algorithm::getList(vector<string>&',
 u'Algorithm::name()',
 u'Algorithm::read(const',
 u'Algorithm::set(const'
 ...]
and if we take the first one's value:
{u'Ptr<AdjusterAdapter>': (u'OpenCV',
  u'2.4',
  u'http://docs.opencv.org/2.4/detectorType)',
  u'ocv:function 1 modules/features2d/doc/common_interfaces_of_feature_detectors.html#$ -')}
I'm pretty sure it is then impossible to write OpenCV cross-references with this file...
Conclusion
I thought intersphinx generated the objects.inv based on the content of the documentation project in an standard way, which seems not to be the case.
As a result, it seems that the proper way to write cross-references is API dependent and one should inspect a specific inventory object to actually see what's available.