How to handle implicit and explicit type conversions between const and non-const fancy pointers?

Question

Given the type definitions

using ConstHandle = MyEntity const*;
using Handle = MyEntity*;

it is possible to conduct the following type conversions:

1. Handle → ConstHandle (implicit)
2. ConstHandle → Handle (explicit via const_cast)
3. Handle* → ConstHandle* (explicit via const_cast)
4. ConstHandle& → Handle& (explicit via const_cast)

Can I achieve similar behaviour with handles based on IDs instead of pointers?

Cases 1 and 2 can be easily solved, e.g.

struct ConstHandle
{
   int id;
};

struct Handle
{
   operator ConstHandle() const { return { id }; }
   int id;
};

Handle const_handle_cast(ConstHandle h) { return { h.id }; }

Case 3 can almost be solved by deriving Handle from ConstHandle, i.e.

struct ConstHandle
{
   int id;
};

struct Handle : ConstHandle
{
};

Here, almost refers to the fact that, now, the conversion Handle* → ConstHandle* is implicit rather than explicit.

Is there a legal solution that incorporates case 4? Legal particularly refers to approaches that do not imply a violation of the strict aliasing rule.