Under Windows, you can create a new protocol using the Registry Editor:
Start / Run / regedit
Look for the HKEY_CLASSES_ROOT hive, and create a new series of keys with the following structure:
tel\shell\open\command
At the tel level, edit the (Default) string, and change its value to URL:tel Protocol
Again at the tel level, add a new string called URL Protocol and no value.
Then at the command level, edit the (Default) string, and in the value section, enter the command you want to execute:
c:\path\to\telephone\application.exe %1
The %1 will be replaced by whatever is placed after the tel: sequence, ie, the phone number.
And that should do it.
Under Ubuntu
This might be version specific, but the following should work:
gconftool-2 -t string -s /desktop/gnome/url-handlers/tel/command "/path/to/telephone/application %s"
gconftool-2 -s /desktop/gnome/url-handlers/tel/needs_terminal false -t bool
gconftool-2 -t bool -s /desktop/gnome/url-handlers/tel/enabled true
In this case the %s will be replaced by the phone number.
In the case of qutecom, it appears that the Windows command line should be:
"c:\program files\qutecom\qutecom.exe" -c call\%1
(adjust for your installation location, and try this from a dos prompt first, replacing %1 with a phone number)