I want to know how I can compare the quality of different voice products, in order that it gives the best audio quality rating for calls through the internet in average network conditions.
How exactly can I benchmark this? Are there benchmarks available on the internet?
The quality of each program will depend on the connection of each person involved in the conversation, as well as the available hardware - not limited to, but including:
If these factors are not a consideration (IE, each person involved has good quality hardware, stable connection, etc) then your best bet would be Skype.
Skype requires a lot of ports open to work. Protocols are all closed. It violates my mostly closed firewall setup severely.
GTalk uses very few ports, and uses open protocols. It works well with my mostly closed firewall setup. It will interwork with your own servers, so you can setup your own border server if needed.
I haven't used Live Messager as I mostly use Linux.
I would expect no significant call quality differences between any of the three services. My choice would be based more on security considerations such as firewall requirements.
I tested GTalk to my wife when she was on a train using wireless. Quality was fine for both voice and video.
This is a fairly subjective question - I can only provide subjective responses :-) but I've used all three (Skype, GTalk, WLM) a fair amount.
Skype's SILK codec is vastly improved over previous iterations but for me - even with over 2 Mbps of upstream bandwidth - it still takes a while to scale up to use all the available bandwidth, then it seems to want to lower in quality at the drop of a hat (or lost packet).
GTalk's quality could in theory be excellent; their plugin supports:
but in real life I've only ever experienced the pleasure of a 16 Kbps Speex call. :-( Windows Live Messenger is pretty much the same quality wise - Skype edges it. However, Google Plus' Hangout (which uses the latest Vidyo codecs) is (at the moment at least) really good quality and pretty much on a par with Skype for pure audio quality. Looking forward to testing more as times goes on.
Skype is also quite tolerant (so it seems, from a weekend of testing in a hotel with overpriced wifi) of 3G connections, at least with a VPN, HSDPA and an acceptable level of contention on the cell tower. to get a call for more than 3 seconds without almost 100% dropped packets I had to VPN through the connection (as expected, as I'm on the middle tier for my mobile broadband package it deprioritises Skype traffic). I had very acceptable latency of about 200-250ms; much better than I thought. Throughput was fairly steady too.
So at the moment, Skype pips it for call quality - but I very much expect Google/Vidyo to surpass that quality, particularly given the SQ and PQ of calls I've seen over standard broadband using the standard Vidyo-branded service.
The article Best Free Video Conference Software Review grades video and voice conferencing software for audio and video. It gives the following grades for audio :
Skype : 7 (The audio is very loud. The audio quality is also very good with very little to no feedback.)
GoogleTalk : 8 (The audio is very good with little feedback)
Live Messenger : 4 (The audio stutters sometimes and has clipped speech. It also echos at times. )
This would be difficult to benchmark without sophisticated audio equipment. Most consumer level equipment, from the speakers to the headphones to the microphones, are very low quality, providing only enough data to basically transmit recognizable human speach.
You would need an accurate sound sample conveying a wide range of possible sounds in different frequencies and with different types of stops (consonant sounds), and you'd need a system that has a very good sound card electronically isolated from the rest of the system to prevent sample contamination. You would need very high quality, reference tuned inputs and outputs (microphones and speakers) and then a recording device that can compare the resulting sound output to the reference input and highlight any differences.
This becomes very expensive a long time before it becomes practical.
The cheap way to do this is to use the same computer and the same people to rehearse a set selection of dialog and rate it subjectively on clarity and comprehension.
Or, you can just go online and read what others who have more money and more time have done.
I think the best strategy to do this would be able using the same technique used to test telephony links. Here you have both conversation sides, called A (the caller) and B (the called one). The technique consists on inject on A side a wide and linear range of acceptable frequencies (for example from 3k to 15kHz) and analyze how this range of frequencies are heard on B side (on telephony and data links this are made by network analyzers). The best link would give exactly the same wide and linear levels of transmitted signal on B side (this is theorical, on phisical links you will never get this enviroment). I dont know if there is already a software solution to test it, but it wolud be great to have some solution. This solution would be a software that generates a specific (and fixed) bandwidth signals, with all components on the same level on the transmiter side; and a frequency component reader on the reception side. The aplication compares the diferences between transmited and received signals and give you a "benchmark note" to the link (and consequently to the voip solution software used). My recomendation is (if you dont find any specific software that does that) is use Matlab to generate a linear audio source, transmit it over the voip link channel, receive and analyze it on the frequency domain. This would be a start to a benchmark software.
