This article describes several tests that can be performed when troubleshooting possible network issues related to VoIP call quality.
On linux we can install the following.
yum install epel-release
yum --enablerepo=epel install mtr iperf
For a windows computer we can download the mtr and iperf v2 equivalents. Do NOT use iperf v3. Iperf v3 is most likely not compatible with the version on the Linux side.
https://sourceforge.net/projects/winmtr
https://iperf.fr/download/windows/iperf-2.0.9-win32.zip
There is also a utility built into windows called tracert. The linux equivalent is called traceroute. Those utilities can be used as a last resort if WinMTR or MTR is not an option.
General network testing
Running MTR or WinMTR to a public IP address or hostname will show the hops, packet loss, and packet loss location. If using tracert (windows) or traceroute (linux), it will only show hops and ping time but not packet loss or packet loss location.
mtr some_ip_address_or_hostname
What we are looking for is a large jump in ping times and/or packet loss starting on one of the hops. If the packet loss continues on from that hop there is a network issue at that location. If the packet loss is only on some hops but does not start on either end then you can usually ignore it.
Example:
The mtr output below is not a network problem even though 1 hop shows 100% packet loss. Average round trip latency (bottom row, 4th column) for VoIP should be < 300ms.
mtr google.com
HOST: mypbx.somedomain.org Loss% Snt Last Avg Best Wrst StDev
1. serverxx.powerpbx.org 0.0% 10 0.0 0.0 0.0 0.0 0.0
1. te0-0-0-6.ccr21.dfw01.atlas. 0.0% 10 1.4 1.6 1.1 3.0 0.6
2. be2432.ccr21.mci01.atlas.cog 0.0% 10 11.3 11.3 11.3 11.4 0.1
3. be2156.ccr41.ord01.atlas.cog 0.0% 10 23.4 23.3 23.3 23.4 0.1
4. be2216.ccr41.ord03.atlas.cog 0.0% 10 23.4 23.5 23.3 23.7 0.1
5. 38.88.204.78 0.0% 10 29.3 29.5 29.2 31.8 0.8
6. 209.85.143.154 0.0% 10 29.4 29.4 29.4 29.5 0.0
7. 209.85.243.173 0.0% 10 34.2 34.3 34.1 34.5 0.1
8. 72.14.239.190 0.0% 10 33.7 34.9 33.7 45.6 3.8
9. 209.85.243.254 0.0% 10 33.9 33.9 33.9 34.0 0.0
10. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
11. yk-in-f100.1e100.net 0.0% 10 33.9 33.9 33.8 33.9 0.0
This next mtr output shows a problem on hop #8. Notice how the packet loss continues down the line from where it started.
mtr google.com HOST: mypbx.somedomain.org Loss% Snt Last Avg Best Wrst StDev 1. serverxx.powerpbx.org 0.0% 10 0.0 0.0 0.0 0.0 0.0 1. te0-0-0-6.ccr21.dfw01.atlas. 0.0% 10 1.4 1.6 1.1 3.0 0.6 2. be2432.ccr21.mci01.atlas.cog 0.0% 10 11.3 11.3 11.3 11.4 0.1 3. be2156.ccr41.ord01.atlas.cog 0.0% 10 23.4 23.3 23.3 23.4 0.1 4. be2216.ccr41.ord03.atlas.cog 0.0% 10 23.4 23.5 23.3 23.7 0.1 5. 38.88.204.78 0.0% 10 29.3 29.5 29.2 31.8 0.8 6. 209.85.143.154 0.0% 10 29.4 29.4 29.4 29.5 0.0 7. 209.85.243.173 0.0% 10 34.2 34.3 34.1 34.5 0.1 8. 72.14.239.190 30.0% 10 33.7 34.9 33.7 45.6 3.8 9. 209.85.243.254 35.2% 10 33.9 33.9 33.9 34.0 0.0 10. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 11. yk-in-f100.1e100.net 36.4% 10 33.9 33.9 33.8 33.9 0.0
NOTE: Sometimes this test can be misleading if only done in one direction.
The Windows operating system has a built-in command ("pathping") that works similar to WinMTR. However, it frequently does not work. For that reason we only recommend WinMTR.
VoIP specific network testing
Realtime Asterisk call monitoring
From linux console command line:
watch -n 1 'asterisk -rx "sip show channelstats"'
# or
watch -n 1 "asterisk -rx 'pjsip show channelstats'"
For this test we want packet loss < 4% and jitter < 0.0200 (20ms)
Iperf test for G.711 simulation and two way calls
For this test we need to run iperf on both ends to simulate G.711 VoIP traffic. We can use the linux or windows version of iperf on either end with these commands.
# On server (listening) end. iperf --server --udp --len 300 --tos 184 -fk --interval 5
# On client (sending) end. # This should be the end behind NAT on a LAN if applicable.
iperf -c SERVER_IP_OR_HOSTNAME --udp --len 300 --bandwidth 67000 --tos 184 -fk --interval 5 --time 60 --listenport 5002 -w 130KB
# If port forwarding or no NAT the test can run both ways simultanously by adding -d or --dualtest option.
iperf -c SERVER_IP_OR_HOSTNAME --udp --len 300 --bandwidth 67000 --dualtest --tos 184 -fk --interval 5 --time 60 --listenport 5002 -130KB
Example:
This is a call simulation between a computer on a LAN and a PBX server in a data center. We want to see average jitter < 20ms and average packetloss < 1% .
iperf --server --udp --len 300 --tos 184 -fk --interval 5
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 300 byte datagrams
UDP buffer size: 130 KByte (default)
------------------------------------------------------------
[ 3] local xx.xx.xx.xx port 5001 connected with xx.xx.xx.xx port 55429
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0- 5.0 sec 41.0 KBytes 67.2 Kbits/sec 1.266 ms 0/ 140 (0%)
[ 3] 5.0-10.0 sec 41.0 KBytes 67.2 Kbits/sec 1.230 ms 0/ 140 (0%)
[ 3] 10.0-15.0 sec 40.7 KBytes 66.7 Kbits/sec 0.968 ms 0/ 139 (0%)
[ 3] 15.0-20.0 sec 40.7 KBytes 66.7 Kbits/sec 0.989 ms 0/ 139 (0%)
[ 3] 20.0-25.0 sec 41.0 KBytes 67.2 Kbits/sec 1.149 ms 0/ 140 (0%)
[ 3] 25.0-30.0 sec 41.0 KBytes 67.2 Kbits/sec 1.063 ms 0/ 140 (0%)
[ 3] 30.0-35.0 sec 40.7 KBytes 66.7 Kbits/sec 1.020 ms 0/ 139 (0%)
[ 3] 35.0-40.0 sec 41.0 KBytes 67.2 Kbits/sec 1.116 ms 0/ 140 (0%)
[ 3] 40.0-45.0 sec 41.0 KBytes 67.2 Kbits/sec 0.929 ms 0/ 140 (0%)
[ 3] 45.0-50.0 sec 40.7 KBytes 66.7 Kbits/sec 1.037 ms 0/ 139 (0%)
[ 3] 50.0-55.0 sec 41.0 KBytes 67.2 Kbits/sec 1.974 ms 0/ 140 (0%)
[ 3] 55.0-60.0 sec 41.0 KBytes 67.2 Kbits/sec 1.102 ms 0/ 140 (0%)
[ 3] 0.0-60.1 sec 491 KBytes 67.0 Kbits/sec 1.143 ms 0/ 1677 (0%)
