One possible option is to perform a long ping test from certain endpoints to traverse the same path as the VoIP packets, while taking packet captures from before and after a network device of concern. Here is a simple illustration:
How could we do better?
Wireshark and its family of tools might come in handy to try to automate certain aspects. For example, to read a packet capture file and display only relevant fields of interest, this could be readily accomplished with tshark. To track an ICMP echo request/reply packet through a device, we need to identify what suitable protocol fields to use to uniquely identify them between packet captures. The followings show populated ICMP echo request and reply fields, respectively, for completeness:
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Code: 0
Checksum: 0xb51c
Identifier (BE): 110 (0x006e)
Sequence number (BE): 38865 (0x97d1)
Data (32 bytes)
Internet Control Message Protocol
Type: 0 (Echo (ping) reply)
Code: 0
Checksum: 0xbd1c
Identifier (BE): 110 (0x006e)
Sequence number (BE): 38865 (0x97d1)
Data (32 bytes)
To uniquely identify an ICMP echo packet, we could use the highlighted ICMP sequence number along with its source and destination IP addresses. The following tshark (version 1.6.2) options will display only these fields along with the packet arrival time:
$ tshark -r ingress.cap -R icmp -T fields -e frame.time -e ip.src -e ip.dst -e icmp.seq
Mar 5, 2017 19:34:11.033332000 172.25.7.31 172.18.4.2 1066
Mar 5, 2017 19:34:11.033429000 172.18.4.2 172.25.7.31 1066
Mar 5, 2017 19:34:12.033172000 172.25.7.31 172.18.4.2 1067
Mar 5, 2017 19:34:12.033267000 172.18.4.2 172.25.7.31 1067
...
$ tshark -r egress.cap -R icmp -T fields -e frame.time -e ip.src -e ip.dst -e icmp.seq
Mar 5, 2017 19:34:11.033354000 172.25.7.31 172.18.4.2 1066
Mar 5, 2017 19:34:11.033425000 172.18.4.2 172.25.7.31 1066
Mar 5, 2017 19:34:12.033186000 172.25.7.31 172.18.4.2 1067
Mar 5, 2017 19:34:12.033261000 172.18.4.2 172.25.7.31 1067
...
The availability of such an output would make it easier to compute the switching delta, however this would still be manual. What if we could code this in a suitable programming language. For our illustration, we will use Python v2.6+.
From within Python, we could invoke tshark using the call function of the subprocess module. Here is a code extract:
import sys
from subprocess import call
file1, file2 = sys.argv[1:]
call("tshark -r "+ file1 + " -R icmp -T fields -e frame.time -e ip.src -e ip.dst -e icmp.seq | awk '{print $4,$5,$6,$7}' > tmp-1.txt", shell=True)
call("tshark -r "+ file2 + " -R icmp -T fields -e frame.time -e ip.src -e ip.dst -e icmp.seq | awk '{print $4,$5,$6,$7}' > tmp-2.txt", shell=True)
Variables file1 and file2 are packet capture files of before (on interface_1 or iface_1) and after (on iface_2) the device in question, respectively, and are populated with values from the command-line arguments. The output of tshark is filtered with awk to include only the time portion of the date/time display along with the remaining fields, and written to a separate tmp file for each capture file.
It is possible that the capture files may be started or stopped at a slightly different time introducing inconsistent number of packets. To cater for such inconsistency, we could compare the tmp files and include only matching lines, where the match will be based on IP addresses and the sequence number. The following code extract shows how awk is invoked using the call function again with an array indexed by a concatenation of IP addresses and the ICMP echo sequence number, and written to another set of tmp files when matching lines are found:
call("awk 'NR==FNR {packets[$2$3$4]++;next}; packets[$2$3$4] > 0' tmp-1.txt tmp-2.txt > tmp-2-m.txt", shell=True)
call("awk 'NR==FNR {packets[$2$3$4]++;next}; packets[$2$3$4] > 0' tmp-2.txt tmp-1.txt > tmp-1-m.txt", shell=True)
We should now have consistent packet details on the resulting files to perform computation. In order to compute per-packet delay, we need to maintain a number of variables. The followings show a number of variables for packets firstly seen on iface_1 and traverses in the iface_1 to iface_2 direction:
lpkt = 0 # number of packets seen
ltotaldelay = 0.0 # cumulative switching delay
lpktarrival = 0 # previous packet switching delay
ljittertotal = 0.0 # cumulative jitter
A similarly-named set of variables starting with 'w' will be used for packets seen on iface_2 first (not shown here). Let us open both files to read and iterate through. We start with the first packet line from tmp-1-m.txt and look for a match in the second tmp file, tmp-2-m.txt. Each line from either file is split and stored as a list. The fourth item on the list represents the sequence number. These sequence numbers are then compared. The following code extract illustrates this logic:
file-iface1 = open("tmp-1-m.txt","r")
file-iface2 = open("tmp-2-m.txt","r")
while file-iface1:
line = file-iface1.readline().split()
timestamp = datetime.strptime(line[0],'%H:%M:%S.%f000');
seq = line[3]
while file-iface2:
line2 = file-iface2.readline().split()
seq2 = line2[3]
if seq2 == seq:
When a match is found, we check to see which packet has a smaller timestamp. If the packet in file-iface1 has a smaller timestamp, it would imply this packet traversed from iface_1 to iface_2 on this network device. Otherwise, the packet traversed in the reverse direction. In both cases, the switching delta could be readily computed and relevant packet counter and total delay is updated as shown below:
timestamp2 = datetime.strptime(line2[0],'%H:%M:%S.%f000')
if timestamp < timestamp2:
delta = timestamp2 - timestamp
lpkt += 1
ltotaldelay += delta.microseconds
if lpktarrival > 0:
ljittertotal += abs(delta.microseconds - lpktarrival)
lpktarrival = delta.microseconds
else:
lpktarrival = delta.microseconds
else:
delta = timestamp - timestamp2
wpkt += 1
wtotaldelay += delta.microseconds
if wpktarrival > 0:
wjittertotal += abs(delta.microseconds - wpktarrival)
wpktarrival = delta.microseconds
else:
wpktarrival = delta.microseconds
break
The nested if blocks above are used to compute jitter for the second packet and onwards for packets traversing iface_1 to iface_2, and vice-versa, respectively. We compute and increment the cumulative variable with the absolute difference of the previous switching delta with current as well as update the previous delta. Once a matching packet line is found on the second file and computation completed, the flow exits the inner while loop and proceeds with the next line on the first file until all packet lines are processed. As such, the inner while loop would have a O(1) worst-case complexity (in big-O notation), due to the pre-processing done during the packet consistency processing.
At the end of the loop, average switching delay as well as jitter for each direction could be readily computed and displayed as follows:
print("Average Switching Delay")
print("="*25)
print("iface_1 -> iface_2 delay : %.2f microsec" % (ltotaldelay/lpkt))
print("iface_1 -> iface_2 jitter: %.2f microsec" % (ljittertotal/(lpkt-1)))
print("iface_2 -> iface_1 delay : %.2f microsec" % (wtotaldelay/wpkt))
print("iface_2 -> iface_1 jitter: %.2f microsec" % (wjittertotal/(wpkt-1)))
Here is a sample output for a set of packet captures called ingress.cap and egress.cap:
$ time_delta.py ingress.cap egress.cap
Average Switching Delay
=========================
iface_1 -> iface_2 delay : 79.19 microsec
iface_1 -> iface_2 jitter: 137.90 microsec
iface_2 -> iface_1 delay : 4.24 microsec
iface_2 -> iface_1 jitter: 1.07 microsec
To estimate the real execution time of this code, I used two rather large capture files with up to 140,000 packets each. 99.8% of the execution time was spent during the tshark calls.
Now, what if you are looking for more than just the average values, for instance, you want to see switching delay behaviour over time. May be one would also want to see the packet loss performance of a network segment, not just of ICMP packets.
SteelCentral PacketAnalyzer Plus
If you haven't heard about this, I am pleased to introduce SteelCentral Packet Analyzer Plus, which runs on Windows machines. To generate a delay over time chart, you will need to create a multi-segment source using the two before/after capture files, and then apply the MSA-Segment-Delay view against this source using just a couple of mouse-clicks. This may sound a bit complicated, but it is far from it. The result will be presented as charts in a short order by this tool. The following shows a screenshot of delay over time along with the per-direction average chart and a pie-chart illustrating the relative direction delay:
From our Python program, here is the corresponding output for the same capture files:
Average Switching Delay
=========================
iface_1 -> iface_2 delay : 18.45 microsec
iface_1 -> iface_2 jitter: 24.59 microsec
iface_2 -> iface_1 delay : 10.48 microsec
iface_2 -> iface_1 jitter: 6.57 microsec
Using PacketAnalyzer Plus, one could readily perform an analysis to show frame size distribution for the same files as presented below:
Similarly, to determine the packet loss performance for a segment, you can apply a Segment-Dropped-Packets-Overview view to the source. The following is a sample output:
Such analyses can be done rather quickly with PacketAnalyzer Plus even with large packet capture files. The use cases described above for PacketAnalyzer Plus are only a very small fraction of what it is capable of. There are numerous other supported use cases, including VoIP related analysis, relevant to the theme of this blog. If this has sparked any interest, check it out for yourself with a trial download. Otherwise, you have the poor-man's version of delay and jitter analysis tool that could work on Linux or Mac machines with tshark v1.6.2+ and Python v2.6+. 😊
