Saturday, July 5, 2014

tshark as a Command Line Troubleshooting tool

Wireshark is one of the must have tool in the arsenal of network professional. It allows one to take a trace on a client host to look deep inside frames to understand an operation or to look for clues of any misbehaving network applications or protocols. One typically uses other tools to take a packet trace on network devices and then analyse it in Wireshark. Another lessor known command line tool that comes with Wireshark is tshark. It has similar features as Wireshark.

For someone who is more comfortable on command line, using tshark along with various Linux tools allows for more advanced filtering. The simplest way to view a trace file with tshark is as below:

$ tshark -r capture.cap
  1 10:12:22.050975 0.000000 172.16.13.253 -> 172.16.1.137 CFLOW 1022 total: 15 (v9) records
  2 10:12:22.051314 0.000339 172.16.99.114 -> 172.16.98.156 TPKT 475 Continuation
  3 10:12:22.055856 0.004542 172.16.98.156 -> 172.16.1.14  TCP 66 55603 > 51513 [ACK] Seq=1827602552 Ack=1175114189 Win=1728 Len=0
  4 10:12:22.055963 0.000107 172.16.99.232 -> 172.16.80.7  SNMP 83 get-request SNMPv2-MIB::sysDescr.0

Each frame in the trace is shown as one line output with summary info of the highest level protocol that tshark is able to decode. The above sample trace appears to be an open capture with various exchanges between different hosts, possibly taken without any specific filter. If a trace file is large, console output will scroll very quickly. Like Wireshark, we can use filters to narrow our focus towards any traffic of interest. For instance, to see just NetFlow packets on port 2055, we can use filter udp.port == 2055 as shown:

$ tshark -r capture.cap -Y "(udp.port == 2055)"
  1 10:12:22.050975 0.000000 172.16.13.253 -> 172.16.1.137 CFLOW 1022 total: 15 (v9) records
 17 10:12:22.077250 0.026275 172.16.20.253 -> 172.16.1.137 CFLOW 1022 total: 15 (v9) records
 21 10:12:22.082645 0.005395 172.16.20.253 -> 172.16.1.137 CFLOW 1022 total: 15 (v9) records

The above output shows that frames 1, 17 and 21 are NetFlow frames. One each line, the third field represents the delta time, and 1022 is the frame size. The details after the frame size represent a summary of NetFlow specific information.

At times, we may be interested to get more than just a single line summary for each frame. To display full decode of each relevant protocol in a trace for a specific traffic of interest, we could use –V. However be warned, this outputs a lot of details. Here is an example against a single frame (removed lines shown with ellipses):

$ tshark -Vnr capture.cap -Y "(frame.number == 1)"
Frame 1: 1022 bytes on wire (8176 bits), 1022 bytes captured (8176 bits)
    Encapsulation type: Ethernet (1)

    Frame Length: 1022 bytes (8176 bits)

    [Protocols in frame: eth:ip:udp:cflow]
Ethernet II, Src: f8:c0:01:8b:7e:81 (f8:c0:01:8b:7e:81), Dst: 00:50:56:82:5c:bf (00:50:56:82:5c:bf)
    Destination: 00:50:56:82:5c:bf (00:50:56:82:5c:bf)

    Type: IP (0x0800)
Internet Protocol Version 4, Src: 172.16.13.253 (172.16.13.253), Dst: 172.16.1.137 (172.16.1.137)
    Version: 4

    Total Length: 1008
    Identification: 0x0000 (0)
    Flags: 0x02 (Don't Fragment)

User Datagram Protocol, Src Port: 12240 (12240), Dst Port: 2055 (2055)
    Source port: 12240 (12240)
    Destination port: 2055 (2055)

Cisco NetFlow/IPFIX
    Version: 9
    Count: 15
    SysUptime: 994443240


The details of all known protocols are decoded. This is usually used when a few relevant frames of interest were identified in the trace. Otherwise, there would be an overload of details.

If you are interested only on a specific protocol, we could display more details just for it using –O <proto>. Here is an example:

$ tshark -O eth -Vnr capture.cap -Y "(frame.number == 1)"
Frame 1: 1022 bytes on wire (8176 bits), 1022 bytes captured (8176 bits)
Ethernet II, Src: f8:c0:01:8b:7e:81 (f8:c0:01:8b:7e:81), Dst: 00:50:56:82:5c:bf (00:50:56:82:5c:bf)
    Destination: 00:50:56:82:5c:bf (00:50:56:82:5c:bf)
        Address: 00:50:56:82:5c:bf (00:50:56:82:5c:bf)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: f8:c0:01:8b:7e:81 (f8:c0:01:8b:7e:81)
        Address: f8:c0:01:8b:7e:81 (f8:c0:01:8b:7e:81)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IP (0x0800)
Internet Protocol Version 4, Src: 172.16.13.253 (172.16.13.253), Dst: 172.16.1.137 (172.16.1.137)
User Datagram Protocol, Src Port: 12240 (12240), Dst Port: 2055 (2055)
Cisco NetFlow/IPFIX


Here, only Ethernet header details were expanded, and all other protocols displayed with a summary line. To display the list of supported protocols, use the following command:

$ tshark -G protocols
For the same frame 1 above, to decode Netflow instead, you could determine which keyword to use by performing a simple filter (i.e. the required keyword is cflow):

$ tshark -G protocols | grep NetFlow
Cisco NetFlow/IPFIX  CFLOW     cflow

There are times when a known protocol is using a non-standard port. The following example shows a trace where NetBIOS traffic is on port 7801, whose packet length is greater than 41 bytes:

$ tshark -nr capture.cap -Y "(tcp.port == 7801 && ip.len > 41)"
 41 21:36:26.608905 0.000000 10.114.22.254 -> 10.114.96.230 TCP 188 61901 > 7801 [PSH, ACK] Seq=3940639060 Ack=3275647333 Win=16425 Len=130
 43 21:36:26.613294 0.004389 10.114.22.254 -> 10.114.96.230 TCP 188 61902 > 7801 [PSH, ACK] Seq=2746223592 Ack=4123396674 Win=16425 Len=130

These are decoded as TCP segments. Since we know it is NetBIOS session traffic, we can explicitly request tshark to decode using -d <port,proto> option as shown here:

$ tshark -d tcp.port==7801,nbss -nr capture.cap -Y "(tcp.port == 7801 && ip.len > 41)"
 41 21:36:26.608905 0.000000 10.114.22.254 -> 10.114.96.230 NBSS 188 NBSS Continuation Message
 43 21:36:26.613294 0.004389 10.114.22.254 -> 10.114.96.230 NBSS 188 NBSS Continuation Message

If we compare the two outputs above, the summary lines are clearly different depending on how the frames are decoded.

Finally, there are some interesting overall statistics that we could glean using tshark, especially for large packet traces. If we have a large HTTP trace, run the following command to determine the distribution of HTTP methods used and HTTP responses from the server:

$ tshark -z http,tree -qr capture.cap
=================================================================
 HTTP/Packet Counter           value       rate           percent
-----------------------------------------------------------------
 Total HTTP Packets             208       0.004884              
  HTTP Request Packets           106       0.002489          50.96%
   GET                            104       0.002442          98.11%
   POST                             2       0.000047           1.89%
  HTTP Response Packets          102       0.002395          49.04%
   ???: broken                      0       0.000000           0.00%
   1xx: Informational               0       0.000000           0.00%
   2xx: Success                    98       0.002301          96.08%
    200 OK                          98       0.002301         100.00%
   3xx: Redirection                 4       0.000094           3.92%
    302 Found                        4       0.000094         100.00%
   4xx: Client Error                0       0.000000           0.00%
   5xx: Server Error                0       0.000000           0.00%
  Other HTTP Packets               0       0.000000           0.00%
=================================================================


The -q option ensures only the summary is shown and not every single frame in the trace. My favourite summary is the packet length distribution. This is one of the more useful facts of a network that could help in troubleshooting certain network behaviour:

$ tshark -z plen,tree -qr capture.cap
=================================================================
 Packet Lengths          value         rate           percent
-------------------------------------------------------------------
 Packet Lengths          5579       0.011575              
  0-19                       0       0.000000           0.00%
  20-39                      0       0.000000           0.00%
  40-79                   2837       0.005886          50.85%
  80-159                   500       0.001037           8.96%
  160-319                  243       0.000504           4.36%
  320-639                  111       0.000230           1.99%
  640-1279                 148       0.000307           2.65%
  1280-2559               1740       0.003610          31.19%
  2560-5119                  0       0.000000           0.00%
  5120-4294967295            0       0.000000           0.00%
=================================================================


I hope you find tshark a useful tool to add to your arsenal. Enjoy!

No comments:

Post a Comment