Netstack Inspect Data

Netstack exposes useful debugging information as inspectable data that can be retrieved from snapshots (inspect.json within snapshots created by fx snapshot) or by using the iquery tool (fx iquery).

In this document we list useful example queries into the inspect data that netstack exposes.

We'll use the jq tool throughout, all queries can be used from an inspect.json or fx iquery --format=json by piping either into jq, e.g.:

cat snapshot/inspect.json | fx jq '...'

fx iquery --format=json show netstack.cmx | fx jq '...'

Inspect data

There are different sources of inspect data in Netstack, referenced by their payload keys.

Socket Info

Socket Info contains information of all currently open sockets, keyed by socket identifier, e.g.:

{
  "12345": {
    "NetworkProtocol": "IPv6",
    "TransportProtocol": "UDP",
    "State": "BOUND",
    "LocalAddress": "[fe80::8d88:9942:4f32:7259]:546",
    "RemoteAddress": ":0",
    "BindAddress": "",
    "BindNICID": "5",
    "RegisterNICID": "5",
    "Stats": { ... }
  }
}

To retrieve all sockets from inspect data use:

fx jq '.[] | select(.moniker == "netstack.cmx") | .payload."Socket Info" | .[]?'

NICs

NICs contains information about each of the network interfaces presently installed in the netstack, keyed by their interface identifier, e.g:

{
  "2": {
    "Name": "wlanx62",
    "NICID": "3",
    "AdminUp": "true",
    "LinkOnline": "true",
    "Up": "true",
    "Running": "true",
    "Loopback": "false",
    "Promiscuous": "false",
    "LinkAddress": "44:07:0b:e2:cf:62",
    "ProtocolAddress0": "[arp] 617270/0",
    "ProtocolAddress1": "[ipv4] 192.168.0.23/24",
    "ProtocolAddress2": "[ipv6] fe80::916b:7615:6b2e:f8b3/64",
    "ProtocolAddress3": "[ipv6] fe80::4607:bff:fee2:cf62/128",
    "DHCP enabled": "true",
    "MTU": 1500,
    "Ethernet Info": { ... },
    "DHCP Info": { ... },
    "Stats": { ... }
  }
}

To retrieve all NICs from inspect data use:

fx jq '.[] | select(.moniker == "netstack.cmx") | .payload."NICs" | .[]?'

To look at a single NIC with id or name simply append | select(.NICID == "id") or | select(.Name == "name"), respectively.

Networking Stat Counters

Networking Stat Counters contain stack-global counters for traffic and errors, e.g.:

{
  "UnknownProtocolRcvdPackets": 0,
  "MalformedRcvdPackets": 0,
  "DroppedPackets": 0,
  "UDP": {
    "PacketsReceived": 51551,
    "UnknownPortErrors": 30509,
    ...
  },
  "TCP": {
    "ActiveConnectionOpenings": 4443,
    "PassiveConnectionOpenings": 53,
    ...
  },
  "IP": {...},
  "ICMP": {...}
}

To get counters use:

fx jq '.[] | select(.moniker == "netstack.cmx")
           | .payload."Networking Stat Counters"
           | select(. != null)'

You can append | .TCP , or | .UDP to look at only the subset of interest if needed.

pprof

Netstack exposes pprof data that can be used to gather more information from the Go runtime.

A typical snapshot will contain periodic pprof data which is gathered at set intervals. You can query the available pprof information with:

fx jq -c '.[] | select(.moniker == "netstack.cmx")
              | select(.payload.root.pprof != null)
              | {file: .metadata.filename, keys: (.payload.root.pprof | keys)}'

Which will generate output like

{"file":"pprof/2020-10-22T16:21:49Z.inspect","keys":["allocs","block","goroutine","heap","mutex","threadcreate"]}
{"file":"pprof/now.inspect","keys":["allocs","block","goroutine","heap","mutex","threadcreate"]}
{"file":"pprof/2020-10-22T16:19:49Z.inspect","keys":["allocs","block","goroutine","heap","mutex","threadcreate"]}
{"file":"pprof/2020-10-22T16:20:49Z.inspect","keys":["allocs","block","goroutine","heap","mutex","threadcreate"]}

Each of the pprof profiles is encoded in base64. The following line gets all the pprof files and decodes them at once (remember to either pipe in your inspect data or add path to your inspect file after the jq command):

fx jq -rc '.[]
            | select(.moniker == "netstack.cmx")
            | select(.payload.root.pprof != null)
            | . as $parent
            | (.payload.root.pprof | to_entries | .[] | ($parent | .metadata.filename) + "_" + .key + " " + .value)' | \
 while read f c; do echo $c | sed s/b64://g | base64 -d > $(echo $f | sed 's/pprof\///g'); done

You can modify the selector .payload.root.pprof != null and append and .metadata.filename == "pprof/now.inspect" to only get the latest profile.