Netstack exposes useful debugging information as inspectable data that can be retrieved from snapshots (inspect.json
within snapshots created by fx snapshot
) or by using the iquery
tool (fx iquery
).
In this document we list useful example queries into the inspect data that netstack exposes.
We'll use the jq
tool throughout, all queries can be used from an inspect.json
or fx iquery --format json
by piping either into jq
, e.g.:
cat snapshot/inspect.json | fx jq '...'
fx iquery --format json show core/network/netstack | fx jq '...'
There are different sources of inspect data in Netstack, referenced by their payload keys.
Socket Info
contains information of all currently open sockets, keyed by socket identifier, e.g.:
{ "12345": { "NetworkProtocol": "IPv6", "TransportProtocol": "UDP", "State": "BOUND", "LocalAddress": "[fe80::8d88:9942:4f32:7259]:546", "RemoteAddress": ":0", "BindAddress": "", "BindNICID": "5", "RegisterNICID": "5", "Stats": { ... } } }
To retrieve all sockets from inspect data use:
fx jq '.[] | select(.moniker == "core/network/netstack") | .payload."Socket Info" | .[]?'
NICs
contains information about each of the network interfaces presently installed in the netstack, keyed by their interface identifier, e.g:
{ "2": { "Name": "wlanx62", "NICID": "3", "AdminUp": "true", "LinkOnline": "true", "Up": "true", "Running": "true", "Loopback": "false", "Promiscuous": "false", "LinkAddress": "44:07:0b:e2:cf:62", "ProtocolAddress0": "[arp] 617270/0", "ProtocolAddress1": "[ipv4] 192.168.0.23/24", "ProtocolAddress2": "[ipv6] fe80::916b:7615:6b2e:f8b3/64", "ProtocolAddress3": "[ipv6] fe80::4607:bff:fee2:cf62/128", "DHCP enabled": "true", "MTU": 1500, "Ethernet Info": { ... }, "DHCP Info": { ... }, "Stats": { ... } } }
To retrieve all NICs from inspect data use:
fx jq '.[] | select(.moniker == "core/network/netstack") | .payload."NICs" | .[]?'
To look at a single NIC with id
or name
simply append | select(.NICID == "id")
or | select(.Name == "name")
, respectively.
Networking Stat Counters
contain stack-global counters for traffic and errors, e.g.:
{ "UnknownProtocolRcvdPackets": 0, "MalformedRcvdPackets": 0, "DroppedPackets": 0, "UDP": { "PacketsReceived": 51551, "UnknownPortErrors": 30509, ... }, "TCP": { "ActiveConnectionOpenings": 4443, "PassiveConnectionOpenings": 53, ... }, "IP": {...}, "ICMP": {...} }
To get counters use:
fx jq '.[] | select(.moniker == "core/network/netstack") | .payload."Networking Stat Counters" | select(. != null)'
You can append | .TCP
, or | .UDP
to look at only the subset of interest if needed.
Routes
contains information about all the routes in the routing table, e.g.:
{ "0": { "Destination": "127.0.0.0/8", "Dynamic": "false", "Enabled": "true", "Gateway": "", "Metric": "100", "MetricTracksInterface": "true", "NIC": "1" }, }
To retrieve all routes from inspect data use:
fx jq '.[] | select(.moniker == "core/network/netstack") | .payload."Routes" | .[]?'
Netstack exposes pprof
data that can be used to gather more information from the Go runtime.
A typical snapshot will contain periodic pprof
data which is gathered at set intervals. You can query the available pprof
information with:
fx jq -c '.[] | select(.moniker == "core/network/netstack") | select(.payload.root.pprof != null) | {file: .metadata.filename, keys: (.payload.root.pprof | keys)}'
Which will generate output like
{"file":"pprof/2020-10-22T16:21:49Z.inspect","keys":["allocs","block","goroutine","heap","mutex","threadcreate"]} {"file":"pprof/now.inspect","keys":["allocs","block","goroutine","heap","mutex","threadcreate"]} {"file":"pprof/2020-10-22T16:19:49Z.inspect","keys":["allocs","block","goroutine","heap","mutex","threadcreate"]} {"file":"pprof/2020-10-22T16:20:49Z.inspect","keys":["allocs","block","goroutine","heap","mutex","threadcreate"]}
Each of the pprof
profiles is encoded in base64. The following line gets all the pprof
files and decodes them at once (remember to either pipe in your inspect
data or add path to your inspect file after the jq
command):
fx jq -rc '.[] | select(.moniker == "core/network/netstack") | select(.payload.root.pprof != null) | . as $parent | (.payload.root.pprof | to_entries | .[] | ($parent | .metadata.filename) + "_" + .key + " " + .value)' | \ while read f c; do echo $c | sed s/b64://g | base64 -d > $(echo $f | sed 's/pprof\///g'); done
You can modify the selector .payload.root.pprof != null
and append and .metadata.filename == "pprof/now.inspect"
to only get the latest profile.