Understanding why a packet drops in the Linux kernel is not always easy. The networking stack is wide and reasons to refuse a given packet are multiple and include invalid data from a protocol, firewall rules, wrong checksum, full queues, qdisc or XDP actions, and many more reasons. It is possible to look at indicators such as MIB counters and statistic counters, but often those are generic and triggered for different reasons, but most importantly their coverage is small, and it's impossible to match a specific packet to a given counter increase.
Socket buffer drop reasons
The socket buffer, SKB (struct sk_buff) is the main data structure representing a packet in the Linux kernel networking stack. When a packet is dropped in the Linux kernel, in most cases, it means its associated socket buffer has dropped. In recent versions of the Linux kernel, starting in v5.17, socket buffers can be dropped with an associated reason. This was introduced in upstream commit c504e5c2f964 ("net: skb: introduce kfree_skb_reason()").
Using this commit and later additions, kernel developers are now able to specify why a given packet dropped. In the following example, a packet is dropped because no socket was found:
- kfree_skb(skb);
+ kfree_skb_reason(skb, SKB_DROP_REASON_NO_SOCKET);
Using tools to retrieve drop reasons
The SKB drop reason can be retrieved in a few different ways, depending on which you are comfortable using, what is available on a given system, and the end goal (some solutions have more flexibility than others).
The main interface to retrieve the drop reason is the skb:kfree_skb tracepoint. It provides a user readable text for all drop reasons. A good way to attach to this tracepoint is to use perf as follows:
# perf record -e skb:kfree_skb curl https://localhost # given no server listens on localhost:443/tcp.
# perf script
curl 883 [001] 340.799805: skb:kfree_skb: skbaddr=0xffff88811f6a7068 protocol=2048 location=tcp_v4_rcv+0x157 reason: NO_SOCKET
curl 883 [001] 340.800860: skb:kfree_skb: skbaddr=0xffff88811f6a6de8 protocol=34525 location=tcp_v6_rcv+0x137 reason: NO_SOCKET
You can see why the two packets where dropped in tcp_v4_rcv and tcp_v6_rcv because no socket was found and we do not have a server listening on localhost:443/tcp.
We can also use other tools such as bpftrace to get the drop reason, which would give us more flexibility, the drawback being the reason isn't converted to a human readable string:
# bpftrace -e 'tracepoint:skb:kfree_skb {printf("%s: %d\n", comm, args->reason)}' -c 'curl https://localhost'
Attaching 1 probe...
curl: 3
curl: 3
curl: (7) Failed to connect to localhost port 443 after 2 ms: Couldn't connect to server
Another method is the dropwatch, an interactive tool to monitor packets dropped in the Linux kernel. When using the packet alert mode, drop reasons are included.
# dropwatch -l kas
Initializing kallsyms db
dropwatch> set alertmode packet
Setting alert mode
Alert mode successfully set
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
drop at: tcp_v4_rcv+0x157/0x1630 (0xffffffff8abc4f87)
origin: software
input port ifindex: 1
timestamp: Thu Feb 23 18:03:36 2023 370138884 nsec
protocol: 0x800
length: 74
original length: 74
drop reason: NO_SOCKET
drop at: tcp_v6_rcv+0x137/0x14f0 (0xffffffff8ad91b37)
origin: software
input port ifindex: 1
timestamp: Thu Feb 23 18:03:36 2023 372335338 nsec
protocol: 0x86dd
length: 94
original length: 94
drop reason: NO_SOCKET
Note:The skb_drop_reason enum defines core drop reasons. It is an internal definition, and the actual value of all its members is not guaranteed to be constant over time. This feature is recent and some of the drop reasons were reordered during development. There is also work ongoing for supporting drop reasons from different subsystems. You should either use tools directly providing the drop reason in a text format (perf or dropwatch) or take the right drop reasons definition as a reference when retrieving the drop reason in a numeric way (bpftrace).
Summary
Not all drop places in the Linux kernel are covered. Converting them to this new facility takes time and resources. There is progress upstream with more additions. Currently, more than 70 reasons are supported. There is also an effort to support more than the core networking subsystem.
SKB drop reasons are now available in Red Hat Enterprise Linux starting with RHEL 8.8 and RHEL 9.2.
