After setting up Netdata on a Ubuntu 18.04.3 server I started getting a lot of warnings regarding dropped packets. The performance had never shown any problems, but looking at the output of ifconfig indeed confirmed netdata's complaints.

RX packets 617659  bytes 350097816 (350.0 MB)
RX errors 0  dropped 13262  overruns 0  frame 0
TX packets 567456  bytes 374999164 (374.9 MB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Also, only RX was being dropped while TX showed no problems.

cat /sys/class/net/enp6s0/statistics/rx_dropped
743836
cat /sys/class/net/enp6s0/statistics/tx_dropped
0

While about 3 packets per second dropped might not be a real problem, it certainly didn't seem normal for a server under light load and made me curious to find out what was going on.

Finding the cause

After spending way too much time reading though different websites and blogs I narrowed down the causes to the ones named here

  • The softnet backlog full
  • Bad VLAN tags
  • Packets received with unknown or unregistered protocols
  • IPv6 frames when the server is configured only for ipv4

Apart from that, I read that broken hardware could result in packet loss, but that seemed unlikely since everything else was working nicely and the dropping occured only in one direction (RX).

Also ethtool didn't show any errors on the interface:

ethtool -S enp6s0
NIC statistics:
     tx_packets: 755651
     rx_packets: 818023
     tx_errors: 0
     rx_errors: 0
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 786013
     broadcast: 31750
     multicast: 627
     tx_aborted: 0
     tx_underrun: 0

The softnet backlog can be checked in netdata and no VLAN tags are being used, which made the "unregistered protocols" the most likely candidate.

Using tcpdump I captured packets for about a minute on the server.

ethtool -K enp6s0 rx-fcs on
ethtool -K enp6s0 rx-all on
tcpdump -w capture1.pcap

This spits out a .pcap file, which can be easily analyzed in wireshark. Apart from a bit of wireguard traffic, there where a lot of packets with the protocol HomePlug AV.

Wireshark

The misterious pakets seemed to always appear in a constant pattern: Three packets total, the first one being of ethtype 0x88e1 and the follwing to send to broadcast and with ethtype 0x8912. While wireshark correctly identified the first protocol I found the other on wikipedia, "HomePlug AV MME". So what is "HomePlug"?

HomePlug is the family name for various power line communications specifications under the HomePlug designation, with each offering unique performance capabilities and coexistence or compatibility with other HomePlug specifications.

It seems the most common use are powerline adapters used by people with thick walls and weak wireless signals. You plug one end into your NIC and the other in a power outlet, then repeat on the other end (Caution: don't forget the adapter in between).

No such adapters are used in my LAN though. Looking at the MAC adress showed, that they where in fact coming from the router, a quite common FRITZ!Box 7490.

To confirm this is the cause of the dropped packets you can run the following command:

tcpdump -p -i enp6s0 ether proto 0x8912

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp6s0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:20:21.886742 34:31:c4:88:5f:4d (oui Unknown) > Broadcast, ethertype Unknown (0x8912), length 64: 
	0x0000:  0170 a000 0000 1f84 2aa3 97a2 5553 bef1  .p......*...US..
	0x0010:  fcf9 796b 5214 13e9 e200 0000 0000 0000  ..ykR...........
	0x0020:  0000 0000 0000 0000 0000 0000 0000 1c00  ................
	0x0030:  22f3                                     ".
12:20:24.167687 34:31:c4:88:5f:4d (oui Unknown) > Broadcast, ethertype Unknown (0x8912), length 64: 
	0x0000:  0170 a000 0000 1f84 33a3 97a2 5553 bef1  .p......3...US..
	0x0010:  fcf9 796b 5214 13e9 e200 0000 0000 0000  ..ykR...........
	0x0020:  0000 0000 0000 0000 0000 0000 0000 05f4  ................
	0x0030:  797c                                     y|

The -p flag stops tcpdump from putting the interface into promiscuous mode. If the packet drops still stop while tcpdump is running, it is a strong indicator, that the specified type of packets are your problem. You can also see the incoming packets while the command is running.

After searching though all options in the interface of the router and consulting the manual, I packed up all the needed information and send the manufacturer an email explaining my problem and asking, if it was possible to disable the powerline-adapter-feature. I don't use this kind of hardware and don't intend to do so in the future.

Well, it turns out you can't. This is supposedly a "feature" to make these adapters more plug-and-play.

Dealing with the packets

Since replacing the hardware with something less consumer-oriented wasn't an option at this point, I started to look at server-side solutions. I would probably not be able to convice my router to stop littering the network, but at least I can tell my server to handle this more gracefully.

HomePlug being a layer 2 protocol, there was not the option of just closing a port and ignoring traffic to it. iptables can't work with this kind of packets. Also since the sender is the router (which provides the much needed internet connection) blocking based on IP or MAC address is also not an option.

I did some research and was lead to ebtables, which can block packets based on their ethtype hexcode. According to the man page the following should work:

ebtables -A INPUT -p 8912 -j DROP
ebtables -A INPUT -p 88e1 -j DROP

Sadly the drops didn't stop and it seemed like my rules where in place, but not receiving any packets. The following command shows the pcnt and bcnt counters after a while, both being at zero.

$ ebtables -L --Lc
Bridge table: filter
Bridge chain: INPUT, entries: 3, policy: ACCEPT
-p 0x8912 -j DROP , pcnt = 0 -- bcnt = 0
-p 0x88e1 -j DROP , pcnt = 0 -- bcnt = 0
Bridge chain: FORWARD, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

After reading the man pages and trying to find my error, I was hinted in the IRC that ebtables does only work on bridges, not common interfaces. All hope was not lost though, as nftables the horribly documented successor of the now deprecated iptables is able to block protocols based on their ethtype aswell, using the ingress hook. Even better, it can coexist with my already existing firewall configuration.

With the following configuration placed in /etc/nftables.conf, both types of packets are silently dropped:

#!/usr/sbin/nft -f

flush ruleset

table netdev filter {
    chain ingress {
        type filter hook ingress device enp6s0 priority 0; policy accept;
        meta protocol {0x8912, 0x88e1} drop
    }
}

After that, it was just a matter of starting the nftables.service and happily watching how the dropped packets graph fell to zero.

Wireshark