After setting up Netdata on a Ubuntu 18.04.3
server I started getting a lot of warnings regarding dropped packets. The
performance had never shown any problems, but looking at the output of
indeed confirmed netdata's complaints.
Also, only RX was being dropped while TX showed no problems.
While about 3 packets per second dropped might not be a real problem, it certainly didn't seem normal for a server under light load and made me curious to find out what was going on.
Finding the cause
- The softnet backlog full
- Bad VLAN tags
- Packets received with unknown or unregistered protocols
- IPv6 frames when the server is configured only for ipv4
Apart from that, I read that broken hardware could result in packet loss, but that seemed unlikely since everything else was working nicely and the dropping occured only in one direction (RX).
Also ethtool didn't show any errors on the interface:
The softnet backlog can be checked in netdata and no VLAN tags are being used, which made the "unregistered protocols" the most likely candidate.
Using tcpdump I captured packets for about a minute on the server.
This spits out a .pcap file, which can be easily analyzed in
wireshark. Apart from a bit of wireguard traffic,
there where a lot of packets with the protocol
The misterious pakets seemed to always appear in a constant pattern: Three
packets total, the first one being of ethtype
0x88e1 and the follwing to send
to broadcast and with ethtype
0x8912. While wireshark correctly identified the
first protocol I found the other on
wikipedia, "HomePlug AV MME". So what
HomePlug is the family name for various power line communications specifications under the HomePlug designation, with each offering unique performance capabilities and coexistence or compatibility with other HomePlug specifications.
It seems the most common use are powerline adapters used by people with thick walls and weak wireless signals. You plug one end into your NIC and the other in a power outlet, then repeat on the other end (Caution: don't forget the adapter in between).
No such adapters are used in my LAN though. Looking at the MAC adress showed, that they where in fact coming from the router, a quite common FRITZ!Box 7490.
To confirm this is the cause of the dropped packets you can run the following command:
tcpdump -p -i enp6s0 ether proto 0x8912 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp6s0, link-type EN10MB (Ethernet), capture size 262144 bytes 12:20:21.886742 34:31:c4:88:5f:4d (oui Unknown) > Broadcast, ethertype Unknown (0x8912), length 64: 0x0000: 0170 a000 0000 1f84 2aa3 97a2 5553 bef1 .p......*...US.. 0x0010: fcf9 796b 5214 13e9 e200 0000 0000 0000 ..ykR........... 0x0020: 0000 0000 0000 0000 0000 0000 0000 1c00 ................ 0x0030: 22f3 ". 12:20:24.167687 34:31:c4:88:5f:4d (oui Unknown) > Broadcast, ethertype Unknown (0x8912), length 64: 0x0000: 0170 a000 0000 1f84 33a3 97a2 5553 bef1 .p......3...US.. 0x0010: fcf9 796b 5214 13e9 e200 0000 0000 0000 ..ykR........... 0x0020: 0000 0000 0000 0000 0000 0000 0000 05f4 ................ 0x0030: 797c y|
-p flag stops tcpdump from putting the interface into promiscuous mode. If
the packet drops still stop while tcpdump is running, it is a strong indicator,
that the specified type of packets are your problem. You can also see the
incoming packets while the command is running.
After searching though all options in the interface of the router and consulting the manual, I packed up all the needed information and send the manufacturer an email explaining my problem and asking, if it was possible to disable the powerline-adapter-feature. I don't use this kind of hardware and don't intend to do so in the future.
Well, it turns out you can't. This is supposedly a "feature" to make these adapters more plug-and-play.
Dealing with the packets
Since replacing the hardware with something less consumer-oriented wasn't an option at this point, I started to look at server-side solutions. I would probably not be able to convice my router to stop littering the network, but at least I can tell my server to handle this more gracefully.
HomePlug being a layer 2 protocol, there was not the option of just closing a
port and ignoring traffic to it.
iptables can't work with this kind of
packets. Also since the sender is the router (which provides the much needed
internet connection) blocking based on IP or MAC address is also not an option.
I did some research and was lead to
ebtables, which can block packets based on
their ethtype hexcode. According to the man page the following should work:
ebtables -A INPUT -p 8912 -j DROP ebtables -A INPUT -p 88e1 -j DROP
Sadly the drops didn't stop and it seemed like my rules where in place, but not
receiving any packets. The following command shows the
after a while, both being at zero.
After reading the man pages and trying to find my error, I was hinted in the IRC
that ebtables does only work on bridges, not common interfaces. All hope was not
lost though, as
horribly documented successor of the now deprecated
iptables is able to block
protocols based on their ethtype aswell, using the ingress hook. Even better, it
can coexist with my already existing firewall configuration.
With the following configuration placed in
/etc/nftables.conf, both types of
packets are silently dropped:
After that, it was just a matter of starting the
nftables.service and happily
watching how the dropped packets graph fell to zero.