I just received my APU2C4 and I'm trying to set it up as a firewall/router. I updated the BIOS to 20160311, which seems to be the latest, and I installed OpenBSD 5.9 on the internal mSATA SSD. I have almost nothing setup or installed beyond the base OpenBSD install except iperf to test performance. I've tested throughput several times on em0 and em2 interfaces, with iperf, and I seem to be maxing out around 670 mb/s as a server and 420 mb/s as a client. I'm testing to two other very different machines connected via an unmanaged gigabit switch -- an Arch Linux desktop machine and a FreeBSD 10.3 machine. Between the Arch Linux and FreeBSD machines I can get a full ~950 mb/s both ways.
If I run top on the APU while the iperf tests are running, I see that 1 out of the 4 cores is at ~100% usage from a system process -- this bounces around to different cores while it runs. So, it would seem that, at least with low mtu sizes (default 1500), this might be the cap on throughput. Is this an OpenBSD limit? Has anybody gotten better throughput using Linux or FreeBSD/pfSense?
I've tried all of the suggestions on https://calomel.org/network_performance.html as well, but they made no real difference. The only thing that got me any real performance benefit was disabling pf, which I obviously won't do once this is in production.
Once I set the NIC's MTU to 9000, I'm able to saturate the gigabit link, even with pf enabled (albeit with no rules currently). Are there any down sides to having the NICs default to 9000 MTU? I remember having compatibility issues at one point due to "jumbo frames", but I don't remember the specifics of it. I've got a mix of wired and wireless devices in the house including: PCs, game consoles, IP cameras, and Android phones. My WAN link via Comcast is only 125 mbps right now, so even the default MTU will saturate that, but I might be getting a gigabit link in the near future and I don't want the router to be the weak link.
I'm new to OpenBSD, so there's a possibility that I'm missing something obvious, but I've tried to do my homework on it. Thanks for any information.