NPF/interface tuning? shell unusable on gateway

[Jeremy C. Reed]

On same hardware, a week ago I changed my router from a different
operating system to NetBSD/amd64 9.2.

It is running a simple NAT gateway using NPF and also runs dhcpd and
unbound for internal LAN.

Periodically my shells on this new NetBSD router become unusable -- too
slow to type.

The interfaces are:

re0 is my WAN
re0 at pci2 dev 0 function 0: RealTek 8168/8111 PCIe Gigabit Ethernet
(rev. 0x03)
re0: interrupting at msix1 vec 0
re0: using 256 tx descriptors
rgephy0 at re0 phy 7: RTL8211B 1000BASE-T media interface

re1 is my LAN
re1 at pci3 dev 1 function 0: RealTek 8169/8110 Gigabit Ethernet (rev.
0x10)
re1: interrupting at ioapic0 pin 16
re1: using 256 tx descriptors
rgephy1 at re1 phy 7: RTL8211C 1000BASE-T media interface

I can reproduce the problem by starting an rsync (over ssh) within my
LAN transferring to or from outside. I can also reproduce by running
"speedtest-cli" within my LAN.

I cannot reproduce the problem by doing the rsync or speedtest-cli
directly on the NetBSD router itself. So it appears not be the NAT nor
the WAN interface.

While my NetBSD router shell is unusable, I can still use remote SSH
shells fine. That is the part that confuses me, so over the NAT and
over the WAN is okay. Even ssh shell on the remote host rsyncing to or
from is usable while the NetBSD gateway shell is unusable (at the same
time).

There is low cpu load when I have problem.

With rsync across my gateway, if I use --bwlimit 1400k, the problem is
noticable but shell is somewhat usable. --bwlimit 1500k or faster then
shell is unusable.

I tried to watch with sysstat ifstat. It appears to hang when re1 out
(to my LAN) reaches around 10 Mbits/s to 11 Mbits/s. One time the
"systat ifstat 0.01" showed it hanged at out 10.883 Mb/s , peak:
12.196 Mb/s. (But since it hangs, it may not have updated timely.)

The shell hangs immediately when doing the rsync. When I suspend the
rsync, my shell recovers in about 10 seconds. I could reproduce this
many times.

speedtest-cli over LAN shows Download: 6.34 Mbit/s
systat ifstat 0.01 shows peak 24.312 Mb/s

another speedtest-cli run over LAN Download: 9.95 Mbit/s
systat peak 20.981 Mb/s

A speedtest-cli over the LAN using same hardware, same interfaces,
different operating system was Download: 62.72 Mbit/s but that was six
months ago, and different target "best server".

I can also get 18.816 Mb/s traffic from the gateway (not over NAT nor
WAN) to LAN and the NetBSD gateway shell is still usuable, but noticably
laggy. So 1.5 times more bandwidth. So maybe it is the NPF NAT that is
the problem.

My npf.conf is:

$ext_if = "re0"
$int_if = "re1"
$ext_addrs = { ifaddrs($ext_if) }
$localnet = { 172.16.1.0/24 }

alg "icmp"

map inet4($ext_if) dynamic $localnet -> inet4($ext_if)

group "external" on $ext_if {
pass stateful out all
block in all
}

group "internal" on $int_if {
pass final all
}

group default {
pass final on lo0 all
block all
}

I am unsure if the NPF is the problem, and maybe my interface has a
problem, but it was working fine for me to login and use the shell on
the system locally fine many times before I put NetBSD on it.

Any suggestions on tuning so my shell on the router is usable?

Here is "sysstat vmstat 0.01" when it hangs:

4 users Load 0.12 0.05 0.05 Sat Mar 26 18:31:58

Proc:r d s Csw Traps SysCal Intr Soft Fault PAGING SWAPPING
1 6 114 1193 1200 1000 in out in out
ops
14.3% Sy 0.0% Us 0.0% Ni 3.6% In 82.1% Id pages
| | | | | | | | | | |
=======%% forks
fkppw
Anon 130180 4% zero 302356 1250 Interrupts fksvm
Exec 24816 % wired 24 TLB shootdown pwait
File 1831888 61% inact 671384 100 cpu0 timer relck
Meta 409088 % bufs 89448 336 ioapic0 pin 16 rlkok
(kB) real swaponly free ioapic0 pin 18 noram
Active 1315476 331500 814 msix1 vec 0 ndcpy
Namei Sys-cache Proc-cache ioapic0 pin 23 fltcp
Calls hits % hits % ioapic0 pin 19 zfod
6 6 100 cow
512 fmin
Disks: sd0 wd0 dk0 dk1 682 ftarg
seeks itarg
xfers flnan
bytes pdfre
%busy

Any suggestions on how I can better diagnose this?

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Jeremy C. Reed]

Are there any packet drops or other errors? `sysctl net.interfaces`,
`sysctl net.inet6.ip6.ifq`, `sysctl net.inet.ip.ifq`, and `netstat -dvI
re0; netstat -dvI re1` may be revealing.

David, thank you for the feedback and hints. I switched re1 to outside
and re0 to my WAN after my previous email just to see if there was any
change. It still had same problem. But you helped me track down to what
it appears to be one link and one service with a problem. I will
provide answers below first:

$ sysctl net.interfaces
net.interfaces.athn0.rcvq.drops = 0
net.interfaces.athn0.sndq.len = 0
net.interfaces.athn0.sndq.maxlen = 256
net.interfaces.athn0.sndq.drops = 0
net.interfaces.re0.rcvq.drops = 0
net.interfaces.re0.sndq.len = 0
net.interfaces.re0.sndq.maxlen = 512
net.interfaces.re0.sndq.drops = 0
net.interfaces.re1.rcvq.drops = 0
net.interfaces.re1.sndq.len = 0
net.interfaces.re1.sndq.maxlen = 512
net.interfaces.re1.sndq.drops = 0
net.interfaces.lo0.rcvq.drops = 0
net.interfaces.lo0.sndq.len = 0
net.interfaces.lo0.sndq.maxlen = 256
net.interfaces.lo0.sndq.drops = 0

$ sysctl net.inet6.ip6.ifq
net.inet6.ip6.ifq.len = 0
net.inet6.ip6.ifq.maxlen = 256
net.inet6.ip6.ifq.drops = 0

(I am not purposely using IPv6.)

$ sysctl net.inet.ip.ifq
net.inet.ip.ifq.len = 0
net.inet.ip.ifq.maxlen = 256
net.inet.ip.ifq.drops = 0

$ netstat -dvI re0; netstat -dvI re1
Name Mtu Network Address Ipkts Ierrs Idrops Opkts Oerrs Colls Odrops
re0 1500 <Link> b8:ac:6f:df:49:9d 19529216 0 0 31150053 0 0 0
re0 1500 172.16/16 172.16.1.1 19529216 0 0 31150053 0 0 0
re0 1500 fe80::%re0/64 fe80::baac:6fff:fedf:499d%re0 19529216 0 0 31150053 0 0 0
Name Mtu Network Address Ipkts Ierrs Idrops Opkts Oerrs Colls Odrops
re1 1500 <Link> f4:f2:6d:00:b7:57 30856346 0 0 19509672 0 0 0
re1 1500 fe80::%re1/64 fe80::200e:d2e4:6900:afc6%re1 30856346 0 0 19509672 0 0 0
re1 1500 47.185.18/24 47.185.18.26 30856346 0 0 19509672 0 0 0

What link speed is negotiated on WAN and LAN ports?
Is any flow-control negotiated?

My LAN interface:
$ ifconfig re0
re0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500

capabilities=3f80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
capabilities=3f80<UDP4CSUM_Rx,UDP4CSUM_Tx>
enabled=0
ec_capabilities=3<VLAN_MTU,VLAN_HWTAGGING>
ec_enabled=0
address: b8:ac:6f:df:49:9d
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet 172.16.1.1/16 broadcast 172.16.255.255 flags 0x0
inet6 fe80::baac:6fff:fedf:499d%re0/64 flags 0x0 scopeid 0x2

My interface to outside:
$ ifconfig re1
re1: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500

capabilities=3f80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
capabilities=3f80<UDP4CSUM_Rx,UDP4CSUM_Tx>
enabled=0
ec_capabilities=3<VLAN_MTU,VLAN_HWTAGGING>
ec_enabled=0
address: f4:f2:6d:00:b7:57
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 47.185.18.26/24 broadcast 47.185.18.255 flags 0x0

It sounds like the LAN is quite slow? I may have misunderstood. Is the
LAN all wired or is there any wireless involved?

Currently the LAN goes to a wireless router (then is double NAT) and it
is primarily used with wifi. It also has a few ethernet from it
including to my main workstation (which happens to be over
ethernet-over-power).

So tried to reproduce problem on a laptop over wifi (second LAN router)
and no problem.

I bypassed both wifi and ethernet-over-power and had no problem.

I bypassed second LAN router and did have problem over
ethernet-over-power.

I used a different computer over the second router and over
ethernet-over-power and did have the problem.

So the problem is over the ethernet-over-power (regardless if it goes
through second router or not).

The problem is: using shell on the netbsd router is basically locked up,
not just slow but entirely unusable, for the same client over
ethernet-over-power that is also doing some downloads/uploads.

I have had a ethernet-over-power fail before and I replaced
it.

I am confused by some things:

- I didn't see the problem before my change to NetBSD as the router.
Maybe I didn't use it enough before to notice it (but had used it over
5+ years). Maybe ethernet-over-power just started failing recent so was
a coincidence.

- While the ssh hangs (even on different port), I can use echo (7/tcp),
chargen (19/tcp), and telnet services fine from the same client to
the same system. (sshd on different port and via inetd does not work.)

- I don't understand why even though my same client cannot use the
NetBSD router's shell, I can route through it fine and use outside shell
fine at same time.

I will replace the ethernet-over-power, but I wonder still how I can
tune my NetBSD router so I can use ssh to it. Maybe some quality of
service configuration. Meanwhile I can use telnet to use my router :(

When unusable, pings to it from same client are like
64 bytes from 172.16.1.1: icmp_seq=32 ttl=254 time=426 ms
64 bytes from 172.16.1.1: icmp_seq=33 ttl=254 time=401 ms
64 bytes from 172.16.1.1: icmp_seq=34 ttl=254 time=329 ms
64 bytes from 172.16.1.1: icmp_seq=35 ttl=254 time=386 ms

While normal is:
64 bytes from 172.16.1.1: icmp_seq=36 ttl=254 time=4.08 ms
64 bytes from 172.16.1.1: icmp_seq=37 ttl=254 time=3.37 ms

Any ideas why telnet works slowly but ssh does not at all in these
cases? telnet is usable but cannot even see one character sent over ssh
when ssh locks up (again it restores about 5 to 10 seconds after I stop
or suspend a speedtest or rsync job).

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[RVP]

Post by Jeremy C. Reed
Any ideas why telnet works slowly but ssh does not at all in these
cases? telnet is usable but cannot even see one character sent over ssh
when ssh locks up (again it restores about 5 to 10 seconds after I stop
or suspend a speedtest or rsync job).

Could be a QoS issue when a lot of packets are being xferred.

Try out a few different QoS options (explicitly) in ssh (though it should
already be setting some kind of low-delay one by default):

ssh -oIPQoS='lowdelay' ...

Adding QoS rules to PF/NPF might also help. pf.conf(5) has a bare-bones
example.

-RVP


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Jeremy C. Reed]

Post by RVP

Post by Jeremy C. Reed
Any ideas why telnet works slowly but ssh does not at all in these
cases? telnet is usable but cannot even see one character sent over ssh
when ssh locks up (again it restores about 5 to 10 seconds after I stop
or suspend a speedtest or rsync job).

Could be a QoS issue when a lot of packets are being xferred.
Try out a few different QoS options (explicitly) in ssh (though it should
ssh -oIPQoS='lowdelay' ...

Thank you! That did not work for me for the client, but gave me a hint.
The two clients I tried this from were Ubuntu Linux. At least one of
them defaulted to that already. I don't think the QoS tagging or DCSP is
honored on NetBSD by default.

All of these work:

1) Connect to NetBSD router via telnet, then connect to itself again
using ssh (so NetBSD ssh using defaults for ssh/sshd).

2) Run the NetBSD sshd with sshd_config "IPQoS none" instead of default
of "af21 cs1" (first is for interactive) and use ssh (as is) from my
Linux client.

3) Run the NetBSD sshd with sshd_config "IPQoS lowdelay throughput"
instead of default of "af21 cs1" and use ssh from my Linux client.

I wonder if the "af21" default on NetBSD sshd doesn't work as expected.

Thanks again for the hint.

I will need to understand the Ubuntu side better as it doesn't appear to
have any iptables rules other than default ACCEPTs so I don't think it
has any QoS. Maybe the ssh client (even when set to none) also honors
the server-side sshd tagging and not the Linux kernel.

Post by RVP
Adding QoS rules to PF/NPF might also help. pf.conf(5) has a bare-bones
example.

I may try it later, but now I think the QoS is done on the Linux system.

(I had done lots of testing with dscp with BIND named and also
extensively tested and wrote about it for pfsense. I had no idea it was
in sshd/ssh nor did I think I had anything utilizing it.)

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Greg Troxel]

Post by Jeremy C. Reed
(I had done lots of testing with dscp with BIND named and also
extensively tested and wrote about it for pfsense. I had no idea it was
in sshd/ssh nor did I think I had anything utilizing it.)

I am not surprised if ssh is setting codepoints that basically are "low
delay" for interactive use and "high throughput" for bulk data.

I am surprised that anything is paying attention to DSCP codepoints
(said as someone who has implemented multiple research systems with new
prioritization controls). I am even further surprised that the
codepoint used for ssh low delay would lead to bad behavior especially
in the absence of congestion.

It will be very interesting if you figure out what's going on; please
post a followup.

[Jeremy C. Reed]

Post by Greg Troxel
I am surprised that anything is paying attention to DSCP codepoints
(said as someone who has implemented multiple research systems with new
prioritization controls). I am even further surprised that the
codepoint used for ssh low delay would lead to bad behavior especially
in the absence of congestion.
It will be very interesting if you figure out what's going on; please
post a followup.

Well I thought my Tenda P1000 Powerline adapter was somewhat dumb. I
didn't know it had a management interface and now I see very brief docs
that it has QoS. https://www.tendacn.com/faq/2673.html
Maybe it handles tos 0x48 different than expected. With tos 0x10, it is
adequate. I may replace these paired devices. Any suggestions?

I am using ethernet-over-power until I can get a strong NetBSD based
wifi access point.









--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de