's journal
networks
Windows routing problem: Network is unreachable
I don’t know what got broken, or how did it get broken but IP connections on my interface which is using AUTO IP (169.254.0.0/16) stopped working all of a sudden.
$ nc -nvvvv 169.254.2.206 23 (UNKNOWN) [169.254.2.206] 23 (?) : Network is unreachable sent 0, rcvd 0
When I explicitly bind to the correct interface it worked.
$ nc -s 169.254.46.68 -nvvvv 169.254.2.206 23 (UNKNOWN) [169.254.2.206] 23 (?) open Welcome to ENDA Administration Terminal Password:
The fact that the error is returned immediately and the content of the error points out that it is a routing problem. So I checked my routing table.
$ route -4 print
===========================================================================
Interface List
37...00 1f 1f ea 7f 52 ......150Mbps Wireless 802.11b/g/n Nano USB Adapter
28...00 ff 2d 67 65 d4 ......TAP-Win32 Adapter V9
10...00 1d ba 68 a5 ac ......Intel(R) 82567LM Gigabit Network Connection
1...........................Software Loopback Interface 1
12...00 00 00 00 00 00 00 e0 Teredo Tunneling Pseudo-Interface
39...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
26...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #3
38...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
===========================================================================
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.1.1 192.168.1.104 25
127.0.0.0 255.0.0.0 On-link 127.0.0.1 306
127.0.0.1 255.255.255.255 On-link 127.0.0.1 306
127.255.255.255 255.255.255.255 On-link 127.0.0.1 306
169.254.46.68 255.255.255.255 On-link 169.254.46.68 356
192.168.1.0 255.255.255.0 On-link 192.168.1.104 281
192.168.1.104 255.255.255.255 On-link 192.168.1.104 281
192.168.1.255 255.255.255.255 On-link 192.168.1.104 281
224.0.0.0 240.0.0.0 On-link 127.0.0.1 306
224.0.0.0 240.0.0.0 On-link 169.254.46.68 356
224.0.0.0 240.0.0.0 On-link 192.168.1.104 281
255.255.255.255 255.255.255.255 On-link 127.0.0.1 306
255.255.255.255 255.255.255.255 On-link 169.254.46.68 356
255.255.255.255 255.255.255.255 On-link 192.168.1.104 281
===========================================================================
Persistent Routes:
Network Address Netmask Gateway Address Metric
0.0.0.0 0.0.0.0 192.168.1.1 Default
0.0.0.0 0.0.0.0 192.168.1.1 Default
===========================================================================
Looks like the problem is
169.254.46.68 255.255.255.255 On-link 169.254.46.68 356
Netmask should have been 255.255.0.0, so that any 169.254.C.D would be carried on with this routing record. So I deleted the faulty line
route -4 delete 169.254.46.68
then issued
route -4 add 169.254.0.0 mask 255.255.0.0 0.0.0.0 metric 3 if 10
10 being my Ethernet adapter as can be seen on the top “route -4 print”. Tricky part for me was to get the gateway was “On-link” in the routing table, after a couple of tries I figured using “0.0.0.0″ as the gateway just makes it “On-link”. After all, whole point of AUTO IP networks is that there’s no router present :)
Now, it works, but after reboots these could be screwed up all over again (despite the fact that route has -p parameter). We’ll see.
Mikrotik RouterOS: How to track each user’s bandwidth usage ?
There are many ways to keep an eye on your users with RouterOS. Two ways to track user bandwidth hoggers:
- Use packet sniffer to collect a sample (about a few minutes when there is problem), then examine the data in Connections and Hosts tab.
- Pros
- Does not consume resources all the time
- Cons
- Not convenient, you have to do a few clicks and wait for some time to see the result.
- You cannot directly act upon the hogger.
- Use Firewall rules to match packets for each host and add rate column.
- Pros
- You can see bandwidth usage of each IP on your network in real time
- You can act upon any user and enforce bandwidth limits
- Cons
- I’ve added ~250 rules to match the traffic for each IP and it caused around ~15% CPU overhead on 300MHz CPU.
As for making all the rules for the firewall. Obviously, I won’t enter those rules by hand, thanks to Mikrotik’s shell it is very easy:
:for i from=2 to=254 do={/ip firewall filter add chain=forward dst-address="193.168.1.$i" action=passthrough}
Passthrough means “do nothing”. We just use the rule to match the packets to each IP then we make use of the rule statistics to get the information we want.
Mysterious performance problems with your good old network
Even though we haven’t changed our network infrastructure for quite some time we started to have network performance problems. Eventually, the network started to stop working from time to time. Router stopped handing out DHCP leases and sometimes freeze all of a sudden.
Not only this, but the networked device which we are developing also started to suffer severely from network problems. The performance was so bad, we couldn’t connect to it over LAN and even if we did, it was a very fragile connection.
We have changed many ADSL Modem routers that you could find on your computer hardware store. None of them could handle it.
Lately, we bought a Mikrotik RB/450. It is a tiny box with 300 MHz CPU and 64MB RAM. With RouterOS installed on it, which is a Linux based solution. I put our ADSL Modem in Bridge Mode, than used the PPPoE client on the RouterOS to connect to the internet. This way, there is basically no load on the ADSL Modem. And voila! Network is performing fantastically.
Couple of observations:
- With around 30 active clients, you get 250-600 active connections in NAT table at any given time.
- Our monthly bandwidth usage is around 190 GB
But why did this started to happen now ?
The reason the routers started falling is probably the evolving technology and internet using habits of users. Browsers are using more and more connections and users are opening more and more tabs. There this web site which claims listing how many simultaneous connections can a router handle. I haven’t checked reliability of their measurement methods but if we assume the list is accurate, you’d see there are plenty of routers that can’t handle 250 connections (which is our minimum).
I’m extremely pleased with the Mikrotik RB/450 and I recommended it to any SOHO.
The performance problem of our networked devices is another story however. Since, our PCs and devices are connected to the same switch, our connection does not even require a router to orchestrate the packets. Only problem a router could cause trouble for us is the DHCP problem, which can be worked around with assigning static IPs.
However, I started to get suspicious about all the broadcast traffic new Windows versions was generating lately. Then, I did some experiments with our networked device which was having serious network problems, I noticed that it works as expected when
- It is directly connected to the PC
- Only the device and the PC was connected to a switch — and nothing else is connected to the switch.
When I plug the company LAN to the switch the device started performing poorly again because it tried to check if all those broadcast messages was actually something useful or not. The problem is, there’s no hardware filtering available on the device. As an additional challenge, hardware buffer of the Ethernet is implemented as a FIFO, so I can’t read arbitrary points in the buffer. So, I have to copy the whole hardware buffer to user space before I can actually filter on it. Filtering is easy and not a problem but copying the data is. I’ll see if I can optimize this in the near future.
Moral of the day, new protocols that use broadcasting aggressively are not good network citizens.