Linux Can't Make new connections, network choked?

G

Gaurish

Level F
Problem:

After 2-3days of Uptime. my home box is unable to to make new connections. any attempt to make a connection simply times out. however existing connections keep functioning properly. means network is live and I get replies to my pings but can't make a new connection. Its a BSNL bb over Enternet

see i can ping but can't connect:|

Code: 
> ping google.com

PING google.com (72.14.204.103) 56(84) bytes of data.

64 bytes from iad04s01-in-f103.1e100.net (72.14.204.103): icmp_seq=1 ttl=52 time=389 ms

64 bytes from iad04s01-in-f103.1e100.net (72.14.204.103): icmp_seq=2 ttl=52 time=386 ms

64 bytes from iad04s01-in-f103.1e100.net (72.14.204.103): icmp_seq=3 ttl=52 time=392 ms

64 bytes from iad04s01-in-f103.1e100.net (72.14.204.103): icmp_seq=4 ttl=52 time=384 ms

^C

--- google.com ping statistics ---

5 packets transmitted, 4 received, 20% packet loss, time 3999ms

rtt min/avg/max/mdev = 384.033/388.206/392.555/3.352 ms

> telnet google.com 80

Trying 72.14.204.99...

Connection failed: Connection timed out

Trying 72.14.204.147...

Connection failed: Connection timed out

What I have tried:

* Restarting Routers

* network.

Sysinfo:

Distro:ArchLinux x86_64

Kernel:2.6.32

Before restarting, I did 'netstat -a' and saved the output. http://paste2.org/p/765884 maybe that might help

Currently only option is to reboot the machine.after which everything is normal again.

Any Idea what's causing this?
 
Is that the full output you posted? Doesnt look suspect to me... (though I'm no expert)... can you look up the "ulimit" command and set yourself to "unlimited" quota? This is for file descriptor (and socket) limits... so setting it to unlimited may give your machine time to recover, else it could indicate a network device/driver problem... (which mobo/NIC do you have?)...

edit: also check if any firewalls are running, try restarting that too?
 
vishalrao said:
Is that the full output you posted? Doesnt look suspect to me... (though I'm no expert)...
Click to expand...
Yes, its the full output
can you look up the "ulimit" command and set yourself to "unlimited" quota?
Click to expand...
Okay, but to verify I need to wait until this problem occurs again.
else it could indicate a network device/driver problem... (which mobo/NIC do you have?)...
Click to expand...
Mobo:GA-G31M-ES2L (rev. 1.x)
NIC: Realtek RTL8111C
edit: also check if any firewalls are running, try restarting that too?
Click to expand...
NA
No firewalls are running.
 
ok, but to "verify" after it occurs you'll need to reboot anway... then you just need to wait and see if it happens again after setting ulimit to unlimited... :)

reason i asked for the mobo/NIC maybe you can search for the model number and linux online to see if others are reporting same issues with this particular kernel version...
 
do you have torrent running? it could cause connections to exhaust

if that netstat is from an idle comp, there there are way too many conns in connected state imho
 
or stuff like choqok :) i wonder if "unix domain sockets" (the ones that are so many connected state) cause any problems like the tcp sockets?
 
vishalrao said:
ok, but to "verify" after it occurs you'll need to reboot anway... then you just need to wait and see if it happens again after setting ulimit to unlimited... :)
Click to expand...

It already defaults to unlimited by default:S

Code: 
gaurish ~ > ulimit

unlimited

nukeu666 said:
do you have torrent running? it could cause connections to exhaust

if that netstat is from an idle comp, there there are way too many conns in connected state imho
Click to expand...

No, no torrents were running. I think I would also agree there were too many UNIX sockets connections, But I don't what's causing this:|
 
What do the following say just after you make an unsuccessful attempt at a network connection?

1. dmesg
2. /var/log/messages (or the equivalent in arch)

As a possible workaround, try "simulating" a reboot by doing network related stuff:

1. optionally, quit applications that use network - browser / torrent client / IM etc.
2. ifdown eth0 (or whatever your network device is)
3. stop the service network/NetworkManager (you mention restarting network alone)
4. Unload the kernel module required for you LAN card

Then start them again in reverse order
 
I am starting to think this is a error in r8169 network card kernel module. what you think?

amitkher said:
What do the following say just after you make an unsuccessful attempt at a network connection?

1. dmesg

2. /var/log/messages (or the equivalent in arch)

As a possible workaround, try "simulating" a reboot by doing network related stuff:

1. optionally, quit applications that use network - browser / torrent client / IM etc.

2. ifdown eth0 (or whatever your network device is)

3. stop the service network/NetworkManager (you mention restarting network alone)

4. Unload the kernel module required for you LAN card

Then start them again in reverse order
Click to expand...

Okay tri.ed everything you said. After I did rmmod & modprobe the network card module and restarted the network.

ifconfig failed to bring up the network interface

Code: 
# ifconfig eth0 up

SIOCSIFFLAGS: Cannot allocate memory

Here is call trace of the error:

Code: 
ifconfig: page allocation failure. order:3, mode:0x4020

Pid: 13841, comm: ifconfig Tainted: P           2.6.33-ARCH #1

Call Trace:

 [<ffffffff810e345b>] __alloc_pages_nodemask+0x6bb/0x700

 [<ffffffff810e34b2>] __get_free_pages+0x12/0x50

 [<ffffffff81115105>] __kmalloc_track_caller+0x125/0x220

 [<ffffffff812ac58f>] ? __netdev_alloc_skb+0x1f/0x40

 [<ffffffff812ab80f>] __alloc_skb+0x6f/0x170

 [<ffffffff812ac58f>] __netdev_alloc_skb+0x1f/0x40

 [<ffffffffa005052b>] rtl8169_rx_fill+0xab/0x240 [r8169]

 [<ffffffff81011860>] ? nommu_map_page+0x0/0xc0

 [<ffffffffa0052ede>] rtl8169_init_ring+0x6e/0xa0 [r8169]

 [<ffffffffa0053a42>] rtl8169_open+0x142/0x460 [r8169]

 [<ffffffff812b7069>] dev_open+0x99/0xf0

 [<ffffffff812b638c>] dev_change_flags+0x9c/0x1d0

 [<ffffffff8131127e>] devinet_ioctl+0x61e/0x760

 [<ffffffff81312960>] inet_ioctl+0x80/0xa0

 [<ffffffff812a01cb>] sock_do_ioctl+0x2b/0x60

 [<ffffffff812a04b1>] sock_ioctl+0x71/0x290

 [<ffffffff810f90de>] ? handle_mm_fault+0x19e/0xa30

 [<ffffffff811324c8>] vfs_ioctl+0x38/0xd0

 [<ffffffff81132670>] do_vfs_ioctl+0x80/0x560

 [<ffffffff811d1276>] ? __up_read+0xa6/0xd0

 [<ffffffff81077e09>] ? up_read+0x9/0x10

 [<ffffffff81132bd1>] sys_ioctl+0x81/0xa0

 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b

Mem-Info:

DMA per-cpu:

CPU    0: hi:    0, btch:   1 usd:   0

CPU    1: hi:    0, btch:   1 usd:   0

DMA32 per-cpu:

CPU    0: hi:  186, btch:  31 usd: 133

CPU    1: hi:  186, btch:  31 usd: 171

active_anon:106271 inactive_anon:46992 isolated_anon:0

 active_file:79648 inactive_file:79905 isolated_file:0

 unevictable:131346 dirty:3117 writeback:0 unstable:0

 free:15704 slab_reclaimable:9148 slab_unreclaimable:5050

 mapped:30774 shmem:419 pagetables:8650 bounce:0

DMA free:8036kB min:40kB low:48kB high:60kB active_anon:80kB inactive_anon:1104kB active_file:2688kB inactive_file:3684kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15296kB mlocked:0kB dirty:116kB writeback:0kB mapped:1240kB shmem:0kB slab_reclaimable:128kB slab_unreclaimable:40kB kernel_stack:0kB pagetables:12kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

lowmem_reserve[]: 0 2003 2003 2003

DMA32 free:54780kB min:5704kB low:7128kB high:8556kB active_anon:425004kB inactive_anon:186864kB active_file:315904kB inactive_file:315808kB unevictable:525384kB isolated(anon):0kB isolated(file):132kB present:2051180kB mlocked:525384kB dirty:12352kB writeback:0kB mapped:121856kB shmem:1676kB slab_reclaimable:36464kB slab_unreclaimable:20160kB kernel_stack:2712kB pagetables:34588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:33 all_unreclaimable? no

lowmem_reserve[]: 0 0 0 0

DMA: 15*4kB 17*8kB 18*16kB 12*32kB 16*64kB 16*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 8036kB

DMA32: 6192*4kB 2875*8kB 432*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54904kB

177321 total pagecache pages

17362 pages in swap cache

Swap cache stats: add 478548, delete 461186, find 275850/300912

Free swap  = 607356kB

Total swap = 996020kB

524000 pages RAM

22245 pages reserved

350106 pages shared

246346 pages non-shared
 
Gaurish said:
Code: 
# ifconfig eth0 up
SIOCSIFFLAGS: Cannot allocate memory
Click to expand...

That error "usually" occurs when a normal user executes the command. Switch to root and then try to bring up the interface.
 
Yeah, could be the kernel module problem.

This seems to be the most likely explanation: Linux-Kernel: Re: b44 ifconfig fails with ENOMEM. Though the solution and the exact problem are both obsolete now. Definitely deserves to be filed as a bug on the ARCH kernel you are using. If this is the problem, "simulating" reboot as I said earlier, won't work.

Maybe the network card requires memory in lowmem. There might be some conflict with some other driver, which also needs memory in lowmem, or some application which doesn't need lowmem acquired lowmem.

You could try playing with lowmem_reserve_ratio. First check what it's value is by
Code: 
sudo sysctl -a | grep  lowmem_reserve_ratio

If it is something like "256 256 32", try halving it by, say,
Code: 
sudo sysctl -w vm.lowmem_reserve_ratio="128   128     16"

I don't understand these settings very well, but I guess this setting will at worst cause your system to crash unexpectedly sometimes, but at best fix your network card issue. Don't do it if it is a critical machine, but I guess if it were critical you wouldn't be running 2.6.33 on it :).

If this started happening after some kernel update - you could try going back to a kernel version before that. Otherwise, try getting differently configured kernel (of the same version) and the problem might go away. If not, keep updating the kernel.

Try compiling the ethernet kernel module yourself. If it is a realtek card, get the source from realtek's website. Distro kernel modules are typically composite of many drivers - work for a large variety of cards. Grab the driver source for your specific network card and build it - should be reasonably simple.
 
Okay, did some more digging.

I think this has something to do with Virtualbox. Because I can't connect on Linux host.but network works(incl new making connections) on guest machine(Winxp) inside virtualbox. I am using bridge networking to connect connect guest and host.

any idea how to explain this?
 
if net connection is not all working on bridge mode ,it may be because of the priority of loading various vbox modules as per their forum.
Check through this:
bridge mode eth0 does not work (View topic) &bull; virtualbox.org
EDIT:
Saw this post :
VirtualBox 3.0: Problems with bridged network connection (View topic) &bull; virtualbox.org

from your first post,I was doubting the cause may be MTU Value.If above solution don't work for you ,Try Changing MTU of Lan Interface(Try decreasing the value in steps of 10 ,supposing the default value on ubuntu is:1492)

G'Luck
 
Hmmm...Changed the network card to Intel Pro Sever, problem seems to be gone. Will bump this thread I get the problem again.

Okay, It reappeared or maybe it wasn't gone. post the logs again, maybe it would help to find out the issue.

Posting these from virtualbox, while network on host does not work.

Interesting thing from dmesg:
Code: 
WARNING! Changing of MTU on this NICMay lead to frame reception errors!

I can one related things here:

Wonder if these have anything to do with the problem I am facing?


full Dmesg output
> dmesg Initializing cgroup s - Anonymous - nw3jWt2S - Pastebin.com

ifconfig

Code: 
eth0      Link encap:Ethernet  HWaddr 00:24:1D:F6:32:E2  
          inet addr:192.168.1.6  Bcast:255.255.255.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1454  Metric:1
          RX packets:11657867 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8077176 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:6022466991 (5743.4 Mb)  TX bytes:634490407 (605.0 Mb)
          Interrupt:27 Base address:0xc000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:30984 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30984 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2419122 (2.3 Mb)  TX bytes:2419122 (2.3 Mb)
netstat -a
> netstat -a Active Internet - Anonymous - yCvKc9cT - Pastebin.com
 
Back
Top