Consultor Eletrônico

Status: Unverified

FACT(s) (Environment):

SonicMQ 4.x
RedHat Linux 7.1

SYMPTOM(s):

SonicMQ broker hangs after 360 TCP connections on Linux

CAUSE:

This is rather a Operating system limit than within the SonicMQ product
like you see on the following link:

http://www.jlinux.org/server.html

In detail you will find on that link the following info:

There are two key limits you may run into while running a Java server on Linux:

1.File descriptor limits
2.Task limits

File descriptor limits

The Linux kernel imposes a limit on the number of file descriptors a given process can have open at once. A
file descriptor is a handle on an open file. These includes sockets, which Java uses for it's TCP/IP communications.

Process/Thread limits

This problem only occurs if you are using native threads. Of course, if you are doing server development, you really want to have access to native
threads because without them you are stuck doing I/O with one client at a time.

A Linux thread and a Linux process look a lot alike. Internally they are both tasks. Linux puts a limit on the maximum number of tasks on a system (512),
and the maximum number of tasks for a given user on a system (256). This means you can't have more than 254 threads in a single program, and you can't
have more than about 450 or so threads on an entire system. Since Java needs threads to do concurrent I/O operations, this can be a real problem.

FIX:

That info is picked from:
http://www.jlinux.org/server.html

The Solution for the File descriptor limits

Make the following changes:

In /usr/include/linux/limits.h:
#define NR_OPEN 1024
should be
#define NR_OPEN 4096
In /usr/include/linux/posix_types.h:
#define __FD_SETSIZE 1024
should be
#define __FD_SETSIZE 4096
In /usr/include/bits/types.h:
#define __FD_SETSIZE 1024
should be
#define __FD_SET_SIZE 4096
Before these changes take affect you have to build a fresh kernel. Please see the Linux Kernel-HOWTO for details on how to do this. Keep in mind you may
want to read up on the solution to the second part of this problem before building a fresh kernel.

You need to give processes the option of increasing their file descriptor limits:
In /etc/security/limits.conf add two lines:
* soft nofile 1024
* hard nofile 4096
In /etc/pam.d/login add:
session required /lib/security/pam_limits.so

Linux also imposes system-wide file descriptor limits. These can be changed at run time, but to automate the process, add the following lines to
/etc/rc.d/rc.local. This will make the change automatically for you every time you boot:
echo 8192 > /proc/sys/fs/file-max
echo 24576 > /proc/sys/fs/inode-max

Solution for Process/Thread limits

Part 1 - the kernel

The easy solution is to move to the 2.4.x kernels, which don't have this problem.

Alternatively, you can modify your existing kernel's limits as follows:
In /usr/include/linux/tasks.h:
#define NR_TASKS 512
should be changed to
#define NR_TASKS 4090
#define MAX_TASKS_PER_USER (NR_TASKS/2)
should be changed to
#define MAX_TASKS_PER_USER NR_TASKS

Part 2 - the runtime

Unfortunately, the Linux kernel isn't the only thing imposing this limit. The standard C library also imposes a limit of 1000 threads per process. So you
have to modify it as well. So, go download your latest copy of Glibc sources and then follow these steps.

In linuxthreads/internals.h
#define STACK_SIZE (2 * 1024 * 1024)
should be changed to
#define STACK_SIZE (64 * PAGE_SIZE)
In linuxthreads/sysdeps/unix/sysv/linux/bits/local_lim.h
#define PTHREAD_THREADS_MAX 1024
should be changed to
#define PTHREAD_THREADS_MAX 8192

After this is completed you can begin the painfull (several hour long process) of rebuilding your glibc shared library.