This is an old revision of the document!
There are two cause that may lead to memory starvation:
Stop all traffic to the system and wait about 20 minutes. This will release memory allocated to transactions etc. Test the system again by placing a small number of calls. If the errors start immediately, there is a memory leak. If you are not sure of the result, consider it as a memory leak.
If the memory pool is too small, adjust the value for package or shared memory and restart kamailio. On the command line this is done using the -M and -m switches for pkg and shared mem respectively. When installed as an RPM on RedHat or CentOS, these values may be set in /etc/sysconfig/kamailio
This requires memory debug support compiled into the application. run kamailio -V
version: kamailio 4.2.0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 11:59:44 Nov 3 2014 with gcc 4.4.7
This example was compiled using the F_MALLOC memory manager and has debug support enabled. The alternative memory manager will show QM_MALLOC and DBG_QM_MALLOC.
If the memory debug support is not compiled, follow the next steps:
Now, you may check the memory status in two situations:
0(17665) Memory status (shm): 0(17665) qm_status (0xb5a7e000): 0(17665) heap size= 33554432 0(17665) used= 1592952, used+overhead=1811564, free=31742868 0(17665) max used (+overhead)= 1811564 0(17665) dumping all alloc'ed. fragments: 0(17665) 0. N address=0xb5ab240c frag=0xb5ab23f4 size=4 used=1 0(17665) alloc'd from mem/shm_mem.c: shm_mem_init_mallocs(199) 0(17665) start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 0(17665) 1. N address=0xb5ab2440 frag=0xb5ab2428 size=4 used=1 0(17665) alloc'd from timer.c: init_timer(52) 0(17665) start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
If you have no clue how to interpret the logs for memory status, post it on a FTP or HTTP server and send the link on the devel@lists.kamailio.org mailing list.
With versions > 3.x, if you have the ctl and cfg_rpc modules loaded, execute: sercmd cfg.set_now_int core mem_dump_pkg <pid> When processing of first SIP message in the process with that PID, you will get the status of pkg memory in syslog.
These gdb scripts can be used on a dumped core file or when attached to a running process that has excessive memory used.
Print used chunks of the first 2000 to 10000 if compiled with DBG_QM_MALLOC:
set $i=0 set $a = mem_block->first_frag while($i<10000) if($i>2000) if($a->u.is_free==0) p *$a end end set $a = ((struct qm_frag*)((char*)($a)+sizeof(struct qm_frag)+((struct qm_frag*)$a)->size+sizeof(struct qm_frag_end))) set $i = $i + 1 end
Print used chunks of the first 2000 to 10000 if compiled with DBG_F_MALLOC:
set $i=0 set $a = mem_block->first_frag while($i<10000) if($i>2000) if($a->u.reserved==0) p *$a end end set $a = ((struct fm_frag*)((char*)($a)+sizeof(struct fm_frag)+((struct fm_frag*)$a)->size)) set $i = $i + 1 end