This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
tutorials:troubleshooting:memory [2015/09/04 00:01] miconda [Memory Manager Debugging] |
tutorials:troubleshooting:memory [2017/02/15 13:27] miconda [Memory Manager Debugging] |
||
---|---|---|---|
Line 46: | Line 46: | ||
< | < | ||
kamctl stats shmem | kamctl stats shmem | ||
+ | kamcmd mod.stats all shm | ||
</ | </ | ||
Line 52: | Line 53: | ||
< | < | ||
kamcmd pkg.stats | kamcmd pkg.stats | ||
+ | kamcmd mod.stats all pkg | ||
</ | </ | ||
Notice that for SHM only one group of statistics is printed, being one zone of memory, while for PKG you get a list with many groups of statistics, each specific for a Kamailio process (child). | Notice that for SHM only one group of statistics is printed, being one zone of memory, while for PKG you get a list with many groups of statistics, each specific for a Kamailio process (child). | ||
+ | |||
+ | In order to merge the free memory fragments one should enable memory join. This is enable by default in latest kamailio: | ||
+ | < | ||
+ | mem_join=1 | ||
+ | </ | ||
+ | |||
===== Analysis of Memory Incidents ===== | ===== Analysis of Memory Incidents ===== | ||
Line 77: | Line 85: | ||
< | < | ||
- | MEMMNG=1 | + | MEMDBG=1 make cfg ... |
</ | </ | ||
Line 98: | Line 106: | ||
Then restart and wait a bit for getting some traffic processed. | Then restart and wait a bit for getting some traffic processed. | ||
+ | |||
+ | MEMMNG=0/ | ||
To get the list of chunks from memory manager, there are two ways: | To get the list of chunks from memory manager, there are two ways: | ||
Line 135: | Line 145: | ||
For PKG is similar format, just SHM replaced with PKG in messages. | For PKG is similar format, just SHM replaced with PKG in messages. | ||
+ | To generate summary report, do: | ||
+ | |||
+ | < | ||
+ | # first set memlog lower than debug | ||
+ | kamcmd cfg.set_now_int core memlog 1 | ||
+ | |||
+ | kamcmd corex.shm_summary | ||
+ | </ | ||
+ | |||
+ | The log for f_malloc with debug enabled should look like: | ||
+ | |||
+ | < | ||
+ | 20(4082) NOTICE: fm_status: summarizing all alloc' | ||
+ | 20(4082) NOTICE: fm_status: | ||
+ | 20(4082) NOTICE: fm_status: | ||
+ | 20(4082) NOTICE: fm_status: | ||
+ | 20(4082) NOTICE: fm_status: | ||
+ | 20(4082) NOTICE: fm_status: | ||
+ | 20(4082) NOTICE: fm_status: | ||
+ | 20(4082) NOTICE: fm_status: | ||
+ | 20(4082) NOTICE: fm_status: | ||
+ | 20(4082) NOTICE: fm_status: | ||
+ | </ | ||
+ | |||
+ | If you dumped the status with qm_malloc, you can extract the logs from syslog and count the unique allocations with next commands: | ||
+ | |||
+ | < | ||
+ | grep qm_status / | ||
+ | |||
+ | # or: | ||
+ | # grep qm_status / | ||
+ | |||
+ | grep alloc qm_status.txt | awk '{ print substr( $0, 16, length($0) ) }' | sort | uniq -c | sort -k1n | ||
+ | </ | ||
===== Using GDB ===== | ===== Using GDB ===== | ||