Table of Contents
List of Examples
xhttp_prom_buf_size parameterxhttp_prom_timeout parameterxhttp_prom_stats parameterxhttp_prom_beginning parameterprom_counter label exampleprom_counter parameterprom_gauge label exampleprom_gauge parameterprom_histogram label exampleprom_histogram parameterprom_counter_reset usageprom_gauge_reset usageprom_counter_inc usageprom_gauge_set usageprom_histogram_observe usageprom_dispatch usageprom_dispatch usage (more complete)prom_check_uri usagexhttp_prom.counter_reset usagexhttp_prom.counter_inc usagexhttp_prom.gauge_reset usagexhttp_prom.gauge_set usagexhttp_prom.histogram_observe usagexhttp_prom.metric_list_print usageTable of Contents
This module generates suitable metrics for a Prometheus monitoring platform.
It answers Prometheus pull requests (HTTP requests to /metrics URL).
The module generates metrics based on Kamailio statistics, and also the user can create his own metrics (currently counters, gauges and histograms).
The xHTTP_PROM module uses the xHTTP module to handle HTTP requests. Read the documentation of the xHTTP module for more details.
NOTE: This module is based on xHTTP_RPC one.
IMPORTANT: This module uses private memory to generate HTTP responses, and shared memory to store all the metrics. Remember to increase size of private and shared memory if you use a huge amount of metrics.
Prometheus URLs:
Specifies the maximum length of the buffer (in bytes) used to write the metric reply information in order to build the HTML response.
Default value is 0 (auto set to 1/3 of the size of the configured pkg mem).
Example 1.1. Set xhttp_prom_buf_size parameter
...
modparam("xhttp_prom", "xhttp_prom_buf_size", 1024)
...
Specifies a timeout in minutes. A metric not used during this timeout is automatically deleted. Listing metrics does not count as using them.
If set to 0 timeout is disabled. Negative values are not allowed.
Default value is 60 minutes.
Example 1.2. Set xhttp_prom_timeout parameter
...
# Set timeout to 10 hours
modparam("xhttp_prom", "xhttp_prom_timeout", 600)
...
Specifies which internal statistics from Kamailio to show. Possible values:
all - Show whole Kamailio statistics
group_name: - Show all statistics for a group
statistic_name - Show a specific statistic. It automatically finds the group.
Default value is "", meaning do not display any Kamailio statistics.
IMPORTANT: Kamailio internal statistics are parsed to convert - into _, so they accomplish with Prometheus guidelines for metric names. https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels User generated statistics and label names are not parsed.
Example 1.3. Set xhttp_prom_stats parameter
...
# show all kamailio statistics.
modparam("xhttp_prom", "xhttp_prom_stats", "all")
# show statistics for sl group.
modparam("xhttp_prom", "xhttp_prom_stats", "sl:")
# Show statistic for 200_replies in sl group.
modparam("xhttp_prom", "xhttp_prom_stats", "200_replies")
# Do not display internal Kamailio statistics. This is the default option.
modparam("xhttp_prom", "xhttp_prom_stats", "")
...
Specifies beginning of string the metrics are build with.
It defaults to "kamailio_", so if not specified every metric will start with "kamailio_".
Void string "" is also allowed, meaning no prefix string for every metric name.
Example 1.4. Set xhttp_prom_beginning parameter
...
# All metrics will start with "my_metric_".
modparam("xhttp_prom", "xhttp_prom_beginning", "my_metric_")
# No string at the beginning.
modparam("xhttp_prom", "xhttp_prom_beginning", "");
...
Create a counter metric.
This function declares a counter but the actual counter is only created when using it (by adding to or resetting it)
It takes a list of attribute=value separated by semicolon, the attributes can be name and label.
name - name of the counter. This attribute is mandatory. It is used to generate the metric name. Each name is unique, no metric shall repeat a name.
label - names of labels in the counter. Optional. Only one label parameter at most allowed in counters. Each label name is separated by : without spaces. At most only three label names allowed in each label parameter.
Example 1.5. prom_counter label example
# Create two labels called method and handler
label = method:handler
This would generate {method="whatever", handler="whatever2"} when building
the metric.
Example 1.6. Set prom_counter parameter
...
# Create cnt_first counter with no labels.
modparam("xhttp_prom", "prom_counter", "name=cnt_first;");
# Create cnt_second counter with no labels.
modparam("xhttp_prom", "prom_counter", "name=cnt_second;");
# Create cnt_third counter with label method
modparam("xhttp_prom", "prom_counter", "name=cnt_third; label=method");
These lines declare the counter but the actual metric will be created when
using it by prom_counter_inc or prom_counter_reset functions.
...
Create a gauge metric.
This function declares the gauge but the actual gauge is only created when using it (by setting or resetting it)
It takes a list of attribute=value separated by semicolon, the attributes can be name and value.
name - name of the gauge. This attribute is mandatory. It is used to generate the metric name. Each name is unique, no metric shall repeat a name.
label - names of labels in the gauge. Optional. Only one label parameter at most allowed in gauges. Each label name is separated by : without spaces. At most only three label names allowed inside each label parameter.
Example 1.7. prom_gauge label example
# Create two labels called method and handler
label = method:handler
This would generate {method="whatever", handler="whatever2"} when building
the metric.
Example 1.8. Set prom_gauge parameter
...
# Create gg_first gauge with no labels
modparam("xhttp_prom", "prom_gauge", "name=gg_first;");
# Create gg_second gauge with no labels
modparam("xhttp_prom", "prom_gauge", "name=gg_second;");
# Create gg_third gauge with two labels method and handler:
modparam("xhttp_prom", "prom_gauge", "name=gg_third; label=method:handler;");
...
Create a histogram metric.
This function declares a histogram but the actual histogram is only created when observing it.
It takes a list of attribute=value separated by semicolon, the attributes can be name, label and buckets.
name - name of the histogram. This attribute is mandatory. It is used to generate the metric name. Each name is unique, no metric shall repeat a name.
label - names of labels in the histogram. Optional. Only one label parameter at most allowed in histograms. Each label name is separated by : without spaces. At most only three label names allowed in each label parameter.
Example 1.9. prom_histogram label example
# Create two labels called method and handler
label = method:handler
This would generate {method="whatever", handler="whatever2"} when building
the metric.
buckets - specifies upper bounds for buckets in the histogram. This attribute is optional.
Bucket values are separated by ":". Each value has to be a number.
"+Inf" upper bucket is always automatically included.
At least one bucket value is needed (other than +Inf).
Every bucket value has to increase in the list.
If no buckets specified, default bucket list is set to these values:
[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
Example 1.10. Set prom_histogram parameter
...
# Create my_histo histogram with no labels and default buckets.
modparam("xhttp_prom", "prom_histogram", "name=my_histo;");
# Create second_histo histogram with one label and default buckets.
# modparam("xhttp_prom", "prom_histogram", "name=second_histo; label=my_lbl");
# Create a histogram with no labels and buckets 3.1, 5, 6.5
modparam("xhttp_prom", "prom_histogram", "name=histo_third; buckets=3.1:5:6.5");
# Create a histogram with a label and buckets 3.1, 5, 6.5
modparam("xhttp_prom", "prom_histogram", "name=histo_fourth; label=lbl; buckets=3.1:5:6.5");
These lines declare the histogram but the actual metric will be created when
using it by prom_histogram_observe function.
...
Get a counter based on its name and labels and reset its value to 0. Name parameter is mandatory. Values of labels are optional (from none up to three). Name in prom_counter_reset has to match same name in prom_counter parameter. Number of labels in prom_counter_reset has to match number of labels in prom_counter parameter. First time a counter is used with this reset function the counter is created if it does not exist already.
This function accepts pseudovariables on its parameters.
Available via KEMI framework as counter_reset_l0, counter_reset_l1, counter_reset_l2 and counter_reset_l3.
Example 1.11. prom_counter_reset usage
...
# Definition of counter with prom_counter with labels method and IP
modparam("xhttp_prom", "prom_counter", "name=cnt01; label=method:IP;");
...
# Reset cnt01 counter with two values "push" and "192.168.0.1" in labels to zero.
# First time we execute this function the counter will be created.
prom_counter_reset("cnt01", "push", "192.168.0.1");
...
# A metric like this will appear when listing this counter:
kamailio_cnt01 {method="push", IP="192.168.0.1"} 0 1234567890
...
Get a gauge based on its name and labels and reset its value to 0. Name parameter is mandatory. Values of labels are optional (from none up to three). Name in prom_gauge_reset has to match same name in prom_gauge parameter. Number of labels in prom_gauge_reset has to match number of labels in prom_gauge parameter. First time a gauge is used with this reset function the gauge is created if it does not exist already.
This function accepts pseudovariables on its parameters.
Available via KEMI framework as gauge_reset_l0, gauge_reset_l1, gauge_reset_l2 and gauge_reset_l3.
Example 1.12. prom_gauge_reset usage
...
# Definition of gauge with prom_gauge with labels method and IP
modparam("xhttp_prom", "prom_gauge", "name=cnt01; label=method:IP;");
...
# Reset cnt01 gauge with two values "push" and "192.168.0.1" in labels to zero.
# First time we execute this function the gauge will be created.
prom_gauge_reset("cnt01", "push", "192.168.0.1");
...
# A metric like this will appear when listing this gauge:
kamailio_cnt01 {method="push", IP="192.168.0.1"} 0 1234567890
...
Get a counter identified by its name and labels and increase its value by a number. If counter does not exist it creates the counter, initializes it to zero and adds the number.
Name is mandatory, number is mandatory. Number has to be positive or zero (integer). l0, l1, l2 are values of labels and are optional.
name value and number of labels have to match a previous counter definition with prom_counter.
This function accepts pseudovariables on its parameters.
Available via KEMI framework as counter_inc_l0, counter_inc_l1, counter_inc_l2 and counter_inc_l3.
Example 1.13. prom_counter_inc usage
...
# Definition of cnt01 counter with no labels.
modparam("xhttp_prom", "prom_counter", "name=cnt01;");
...
# Add 10 to value of cnt01 counter (with no labels) If counter does not exist it gets created.
prom_counter_inc("cnt01", "10");
...
# Definition of cnt02 counter with two labels method and IP
modparam("xhttp_prom", "prom_counter", "name=cnt02; label=method:IP;");
...
# Add 15 to value of cnt02 counter with labels method and IP. It creates the counter if it does not exist.
prom_counter_inc("cnt02", "15", "push", "192.168.0.1");
# When listed the metric it will show a line like this:
kamailio_cnt02 {method="push", IP="192.168.0.1"} 15 1234567890
...
Get a gauge identified by its name and labels and set its value to a number. If gauge does not exist it creates the gauge and assigns the value to it.
Name is mandatory, number is mandatory. Number is a string that will be parsed as a float. l0, l1, l2 are values of labels and are optional.
name value and number of labels have to match a previous gauge definition with prom_gauge.
This function accepts pseudovariables on its parameters.
Available via KEMI framework as gauge_set_l0, gauge_set_l1, gauge_set_l2 and gauge_set_l3.
Example 1.14. prom_gauge_set usage
...
# Definition of gg01 gauge with no labels.
modparam("xhttp_prom", "prom_gauge", "name=gg01;");
...
# Assign -12.5 to value of gg01 gauge (with no labels) If gauge does not exist it gets created
prom_gauge_set("gg01", "-12.5");
...
# Definition of gg02 gauge with two labels method and IP
modparam("xhttp_prom", "prom_gauge", "name=gg02; label=method:IP;");
...
# Assign 2.8 to value of gg02 gauge with labels method and IP. It creates the gauge if it does not exist.
prom_gauge_set("gg02", "2.8", "push", "192.168.0.1");
# When listed the metric it will show a line like this:
kamailio_gg02 {method="push", IP="192.168.0.1"} 2.8 1234567890
...
Get a histogram identified by its name and labels and observe a value in it. If histogram does not exist it creates the histogram and accumulate the value in its buckets, counter and sum.
Name is mandatory, number is mandatory. Number is a string that will be parsed as a float. l0, l1, l2 are values of labels and are optional.
name value and number of labels have to match a previous histogram definition with prom_histogram.
This function accepts pseudovariables on its parameters.
Available via KEMI framework as histogram_observe_l0, histogram_observe_l1, histogram_observe_l2 and histogram_observe_l3.
Example 1.15. prom_histogram_observe usage
...
# Definition of hist01 histogram with no labels and default buckets.
modparam("xhttp_prom", "prom_histogram", "name=hist01;");
...
# Observe -12.5 value in hist01 histogram (with no labels). If histogram does not exist it gets created:
prom_histogram_observe("hist01", "-12.5");
...
# Definition of hist02 histogram with two labels method and IP and buckets [2.3, 5.8, +Inf]:
modparam("xhttp_prom", "prom_histogram", "name=hist02; label=method:IP; buckets=2.3:5.8");
...
# Observe 2.8 value in hist02 histogram with labels method and IP.
# It creates the histogram if it does not exist.
prom_histogram_observe("hist02", "2.8", "push", "192.168.0.1");
# When listed the metric it will show lines like this:
hist02_bucket{method="push", IP="192.168.0.1", le="2.300000"} 0 1592574659768
hist02_bucket{method="push", IP="192.168.0.1", le="5.800000"} 1 1592574659768
hist02_bucket{method="push", IP="192.168.0.1", le="+Inf"} 1 1592574659768
hist02_sum{method="push", IP="192.168.0.1"} 2.800000 1592574659768
hist02_count{method="push", IP="192.168.0.1"} 1 1592574659768
...
Handle the HTTP request and generate a response.
Available via KEMI framework as xhttp_prom.dispatch
Example 1.16. prom_dispatch usage
...
# Needed to use SIP frames as HTTP ones.
tcp_accept_no_cl=yes
...
# xhttp module depends on sl one.
loadmodule "sl.so"
loadmodule "xhttp.so"
loadmodule "xhttp_prom.so"
...
# show all kamailio statistics.
modparam("xhttp_prom", "xhttp_prom_stats", "all")
...
event_route[xhttp:request] {
$var(xhttp_prom_root) = $(hu{s.substr,0,8});
if ($var(xhttp_prom_root) == "/metrics")
prom_dispatch();
else
xhttp_reply("200", "OK", "text/html",
"<html><body>Wrong URL $hu</body></html>");
}
...
Example 1.17. prom_dispatch usage (more complete)
Another example with counters and gauge:
...
# Needed to use SIP frames as HTTP ones.
tcp_accept_no_cl=yes
# xhttp module depends on sl one.
loadmodule "sl.so"
loadmodule "xhttp.so"
loadmodule "xhttp_prom.so"
# show Kamailio statistics for sl group
modparam("xhttp_prom", "xhttp_prom_stats", "sl:")
# Define two counters and a gauge
modparam("xhttp_prom", "prom_counter", "name=cnt_first;");
modparam("xhttp_prom", "prom_counter", "name=cnt_second; label=method:IP");
modparam("xhttp_prom", "prom_gauge", "name=gg_first; label=handler");
event_route[xhttp:request] {
$var(xhttp_prom_root) = $(hu{s.substr,0,8});
if ($var(xhttp_prom_root) == "/metrics") {
prom_counter_reset("cnt_first");
prom_counter_inc("cnt_second", "10", "push", "192.168.0.1");
prom_gauge_set("gg_first", "5.2", "my_handler");
prom_dispatch();
} else
xhttp_reply("200", "OK", "text/html",
"<html><body>Wrong URL $hu</body></html>");
}
...
We can manually check the result with a web browser:
We assume Kamailio runs in localhost and port is set to default (same as SIP: 5060)
http://localhost:5060
...
# User defined metrics
kamailio_cnt_first 0 1554839325427
kamailio_cnt_second {method="push", IP="192.168.0.1"} 10 1554839325427
kamailio_gg_first{handler="my_handler"} 5.2 1554839325427
# Kamailio internal statistics
kamailio_sl_1xx_replies 0 1554839325427
kamailio_sl_200_replies 15 1554839325427
kamailio_sl_202_replies 0 1554839325427
kamailio_sl_2xx_replies 0 1554839325427
kamailio_sl_300_replies 0 1554839325427
kamailio_sl_301_replies 0 1554839325427
kamailio_sl_302_replies 0 1554839325427
kamailio_sl_3xx_replies 0 1554839325427
kamailio_sl_400_replies 0 1554839325427
kamailio_sl_401_replies 0 1554839325427
kamailio_sl_403_replies 0 1554839325427
kamailio_sl_404_replies 0 1554839325427
kamailio_sl_407_replies 0 1554839325427
kamailio_sl_408_replies 0 1554839325427
kamailio_sl_483_replies 0 1554839325427
kamailio_sl_4xx_replies 0 1554839325427
kamailio_sl_500_replies 0 1554839325427
kamailio_sl_5xx_replies 0 1554839325427
kamailio_sl_6xx_replies 0 1554839325427
kamailio_sl_failures 0 1554839325427
kamailio_sl_received_ACKs 0 1554839325427
kamailio_sl_sent_err_replies 0 1554839325427
kamailio_sl_sent_replies 15 1554839325427
kamailio_sl_xxx_replies 0 1554839325461
...
Check if path of HTTP URL equals /metrics. This avoids us to check hu variable from xHTTP module.
NOTE: Remember not to block /metrics URL in xHTTP module
Available via KEMI framework as xhttp_prom.check_uri
Example 1.18. prom_check_uri usage
...
# Needed to use SIP frames as HTTP ones.
tcp_accept_no_cl=yes
...
# xhttp module depends on sl one.
loadmodule "sl.so"
loadmodule "xhttp.so"
loadmodule "xhttp_prom.so"
...
# show all kamailio statistics.
modparam("xhttp_prom", "xhttp_prom_stats", "all")
...
event_route[xhttp:request] {
if (prom_check_uri())
prom_dispatch();
else
xhttp_reply("200", "OK", "text/html",
"<html><body>Wrong URL $hu</body></html>");
}
...
Set a counter to zero.
Name: xhttp_prom.counter_reset
Parameters:
name: name of the counter (mandatory)
l0: value of the first label (optional)
l1: value of second label (optional)
l2: value of the third label (optional)
Example 1.19. xhttp_prom.counter_reset usage
... kamcmd xhttp_prom.counter_reset "cnt01" "push" "192.168.0.1" ...
Add a number to a counter based on its name and labels.
Name: xhttp_prom.counter_inc
Parameters:
name: name of the counter (mandatory)
number: integer to add to counter value. Negative values not allowed.
l0: value of the first label (optional)
l1: value of second label (optional)
l2: value of the third label (optional)
Example 1.20. xhttp_prom.counter_inc usage
... kamcmd xhttp_prom.counter_inc "cnt01" 15 "push" "192.168.0.1" ...
Set gauge value to zero. Select gauge based on its name and labels.
Name: xhttp_prom.gauge_reset
Parameters:
name: name of the gauge (mandatory)
l0: value of the first label (optional)
l1: value of second label (optional)
l2: value of the third label (optional)
Example 1.21. xhttp_prom.gauge_reset usage
... kamcmd xhttp_prom.gauge_reset "gg01" "push" "192.168.0.1" ...
Set a gauge to a number. Select the gauge by its name and labels.
Name: xhttp_prom.gauge_set
Parameters:
name: name of the gauge (mandatory)
number: float value to set the gauge to (mandatory)
l0: value of the first label (optional)
l1: value of second label (optional)
l2: value of the third label (optional)
Observe a number in a histogram. Select the histogram by its name and labels.
Name: xhttp_prom.histogram_observe
Parameters:
name: name of the histogram (mandatory)
number: float value to observe in the histogram (mandatory)
l0: value of the first label (optional)
l1: value of second label (optional)
l2: value of the third label (optional)
Example 1.23. xhttp_prom.histogram_observe usage
... kamcmd xhttp_prom.histogram_observe "hist01" -- -5.2 ...
List of all user defined metrics.
Name: xhttp_prom.metric_list_print
Parameters:none
NOTE:: If you list a lot of metrics you may need to increase buffer size of your RPC transport layer.