The XMLRPC Module

Jan Janak

iptelorg GmbH

Table of Contents

1. Admin Guide
1. Parameters
1.1. route (string)
1.2. autoconversion (string)
1.3. escape_cr (integer)
1.4. double_lf_to_crlf (integer)
1.5. mode (integer)
1.6. url_skip (str)
1.7. url_match (str)
2. Functions
2.1. dispatch_rpc()
2.2. xmlrpc_reply(code, reason)

List of Tables

1. Data Type Conversion

List of Examples

1.
1.1. Set route parameter
1.2. Set the autoconversion parameter
1.3. Set the escape_cr parameter
1.4. Set the double_lf_to_crlf parameter
1.5. Set the mode parameter
1.6. Set url_skip parameter
1.7. Set url_match parameter
1.8. dispatch_rpc usage
1.9. xmlrpc_reply usage

1. Design Goals

  • Implemented as a module.

  • API independent of transport protocols.

  • Reuse transports available in SER.

  • The possibility to encrypt all communication.

  • The possibility to authenticate clients.

  • Easy integration with existing languages and implementations.

  • Easy and straightforward implementation of management functions in SER modules.

2. Overview of Operation

This module implements the XML-RPC transport and encoding interface for ser RPCs.

The XML-RPC protocol encodes the name of the method to be called along with its parameter in an XML document which is then conveyed using HTTP (Hyper Text Transfer Protocol) to the server. The server will extract the name of the function to be called along with its parameters from the XML document, execute the function, and encode any data returned by the function into another XML document which is then returned to the client in the body of a 200 OK reply to the HTTP request.

XML-RPC is similar to more popular SOAP (Simple Object Access Protocol), which is an XML-based messaging framework used in Web Services developed within the World Wide Web Consortium. Both protocols are using HTTP as the transport protocol for XML documents, but XML-RPC is much simpler and easier to implement than SOAP.

Here is an example of single XML-RPC function call to determine current time:

POST /RPC2 HTTP/1.0
User-Agent: Radio UserLand/7.1b7 (WinNT)
Host: time.xmlrpc.com
Content-Type: text/xml
Content-length: 131
	    
<?xml version="1.0"?>
<methodCall>
<methodName>currentTime.getCurrentTime</methodName>
<params>
</params>
</methodCall>

And the response returned by the server:

HTTP/1.1 200 OK
Connection: close
Content-Length: 183
Content-Type: text/xml
Date: Wed, 03 Oct 2001 15:53:38 GMT
Server: UserLand Frontier/7.0.1-WinNT

<?xml version="1.0"?>
<methodResponse>
<params>
<param>
<value><dateTime.iso8601>20011003T08:53:38</dateTime.iso8601>
</value>
</param>
</params>
</methodResponse>

XML-RPC specification spells HTTP as the official transport protocol for XML-RPC documents. SER does not directly support HTTP, it is a SIP server so SIP is the only protocol supported by SER. Because we would like to reuse all transport protocols available in SER, such as TCP and TLS, it would be natural to use modified version of XML-RPC which would run on top of SIP instead of HTTP. XML-RPC documents would be then encoded in the bodies of SIP requests and replies would be sent by the server in the bodies of SIP replies. This way we could reuse all transport protocols (including UDP) and message parsers available in SER.

Although this approach seems to be the logical choice, there is one big drawback. No existing XML-RPC implementations support SIP as the transport protocol, and there are many existing implementations available for vast majority of existing languages. See the XML-RPC implementation page for more details. Extending existing implementations with SIP support would not be easy.

Because extending available XML-RPC implementation would be too expensive, we could also do it the other way around, keep existing XML-RPC implementations and extend SER to support HTTP. Extending SER with HTTP support is easier than it might seem at a first glance, due to the similarity between SIP requests and HTTP requests.

SER already supports TCP, so existing HTTP implementations can send HTTP requests to it. HTTP requests are missing certain mandatory SIP header fields, such as Via, From, and CSeq. The contents of the header fields is mainly used for transaction matching. A SIP server could perform two basic operations when processing an HTTP request:

  • Terminate the request, execute the function and send a reply back.

  • Forward the request to another SIP server.

Nothing special is needed on the SIP server terminating the request, except that it has to know where it should send the reply. Parsing of HTTP header field bodies would fail because we do not have parsers for them in SER, but that does not matter anyway because all the information is encoded in the body of the request. HTTP requests contain no Via header fields. Via header fields are used by SIP implementations to determine the destination (IP, transport protocol, and port number) for replies. When processing HTTP requests the SIP server needs to create a fake Via header field based on the source IP address and port number of the TCP connection. The SIP server will use this information when sending a reply back.

Forwarding of HTTP requests by SIP proxies is a little bit more complicated and there are several limitations. First of all, we can only use stateless forwarding, no transactional forwarding, because HTTP requests do not contain all the header fields needed for transaction matching. Any attempt to call t_relay on an HTTP requests would fail. HTTP requests always use TCP and thus we could use stateless forwarding on the SIP server, provided that the request will be also forwarded over TCP. Stateless forwarding does not require the mandatory header fields (which are missing here) and it would work. In addition to that, the SIP server would also append fake Via header field to the request and change the contents of the Request-URI. The Request-URI of HTTP requests sent by XML-RPC implementations typically contain something like "/RPC2" and the first SIP server processing the request should rewrite the value with a valid SIP URI.

Figure RPC Example shows a scenario which involves two SIP servers, one performs HTTP request "normalization" and forwarding, and the other terminates the request, executes corresponding function, and generates a reply.

Example RPC Scenario

Step 1. An HTTP user agent sends an ordinary HTTP request to a SIP server. The user agent can either establish a connection directly to port 5060 or the SIP server can be configured to listen on port 80. The request contains standard HTTP headers and an XML-RPC document in the body:

POST / HTTP/1.0.
Host: localhost:5060
User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)
Content-Type: text/xml
Content-Length: 111

<?xml version='1.0'?>
<methodCall>
<methodName>usrloc.statistics</methodName>
<params>
</params>
</methodCall>

This particular request calls method "statistics" from from usrloc module of SER. The method has no parameters.

The outbound SIP server receives the HTTP request and performs a set of actions called "SIP-normalization". This includes creation of fake Via header field based on the source IP and port of the TCP connection, looking up of the target SIP server that should terminate and process the request, and rewriting of the Request-URI with the SIP URI of the target SIP server. Modified HTTP request will be then forwarded statelessly to the target SIP server.

POST sip:proxy01.sip-server.net HTTP/1.0
Via: SIP/2.0/TCP 127.0.0.1:3571
Host: localhost:5060
User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)
Content-Type: text/xml
Content-Length: 111

<?xml version='1.0'?>
<methodCall>
<methodName>usrloc.statistics</methodName>
<params>
</params>
</methodCall>

Step 2. "normalized" HTTP request is statelessly forwarded to the target SIP server over TCP.

Step 3. The target SIP server receives the HTTP request and executes function called dispatch_rpc from xmlrpc SER module. This function will parse the XML-RPC document in the body of the request and lookup the function to be called among all RPC functions exported by the SER core and modules. Function dispatch_rpc will be called from the configuration file just like any other function:

if (method == "POST" || method == "GET") {
    dispatch_rpc();
    break;
};

This particular configuration snippet executes the function whenever SER receives GET or POST requests. These two method names indicate HTTP.

Step 4. The function dispatch_rpc scans through the list of all exported RPC functions searching for the statistics function of the usrloc module. The SER RPC Module API describes in detail how modules export RPC functions.

Step 5. As the RPC function from usrloc module is running and gathering statistics, it calls functions of RPC interface to prepare the result for the caller.

Step 6. Once the RPC function finishes, xmlrpc module will build the XML-RPC document from the data received from usrloc module and generate a reply which will be sent to the caller.

Steps 7. and 8. HTTP reply is sent back to the caller and the remote procedure call finishes.

HTTP/1.0 200 OK
Via: SIP/2.0/TCP 127.0.0.1:3571
Server: Sip EXpress router (0.10.99-janakj_experimental (i386/linux))
Content-Length: 651
Warning: 392 127.0.0.1:5060 "Noisy feedback tells:  pid=9975 req_src_ip=127.0.0
1 req_src_port=3571 in_uri=/ out_uri=sip:proxy01.sip-server.net via_cnt==1"

<?xml version="1.0" encoding="UTF-8"?>
<methodResponse>
<params>
<param><value><array><data>
<value><struct>
<member><name>domain</name>
<value><string>aliases</string></value></member>
<member><name>users</name>
<value><i4>0</i4></value></member>
<member><name>expired</name>
<value><i4>0</i4></value></member>
</struct></value>
<value><struct>
<member><name>domain</name>
<value><string>location</string></value></member>
<member><name>users</name>
<value><i4>0</i4></value></member>
<member><name>expired</name>
<value><i4>0</i4></value></member>
</struct></value>
</data></array></value></param>
</params>
</methodResponse>

Note

The scenario described on Figure RPC Example involves two SIP servers. This is just to demonstrate that in setups containing more SIP servers it is possible to forward HTTP requests from one SIP server to another and use standard SIP routing mechanisms to decide which SIP server should process the request. There is no need to have multiple SIP servers in simple setups, because one SIP server can both add fake Via header field and process the RPC at the same time. Modified configuration file snipped could then look like this:

if (method == "POST" || method == "GET") {
    dispatch_rpc(); # Process the request
    break;
};

3. XML-RPC Implementation

The purpose of the functions of this module is to convert XML-RPC document carried in the body of HTTP requests into data returned by the RPC interface and back. The module also contains functions necessary to "normalize" HTTP requests. The module uses xmlrpc-c library to perform XML-RPC related functions.

The module always returns 200 OK HTTP reply, it will never return any other HTTP reply. Failures are expressed in XML-RPC documents in the body of the reply. There is basic method introspection support in the module. Currently the module can list all functions exported by the server and for each function it can return the documentation string describing the function.

3.1. Requests

Requests processed by the module are standard XML-RPC requests encoded in bodies of HTTP requests.

POST / HTTP/1.0
Host: localhost:5060
User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com)
Content-Type: text/xml
Content-Length: 112

<?xml version='1.0'?>
<methodCall>
<methodName>system.listMethods</methodName>
<params>
</params>
</methodCall>

The name of the method to be called in this example is "listMethods". This is one of the introspection methods. SER will call dispatch_rpc function of xmlrpc module to handle the request. The function will parse the XML-RPC document, lookup listMethods function in the list of all export RPC functions, prepare the context for the function and execute it.

3.2. Replies

The module will always generate 200 OK. Other response codes and classes are reserved for SER. The status code of the XML-RPC reply, response code, and additional data will be encoded in the body of the reply. Failure replies do not contain any data, just the response code and reason phrase:

HTTP/1.0 200 OK
Via: SIP/2.0/TCP 127.0.0.1:2464
Server: Sip EXpress router (0.10.99-janakj_experimental (i386/linux))
Content-Length: 301

<?xml version="1.0" encoding="UTF-8"?>
<methodResponse>

<fault>
<value><struct>
<member><name>faultCode</name>
<value><i4>501</i4></value></member>
<member><name>faultString</name>
<value><string>Method Not Implemented</string></value></member>
</struct></value>
</fault>

</methodResponse>

This particular reply indicates that there is no such RPC method available on the server.

Success replies always contain at least one return value. In our case the simplest success replies contain single boolean with value 1:

HTTP/1.0 200 OK
Via: SIP/2.0/TCP 127.0.0.1:4626
Server: Sip EXpress router (0.10.99-janakj_experimental (i386/linux))
Content-Length: 150

<?xml version="1.0" encoding="UTF-8"?>
<methodResponse>
<params>
<param><value><boolean>1</boolean></value></param>
</params>
</methodResponse>

This is exactly how the reply looks like when an RPC function does not add any data to the reply set.

If an RPC function adds just a single item (it calls add once with just one character in the formatting string) then the data will be converted to XML-RPC representation according to the rules described in SER RPC Type Conversion and the reply will contain just the single value:

HTTP/1.0 200 OK
Via: SIP/2.0/TCP 127.0.0.1:3793
Server: Sip EXpress router (0.10.99-janakj_experimental (i386/linux))
Content-Length: 216

<?xml version="1.0" encoding="UTF-8"?>
<methodResponse>
<params>
<param><value><string>Server: Sip EXpress router (0.10.99-janakj_experimental (i386/linux))</string></value></param>
</params>
</methodResponse>

If an RPC function adds more than one data items to the result set then the module will return an array containing all the data items:

HTTP/1.0 200 OK
Via: SIP/2.0/TCP 127.0.0.1:2932
Server: Sip EXpress router (0.10.99-janakj_experimental (i386/linux))
Content-Length: 276

<?xml version="1.0" encoding="UTF-8"?>
<methodResponse>
<params>
<param><value><array><data>
<value><string>./ser</string></value>
<value><string>-f</string></value>
<value><string>ser.cfg</string></value>
</data></array></value></param>
</params>
</methodResponse>

This is probably the most common scenario.

3.3. Type Conversion

The data types of the RPC API are converted to the data types of XML-RPC and vice versa. Table 1, “Data Type Conversion” shows for each RPC API data type corresponding XML-RPC data type.

Table 1. Data Type Conversion

RPC API XML-RPC RPC Example XML-RPC Example
Integer <i4></i4> rpc->add("d", 42) <i4>42</i4>
Float <double></double> rpc->add("f", -12.214) <double>-12.214</double>
String <string></string> rpc->add("s","Don't panic") <string>Don't panic</string>
Struct <struct></struct> rpc->struct_add(handle,"sd","param1",42,"param2",-12.214)
<struct>
  <member>
    <name>param1</name>
    <value>
      <i4>42</i4>
    </value>
  </member>
  <member>
    <name>param2</name>
    <value>
      <double>-12.214</i4>
    </value>
  </member>
</struct>


3.4. Limitations

SER xmlrpc modules does not implement all data types allowed in XML-RPC. As well it does not implement arrays and nested structures. This simplification is a feature, not bug. In our case the XML-RPC interface will be used mainly for management purposes and we do not need all the bells and whistles of XML-RPC. Parsing and interpreting nested structures is complex and we try to avoid it.

3.5. Interoperability Problems

Due to a bug in Python xmlrpclib there is an interoperability problem with basic clients using it: by default an xmlrpclib client expects the server to immediately close the connection after answering and if the server does not close the connections the xmlrpclib client will wait forever.

There are 2 ways to work around this problem: write a "fixed" Transport class and initialize xmlpclib using it (recommended) or make ser close the tcp connection after each request.

The examples/xmlrpc_test.py provides a very simple example of using xmlrpclib with a Transport class that works.

For the second solution (force closing tcp connections after answering) the XMLRPC route should have a set_reply_close() command before dispatch_rpc(). set_reply_no_connect() is also recommended (avoid trying to open tcp connection to xmlrpc clients that closed it). Alternatively ending the XMLRPC route with return -1, exit -1 or drop -1 can also be used, but note that this will not work for async rpcs (it will close the connection immeidately and not on the async response).

Example 1. 

route[XMLRPC]{
	# close connection only for xmlrpclib user agents
	if search("^User-Agent:.*xmlrpclib"))
		set_reply_close();
	set_reply_no_connect(); # optional
	dispatch_rpc();
}


Another common problem is CRLF handling. According to the xml spec CR ('\r') must be escaped (to &#xD;) or they will be "normalized" when parsing the xml document. However some xmlrpc clients do not follow this rule (e.g. clients based on the python or php xmlrpclib) and send CRLF unescaped. A possible workaround is to enable automatic LFLF to CRLF conversion (using the double_lf_to_crlf modules parameter) and replace CRLF with LFLF in the client queries.

4. Client Examples

  • examples/xmlrpc_test.pl (basic perl application that builds and sends an XMLRPC request from its commandline parameters).

  • examples/xmlrpc_test.py (basic python application that builds and sends an XMLRPC request from its commandline parameters).

  • ser_ctl (complex python application that uses the XML-RPC interface implemented by the xmlrpc module).

  • serweb (php application that can use the XML-RPC interface to call ser functions).

Chapter 1. Admin Guide

1. Parameters

1.1. route (string)

Name of the route called for XMLRPC messages.

This route will be called only for HTTP messages whose method is either GET or POST. The message visible inside the route will be a HTTP request converted to SIP (the uri will be fixed and a fake via will be added).

The route should perform additional security checks to ensure the client is authorized to execute management/RPC functions and then it should call the dispatch_rpc().

Default: the main route is used.

Example 1.1. Set route parameter

modparam("xmlrpc", "route", "route_for_xmlrpcs")

1.2. autoconversion (string)

Enable or disable automatic parameter type conversion globally, for all the methods parameters. If on, a type mismatch in a method parameter will not cause a fault if it is possible to automatically convert it to the type expected by the method.

Default: off.

It is recommended to leave this parameter to its default off value and fix instead the client application (which should use the proper types) or to modify the target rpc to accept any type (see the rpc scan '.' modifier).

Example 1.2. Set the autoconversion parameter

modparam("xmlrpc", "autoconversion", 1)

1.3. escape_cr (integer)

Enable CR ('\r') escaping in replies. If enabled each '\r' in the xmlrpc reply will be replaced with "&#xD;", according to the xml spec.

It should be turned off only if you suspect interoperability problems with older clients.

Default: on.

Example 1.3. Set the escape_cr parameter

modparam("xmlrpc", "escape_cr", 1)

1.4. double_lf_to_crlf (integer)

When enabled double LFs ('\n\n') in the input xmlrpc strings will be replaced with CR LF ('\r\n'). This makes LF LF behave like an escape character for CR LF and is needed for compatibility with kamailio tools and to work around buggy xmlrpc clients that don't escape the CR in CR LF ('\r' should be escaped to "&#xD;" otherwise according to the xml spec "\r\n" will be transformed to '\n'), but need to send CR LF in the strings (e.g. they use tm.t_uac_wait).

Note: when this option is turned on, there is no way to send a double LF ('\n\n'), it will always be transformed in CR LF ('\r\n').

Default: off.

Example 1.4. Set the double_lf_to_crlf parameter

modparam("xmlrpc", "double_lf_to_crlf", 1)

1.5. mode (integer)

When set to 1, xmlrpc module does not register to core the callback functions for non-SIP messages. Useful when other module register a callback for HTTP request, being the decision of admin when to call the XMLRPC route (or functions).

Default: 0.

Example 1.5. Set the mode parameter

modparam("xmlrpc", "mode", 1)

1.6. url_skip (str)

Regular expression to match the HTPP URL. If there is match, then xmlrpc route is not executed.

Default value is null (don't skip).

Example 1.6. Set url_skip parameter

...
modparam("xmlrpc", "url_skip", "^/sip")
...

1.7. url_match (str)

Regular expression to match the HTTP URL. If there is no match, then xmlrpc route is not executed. This check is done after url_skip, so if both url_skip and url_match would match then the xmlrpc route is not executed (url_skip has higher priority).

Default value is null (match everything).

Example 1.7. Set url_match parameter

...
modparam("xmlrpc", "url_match", "^/RPC2")
...

2. Functions

2.1.  dispatch_rpc()

This function processes an XMLRPC request, found in the body of the request.

It should be used only in a route specified using the "route" module parameter or if the request method is GET or POST (using it for other request methods will not have adverse side-effects, but it will probably not work).

dispatch_rpc() extracts the XML-RPC document from the body of the request to determine the name of the RPC method to be called and then it searches through the list of all the RPC functions to find a function with matching name. If such a function is found then dispatch_rpc() will pass control to the function to handle the request.

Example 1.8. dispatch_rpc usage

#...
modparam("xmlrpc", "route", "XMLRPC");
#...
route[XMLRPC]{
	if search("^User-Agent:.*xmlrpclib"))
		set_reply_close();
	set_reply_no_connect(); # optional
	dispatch_rpc();
}

2.2.  xmlrpc_reply(code, reason)

This function can be called from the config script to directly generate an XML-RPC reply.

Example 1.9. xmlrpc_reply usage

#...
modparam("xmlrpc", "route", "XMLRPC");
#...
route[XMLRPC]{
	# allow XMLRPC requests only on TLS and only if the client
	# certificate is valid
	if (proto!=TLS){
		xmlrpc_reply("400", "xmlrpc allowed only over TLS");
		return;
	}
	if (@tls.peer.verified!=""){
		xmlrpc_reply("400", "Unauthorized");
		return;
	}
	if search("^User-Agent:.*xmlrpclib"))
		set_reply_close();
	set_reply_no_connect(); # optional
	dispatch_rpc();
}