First public version of rspamd is available for beta-testing. Rspamd is fast, modular and lightweight spam filter. It is designed to
work with big ammount of mail and can be easily extended with own
filters written in lua or perl.
Rspamd can be found here: http://cebka.pp.ru/trac
Rspamd can be found here: http://cebka.pp.ru/trac
Here is mercurial repository of some of my software.
- http://cebka.pp.ru/hg/rmilter/ - mail filter for sendmail/postfix, that can check mail with spamassassin/clamav/dcc/spf, check limits and do greylisting (via memcached), and check regexps.
- http://cebka.pp.ru/hg/nginx-smtp-policy/ - async (libevent based) DNS resolver with patch to nginx (versions up to 0.6.11) mail module (originnally written by Max Dounin http://mdounin.ru/) that can do load balancing of incoming mail (weight based) check rbls and monitor state of mx's (with nginx-smtp-watchdog). Nginx-smtp-policy designed for huge mail traffic and can help to low load on backend mx's.
- http://cebka.pp.ru/hg/libevent/ - libevent patches for faster DNS search (hashed list instead of linear list) and for allowing TXT requests (for RBL checks).
- http://cebka.pp.ru/hg/rmilter/ - mail filter for sendmail/postfix, that can check mail with spamassassin/clamav/dcc/spf, check limits and do greylisting (via memcached), and check regexps.
- http://cebka.pp.ru/hg/nginx-smtp-policy/ - async (libevent based) DNS resolver with patch to nginx (versions up to 0.6.11) mail module (originnally written by Max Dounin http://mdounin.ru/) that can do load balancing of incoming mail (weight based) check rbls and monitor state of mx's (with nginx-smtp-watchdog). Nginx-smtp-policy designed for huge mail traffic and can help to low load on backend mx's.
- http://cebka.pp.ru/hg/libevent/ - libevent patches for faster DNS search (hashed list instead of linear list) and for allowing TXT requests (for RBL checks).
Here is script that allows to convert jabberd2 database items to xml format that can be used by openfire import/export plugin.
This patch fixes memcached support of udp protocol. Now UDP packets with wrong header can break memcached server down. Also I've made some fixes in cleaning UDP sessions after closing.
Results of testing (UDP vs TCP):
UDP:
Results of memcached stress test:
Total number of connections: 10000
Number of seconds for test: 2.23
Number of successfull connections: 10000
Connections per second: 4494.25
TCP:
Results of memcached stress test:
Total number of connections: 10000
Number of seconds for test: 4.38
Number of successfull connections: 10000
Connections per second: 2280.87
patch-memcached-udp
Results of testing (UDP vs TCP):
UDP:
Results of memcached stress test:
Total number of connections: 10000
Number of seconds for test: 2.23
Number of successfull connections: 10000
Connections per second: 4494.25
TCP:
Results of memcached stress test:
Total number of connections: 10000
Number of seconds for test: 4.38
Number of successfull connections: 10000
Connections per second: 2280.87
patch-memcached-udp
I wrote small patch for exim 4.68 that allows using of XCLIENT command in exim. The description of this command you can find here: http://www.postfix.org/XCLIENT_README.html
By default xclient is not allowed from any hosts, but you can specify hosts in option xclient_allow_hosts in configuration file:
xclient_allow_hosts = 127.0.0.1 : 192.168.1.1
Example of smtp dialog with XCLIENT:
Connected to localhost.
Escape character is '^]'.
220 dhcp-ng2 ESMTP Exim 4.68 Mon, 10 Dec 2007 19:26:44 +0300
XCLIENT NAME=spike.porcupine.org ADDR=168.100.189.2 HELO=blah
220 XCLIENT success
MAIL FROM:<wietse@porcupine.org>
250 OK
RCPT TO:<user@example.com>
550 relay not permitted
patch-exim-xclient
By default xclient is not allowed from any hosts, but you can specify hosts in option xclient_allow_hosts in configuration file:
xclient_allow_hosts = 127.0.0.1 : 192.168.1.1
Example of smtp dialog with XCLIENT:
Connected to localhost.
Escape character is '^]'.
220 dhcp-ng2 ESMTP Exim 4.68 Mon, 10 Dec 2007 19:26:44 +0300
XCLIENT NAME=spike.porcupine.org ADDR=168.100.189.2 HELO=blah
220 XCLIENT success
MAIL FROM:<wietse@porcupine.org>
250 OK
RCPT TO:<user@example.com>
550 relay not permitted
patch-exim-xclient
The order of loading kernel modules of packet filters affects the order of registering hooks in pfil interface.
So this code shows how pfil calls specified hook:
for (pfh = pfil_hook_get(dir, ph);
pfh != NULL;
pfh = TAILQ_NEXT(pfh, pfil_link)) {
if (pfh->pfil_func != NULL) {
rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir, inp);
if (rv != 0 || m == NULL)
break;
}
}
So we check all hooks begining from TAILQ_FIRST and pass packet to each hook as parameter.
If hook drops packet, we do not check other hooks.
This code register hooks in pfil interface:
/*
* insert the input list in reverse order of the output list
* so that the same path is followed in or out of the kernel.
*/
if (flags & PFIL_IN)
TAILQ_INSERT_HEAD(list, pfh1, pfil_link);
else
TAILQ_INSERT_TAIL(list, pfh1, pfil_link);
So for inbound traffic module that adds its hook last would process packets first, and for outbound traffic module that adds its hook first would process packets first.
Now let's determine load order of kernel modules.
Modules on load attach to subsystems, that are described in sys/kernel.h:
SI_SUB_PROTO_IF = 0x8400000, /* interfaces*/
SI_SUB_PROTO_DOMAIN = 0x8800000, /* domains (address families?)*/
SI_SUB_PROTO_IFATTACHDOMAIN = 0x8800001, /* domain dependent data init*/
Subsystems are loading in order of their number, first are loaded subsystems with lesser number.
Module on load can be attached to subsystem in different module too, this order is declared in macro DECLARE_MODULE:
enum sysinit_elem_order {
SI_ORDER_FIRST = 0x0000000, /* first*/
SI_ORDER_SECOND = 0x0000001, /* second*/
SI_ORDER_THIRD = 0x0000002, /* third*/
SI_ORDER_MIDDLE = 0x1000000, /* somewhere in the middle */
SI_ORDER_ANY = 0xfffffff /* last*/
};
So we should check subsystem of module and order of attachement to subsystem for each packet filter module:
pf:
DECLARE_MODULE(pf, pf_mod, SI_SUB_PROTO_IFATTACHDOMAIN, SI_ORDER_FIRST);
ipfw:
DECLARE_MODULE(ipfw, ipfwmod, SI_SUB_PROTO_IFATTACHDOMAIN, SI_ORDER_ANY);
ipfilter:
DECLARE_MODULE(ipfilter, ipfiltermod, SI_SUB_PROTO_DOMAIN, SI_ORDER_ANY);
Subsystems loads in this order: SI_SUB_PROTO_DOMAIN -> SI_SUB_PROTO_IFATTACHDOMAIN (ipfilter -> (ipfw, pf)), according to order of attachement of modules to subsystems the chain of loading must look like ipfilter -> pf -> ipfw.
Process order for inbound traffic:
ipfw -> pf -> ipfilter -> stack
and for outbound:
stack -> ipfilter -> pf -> ipfw
On the other hand ps is often loaded as kld, and is not statically compiled in the kernel, so if you can see pf.ko in loaded modules list (kldstat), pf is loaded after ipfw, not before. So traffic pass chains must look like this:
pf -> ipfw -> ipfilter -> stack - for inbound
stack -> ipfilter -> ipfw -> pf - for outbound
The order of passing trafic throught packet filters is very important in case of NAT, as next filter in chain can receive modified packet.
So this code shows how pfil calls specified hook:
for (pfh = pfil_hook_get(dir, ph);
pfh != NULL;
pfh = TAILQ_NEXT(pfh, pfil_link)) {
if (pfh->pfil_func != NULL) {
rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir, inp);
if (rv != 0 || m == NULL)
break;
}
}
So we check all hooks begining from TAILQ_FIRST and pass packet to each hook as parameter.
If hook drops packet, we do not check other hooks.
This code register hooks in pfil interface:
/*
* insert the input list in reverse order of the output list
* so that the same path is followed in or out of the kernel.
*/
if (flags & PFIL_IN)
TAILQ_INSERT_HEAD(list, pfh1, pfil_link);
else
TAILQ_INSERT_TAIL(list, pfh1, pfil_link);
So for inbound traffic module that adds its hook last would process packets first, and for outbound traffic module that adds its hook first would process packets first.
Now let's determine load order of kernel modules.
Modules on load attach to subsystems, that are described in sys/kernel.h:
SI_SUB_PROTO_IF = 0x8400000, /* interfaces*/
SI_SUB_PROTO_DOMAIN = 0x8800000, /* domains (address families?)*/
SI_SUB_PROTO_IFATTACHDOMAIN = 0x8800001, /* domain dependent data init*/
Subsystems are loading in order of their number, first are loaded subsystems with lesser number.
Module on load can be attached to subsystem in different module too, this order is declared in macro DECLARE_MODULE:
enum sysinit_elem_order {
SI_ORDER_FIRST = 0x0000000, /* first*/
SI_ORDER_SECOND = 0x0000001, /* second*/
SI_ORDER_THIRD = 0x0000002, /* third*/
SI_ORDER_MIDDLE = 0x1000000, /* somewhere in the middle */
SI_ORDER_ANY = 0xfffffff /* last*/
};
So we should check subsystem of module and order of attachement to subsystem for each packet filter module:
pf:
DECLARE_MODULE(pf, pf_mod, SI_SUB_PROTO_IFATTACHDOMAIN, SI_ORDER_FIRST);
ipfw:
DECLARE_MODULE(ipfw, ipfwmod, SI_SUB_PROTO_IFATTACHDOMAIN, SI_ORDER_ANY);
ipfilter:
DECLARE_MODULE(ipfilter, ipfiltermod, SI_SUB_PROTO_DOMAIN, SI_ORDER_ANY);
Subsystems loads in this order: SI_SUB_PROTO_DOMAIN -> SI_SUB_PROTO_IFATTACHDOMAIN (ipfilter -> (ipfw, pf)), according to order of attachement of modules to subsystems the chain of loading must look like ipfilter -> pf -> ipfw.
Process order for inbound traffic:
ipfw -> pf -> ipfilter -> stack
and for outbound:
stack -> ipfilter -> pf -> ipfw
On the other hand ps is often loaded as kld, and is not statically compiled in the kernel, so if you can see pf.ko in loaded modules list (kldstat), pf is loaded after ipfw, not before. So traffic pass chains must look like this:
pf -> ipfw -> ipfilter -> stack - for inbound
stack -> ipfilter -> ipfw -> pf - for outbound
The order of passing trafic throught packet filters is very important in case of NAT, as next filter in chain can receive modified packet.

