LL      IIIII NN   NN KK  KK EEEEEEE RRRRRR  RRRRRR   OOOOO  RRRRRR 
LL       III  NNN  NN KK KK  EE      RR   RR RR   RR OO   OO RR   RR
LL       III  NN N NN KKKK   EEEEE   RRRRRR  RRRRRR  OO   OO RRRRRR 
LL       III  NN  NNN KK KK  EE      RR  RR  RR  RR  OO   OO RR  RR 
LLLLLLL IIIII NN   NN KK  KK EEEEEEE RR   RR RR   RR  OOOOO  RR   RR
                                                           ramblings
____________________________________________________________________
Posted on: Tuesday, April 28th, 2009 at 20:37.
Filed under: All, ServerAdmin.
RSS 2.0 feed for comments.
You can leave a response, or trackback from your own site.

I had mentioned before that I was experiencing some problems when using natd with ipfw; more specifically traffic slowing down gradually until reaching a standstill.

I had always suspected that this is due to some recursive loop in the firewall, or natd diverting more than it should…

I finally solved the issue by making the ipfw divert rule more strict about what traffic to divert to natd.

I also added some rules that detect diverted traffic, and skip the normal allow rules, to prevent further mixups.

I am still using ipfw’s fwd feature to do the actual port forwarding, since it is always going to be faster than the natd daemon, which isn’t running inside the kernel space. (Note that there is support for in-kernel nat in FreeBSD, but I need further testing to set that up, since the last time I tried it, it caused a kernel panic, and having no kvm access on that machine makes these kinds of experiments undesirable.)

So, this is what the firewall rules ended up looking like:

# Initialize script -----------------------------------------------------------

# ip address(es) exposed to internet

inet="xxx.xxx.xxx.xxx/xx"

# jails

jail1="xxx.xxx.xxx.xxx"
# ... add more jail ip's here

any_jail="xxx.xxx.xxx.xxx/xxx"

# define how we call ipfw

IPF="ipfw -q add"

# Flush the firewall rules. We want a clean slate.

ipfw -q -f flush


# Port forwarding from internet to jails. --------------------------------------

$IPF 2 fwd $jail1,80 tcp from any to $inet 80
$IPF 3 fwd $jail1,443 tcp from any to $inet 443

# Allow local to local --------------------------------------------------------

$IPF 8 allow ip from any to any via lo0

# NATD out divert. This allows internet access from within jails. -------------

$IPF 10 divert natd ip from $any_jail to not me out via msk1
$IPF 11 skipto 10000 ip from any to any diverted

# Allow out traffic.

$IPF 12 allow ip from $inet to any out

# Services. -------------------------------------------------------------------

# DNS

$IPF 100 allow ip from any to $inet 53 in via msk1

# Apache

$IPF 101 allow tcp from any to $inet 80 in via msk1
$IPF 101 allow tcp from any to $inet 443 in via msk1

# Mail (pop3,pop3s,imap,imaps,smtp,smtps)

$IPF 102 allow tcp from any to $inet 25 in via msk1
$IPF 102 allow tcp from any to $inet 110 in via msk1
$IPF 102 allow tcp from any to $inet 143  in via msk1
$IPF 102 allow tcp from any to $inet 456 in via msk1
$IPF 102 allow tcp from any to $inet 993 in via msk1
$IPF 102 allow tcp from any to $inet 995 in via msk1

# SSH

$IPF 103 allow ip from any to $inet 22 in via msk1

# FTP

$IPF 104 allow tcp from any to $inet 21 in via msk1
$IPF 104 allow tcp from any to $inet 20 in via msk1
$IPF 104 allow tcp from any to $inet dst-port 9000-9040 in via msk1

# etc... add more services as needed

# Natd in divert. this allows internet access from within jails. --------------

$IPF 8000 divert natd ip from not me to any in via msk1
$IPF 8001 skipto 10000 ip from any to any diverted
 
# Default deny ----------------------------------------------------------------

$IPF 9000 deny log logamount 10000 ip from any to any

# Anything after 10000 is traffic re-inserted from natd. ----------------------

$IPF 10000 allow ip from any to any

If you look up almost any natd example, a divert from all to all via $iface is depicted.

In the end, for some reason when you’re diverting from a local interface to aliases on another local interface (as is typically the case with jails), in both directions, diverting from any to any is way too generic, and will cause trouble.

Try to define the divert rule as specific as possible, and keep in mind that you can match any diverted traffic with the diverted keyword.

Some debugging tips:

Install cmdwatch from ports, and run:

cmdwatch -n1 'ipfw -a list'

This allows you to view the number of packets matched by each firewall rule in real time.
You could run this in a screen session, with a split screen setup, while in the other screen running atail -f /var/log/ipfw.log  and perhaps a tcpdump session.

Also, when working remotely it’s probably a good idea to add something to your crontab that shuts down ipfw every 10 minutes or so, just in case you lock yourself out ( which is something very common while debugging firewalls remotely, no matter who you are ;) )

Example temporary failsafe crontab entry for a debug session:

*/10 * * * * /etc/rc.d/ipfw stop

However, it’s also frustrating when you’re thinking your nat is broken, when it’s really your crontab that just disabled your firewall. Therefore it’s a good idea to keep an eye on the firewall status during debugging, and run something like this in one of your screens :


cmdwatch -n1 'sysctl net.inet.ip.fw.enable'

Or, you can combine the above cmdwatch lines with:


cmdwatch -n1 'sysctl net.inet.ip.fw.enable ; ipfw -a list'

(If you’re a GNU/Linux user, the cmdwatch utility on BSD is the same as the watch command on GNU/Linux. The only difference, besides the name, is that the GNU/Linux version allows for refresh intervals below a second. The watch command on FreeBSD is actually an utility to snoop on tty sessions.)

pixelstats trackingpixel
____________________________________________________________________

One Response to “natd with ipfw and jails”

  1. mouser says:

    i don’t understand a word of it but it’s great you figured out the problem.

____________________________________________________________________

Leave a Reply