Setting up a gateway on OpenBSD

From 44Net Wiki
Revision as of 00:12, 8 January 2021 by Cross (talk | contribs)
Jump to navigation Jump to search

OpenBSD is a mature Unix-like operating system that focuses on security and correctness. It features a flexible, robust, performant TCP/IP stack and a highly configurable firewall. This page describes how to configure a computer running OpenBSD as an AMPRNet router to transfer traffic between AMPRNet subnets and the Internet.

OpenBSD natively supports IPENCAP (IP-IP) tunnels through gif(4) pseudo-devices. Each gif device is a virtual network interface, synthesized by the operating system, that implements a point-to-point tunnel. Unlike Linux, OpenBSD requires a separate gif interface for each tunnel. An essentially arbitrary number of such interfaces can be created and it scales to the number required to route all of AMPRNet.

One can manually configure gif tunnels and routes at the command line, or configure the system to establish tunnels and routes at boot time.

We will describe how to set things up by way of example. Assume a system configuration that looks substantially similar to the following:

  • A dedicated static IP address to use as an endpoint for AMPRNet traffic.
  • An ISP-provided router that is just a router; no NATing, no firewall.
  • An OpenBSD computer with three ethernet interfaces. For example, using a Ubiquiti EdgeRouter 3 Lite:
    1. cnmac0 is the external interface connected to the ISP's network (in this example, we use 23.30.150.141 routing to 23.30.150.142)
    2. cnmac1 connects to an internal network (its configuration is irrelevant)
    3. cnmac2 is the internal interface connected to the subnet (in this example, we use 44.44.107.1 routing for 44.44.107.0/24)

Let us start by configuring a single tunnel to the AMPRNet gateway at UCSD: 

ifconfig gif0 create
ifconfig gif0 tunnel 23.30.150.141 169.228.34.84
ifconfig gif0 inet 44.44.107.1 netmask 255.255.255.255
route add 44.0.0.1 -link -iface gif0 -llinfo


Once I had a tunnel up to UCSD, I found that I could ping my 44.44.107.1 machine from a host on my internal network, but not from arbitrary machines. This was interesting; it turns out that hosts on my internal network get NAT'ed to another IP address on the small subnet I got from Comcast (through another, completely separate router -- not comcast's router but another ERL). What was happening was that as I ping'ed 44.44.107.1 from e.g. my laptop, ICMP echo request packets got NAT'ed to this other address and routed over to amprgw.sysnet.ucsd.edu and tunneled back to the external interface of my AMPRNet gateway. The gateway accepted the encapsulated ICMP echo requests (I have a PF rule that explicitly allows ping) and forwarded them across the tunnel interface where they were unencapsulated; the IP stack saw that the result was addressed to an IP address on a local interface (i.e., they were for the router) and generated an ICMP echo response packet with a

  • source* address of 44.44.107.1 and a *destination* address of the

external address of my other router (that is, the address the ICMP echo request was NAT'ed to). This matched the network route for my local Comcats subnet and so my AMPRNet router realized it could pass the packet back to my other router directly. It did so and the other router happily took the packet, matched it back through the NAT back to the original requesting machine (my laptop) and forwarded it: hence, I got my ping responses back. But note that the response was not going through the tunnel back to UCSD: it was being routed directly through the external interface.

Now consider what happens when I tried to ping 44.44.107.1 from a different machine on some other network. The ICMP echo request packet gets routed through the UCSD gateway and tunneled back to my gateway as before, but since responses don't go through back through the tunnel, the response packet matches the default route of my gateway and get's forwarded to comcast's router. Comcast would look at it, see that 44.44.107.1 wasn't on one of it's known networks that it would route floor, and discard the response. Oops.

The solution was to set up a separate routing table in a different routing domain specifically for AMPRNet traffic, and tie the two together using firewall rules. In the AMPRNet routing table, I could set my default route to point to the UCSD gateway, so any traffic sent from one of my 44.44.107.0/24 addresses that doesn't match a route to a known tunnel gets forwarded through amprgw.sysnet.ucsd.edu. With that in place, I could ping my gateway from random machines. This must seem obvious to a lot of folks here, but it took me a little while to figure out what was going on. Things are working now, however.

So far I have encountered two other caveats: I decided to configure two tunnel interfaces statically at boot time: 'gif0' goes to the UCSD tunnel, and 'gif1' sets up a tunnel to N1URO for his 44.88 net. Under OpenBSD, I assumed that the natural way to do this would be to add /etc/hostname.gif0 and /etc/hostname.gif1 files and this does in fact create the tunnels at boot time. However, traffic going out from my gateway doesn't seem to get sent through the tunnels; I did not bother to track down exactly why, but I believe it has to do with some kind of implicit ordering dependency when initializing PF. When I set up the separate routing domain, it struck me that the language accepted by /etc/netstart in an /etc/hostname.if file was not sufficiently rich to set up tunnels in a routing domain, so I capitulated and just set up the static interfaces from /etc/rc.local; imperfect but it works.

The second caveat is that I seem to have tickled a kernel error trying to set up an alias of a second IP address on my 44.44.107.1 NIC; I get a kernel panic due to an assertion failure. It looks a bug to me, but I haven't had the bandwidth to track it down. In the meanwhile, simply don't add aliases to interfaces in non-default routing domains.

The biggest piece missing was a daemon to handle receiving 44net RIP packets and use that data to maintain tunnels and routes. I thought about porting one, but decided of write my own instead. It has been running for a few weeks now on my node and while it's still not quite "done" it seems to work well enough that I decided it was time to cast a somewhat a wider net and push it up to GitHub for comment from others.

A couple of quick notes on implementation:

  1. The program maintains a copy of the AMPRNet routing table in a modified PATRICIA trie (really a compressed radix tree). Routes are expired after receiving a RIP packet.
  2. A similar table of tunnels is maintained.
  3. Tunnel interfaces are reference counted and garbage collected. A bitmap indicating which tunnels are in use is maintained.
  4. The program is completely self-contained in the sense that I do not fork/exec external commands to e.g. configure tunnels or manipulate routes. That is all done via ioctls or writing messages to a routing socket.

There is more to do; I'm sure there are a few bugs. I'd also like to query the system state at startup to initialize the routing and tunnel tables. Exporting and/or parsing an encap file would be nice. Logging and error checking can, I'm sure, be improved.

It's about 1200 lines of non-comment code, compiles down to a 28K MIPS64 executable (stripped). The code is at https://github.com/dancrossnyc/44ripd

Retrieved from "https://wiki.ampr.org/w/index.php?title=Setting_up_a_gateway_on_OpenBSD&oldid=907"