IPIP Mesh/MikroTik RouterOS 7 Container

Setting up a gateway in a ROS7 Mikrotik router running in a container on arm and arm64 models and x86-64 CHR

This is an experimental software build for the 'enthusiasts' out there.

Info

These are the steps for setting up a fully functional AMPR gateway on an arm/arm64 MikroTik router Tested and found working on CRS2116 and RB3011 for now.

MikroTik ARM64 devices:

Routers: CCR2004, CCR2116, CCR2216, RB5009
Switches: CRS520
Wireless & 5G: Netmetal ax, LHG-LTE6, ATL-LTE18
SOHO: hAP-ax2, cAP-ax, hAP-ax3, Chateau-ax
Others: AMPERE

MikroTik ARM32 devices:

Routers: L009, RB3011, RB4011, RB1100AHx4, 
Switches: CRS305, CRS309, CRS310, CRS317, CRS320, CRS326, CRS328
Wireless & 5G: SXTsq-5ac, NetBox-5ax, LHGXL-5ac
SOHO: hAP-ax lite, hap-ac2, cAP-ac, wAP-ac, cAPXL-ac, hAP-ac3, Chateau
Routerboard: L11UG, L23UGSR, RB450Gx4

MikroTik x86-64 devices:

Others: Cloud Hosted Router

Containers are not available on MIPSBE, MMIPS, SMIPS, TILE or PPC architectures.

General concept

Mikrotik routers running ROS 7 (7.15.3 being current at the time of writing) based on arm and arm64 processor, as well as CHR setups are able to run software containers (similar to docker). This opens the possibility to host a virtualized gateway in such a container, allowing a simple and efficient setup on modern systems.

The gateway will be hosted in a VRF on the router, providing gateway services using policy routing.

As a concept, the container has a single VETH interface which will decapsulate all incoming IPIP traffic from the tunnels, and encapsulate all outgoing traffic towards them. The container itself is isolated behind a bridge and offers some basic filtering function (e.g. restrict access from internet hosts). It will receive the RIPv2 broadcasts from the AMPR gateway and provide the obtained routes as RIP broadcasts to the router itself inside the mentioned VRF.

The container does not save anything to disk (which would be a bad idea on the router's flash memory), so the AMPR routes are lost on container or router restart, and you need to wait the now classical 5 minutes. But this should be no problem on a 24/7 on router.

Limitations

The router does not forward multicast frames at all, nor does it send out broadcasts. Incoming broadcasts are accepted and forwarded to the local VRF.

The container itself needs to sit behind a bridge due to a kernel bug in the version used by Mikrotik which sends out "Port unreachable" ICMP messages on incoming IPIP traffic if it is handled in user space (The same thing causing the need of a kernel filter in amprd. This is fixed in newer kernel releases but it will take a while for it to make its way into ROS). Bridge filtering is used to mask those messages.

New router set up

As a prerequisite, get your internet connection working based on the default mikrotik configuration. Basically set up your ISP uplink either via DHCP or by setting up a PPPoE or similar connection. Leave the firewall rule as they are.

First you need to enable container support according to the info provided by Mikrotik. In a console type in:

/system/device-mode/update container=yes

The device will ask you to reset it by hand (you can not do this remotely).

Next we need to install the container according to your hardware. Please chose the correct setup script variant:

ARM32 -  ampr_arm32.rsc
ARM64 -  ampr_arm64.rsc
CHRx86 - ampr_x86_64.rsc

Unfortunately, containers are not available on Mips, Tile or PowerPC devices.

The example assumes you use an arm32 device. Please use the proper one...

Open a route console window.

1. Check is the remote server is available:

[admin@MikroTik] > ping yo2loj.ro
 SEQ HOST                                     SIZE TTL TIME       STATUS                    
   0 89.33.44.100                               56  58 10ms574us 
   1 89.33.44.100                               56  58 9ms141us  
   2 89.33.44.100                               56  58 9ms5us    
   sent=3 received=3 packet-loss=0% min-rtt=9ms5us avg-rtt=9ms573us max-rtt=10ms574us

2. Download the configuration script

[admin@MikroTik] > /tool fetch url="http://yo2loj.ro/containers/ampr_arm32.rsc"
     status: finished
 downloaded: 5KiB
      total: 5KiB
   duration: 1s

3. Run the configuration script

[admin@MikroTik] > import ampr_arm32.rsc
AMPR: Creating bridge and VRF
AMPR: Setting up RIP
AMPR: Creating container envs
AMPR: Setting up firewall rules
AMPR: Creating container update script
AMPR: Creating routing rules
AMPR: Installing container
No container is installed
     status: finished
 downloaded: 366KiB
      total: 366KiB
   duration: 1s
AMPR: Script finished successful
AMPR: Now update your container envs and start the container

Your container is now installed. You need to configure its environment variables according to the description given below.

After configuration is complete, go to "containers" and star it up. It should show "running" and you should see it's messages in the log window.

After at most 5 minutes, you should get the tunnel routes in your vrf, and your gateway should be fully up and running.

If logging/debugging is not needed anymore, please disable it by clicking on the container and unchecking te logging box.

Container configuration parameters

You need to adapt the pre-existing container environment variables to your particular gateway before starting it again. The following ENV parameters are preset in Container-> Envs:

AMPR_SUBNETS - holds your local subnets as defined in the portal, as comma separated list of <SUBNET>/<MASK> tupples, e.g. "44.128.0.0/24,44.128.1.0/24"
ALL_VIA_AMPRGW - enables forwarding of all AMPR destinations via AMPRGW, values are "0" or "1"
FORWARD_INTERNET - enables forward of traffic from/to internet hosts, values are "0" or "1"
IGNORED_SUBNETS - allows you to ignore specific subnets provided by RIP, by <SUBNET>/<MASK> or gateway address e.g. "44.128.0.0/16"
CALL_HOME - the classic string, <CALLSIGN>@<LOCATOR> to show up on the map. You will get a yellow dot. e.g. "YO2LOJ@KN05OR". Leaving the field empty disables call home.

Please note that the provided default will allow you to play around, but will not provide a working set up.

Next, you need to set up a local AMPR LAN on your router router, or, if you have only a single IP address assigned, add it to one of your router's interfaces with a /32 netmask Anyway, you need to add a src-nat rule to the router's IP address to get your traffic flowing (let's assume its 44.128.0.1).

For a single address:

/ip address add address=44.128.0.1 interface=bridge

For a subnet:

/ip address add address=44.128.0.1/24 interface=<interface name>

And your src-nat NAT rule:

/ip firewall nat add action=src-nat chain=srcnat out-interface=bridge-ampr-gw to-addresses=44.128.0.1

Of course you need to set up firewall rules & stuff, but if you do not enable internet forward, you should be pretty safe.

Please note that for your firewall rules the incoming interface from the tunnels is "vrf_ampr" and the outgoing interface is "bridge-ampr-gw".

Additional optional configuration

You may notice that on an external traceroute your router's IP address will show up as 172.17.0.1. To fix this small glitch, you need to modify your existing "rip-ampr-in" RIP input filter rule to provide the correct preferred source address.

Modify the existing rule from

accept;

to set your router's local AMPR IP as its preferred source

set pref-src 44.128.0.1;
accept;

Configuration on an existing working router

Basically you need to do 6 steps by snooping around in the provided rsc files:

1 - Bridge, VETH and VRF setup: http://yo2loj.ro/containers/1_ampr_bridge_vrf.rsc
2 - RIP setup: http://yo2loj.ro/containers/2_rip.rsc
3 - Firewall rules, Filter, NAT and Mangle: http://yo2loj.ro/containers/3_firewall.rsc
4 - Container environment setup: http://yo2loj.ro/containers/4_container_env.rsc
5 - Container installation, architecture dependent. Files hold the download and update script:
 ARM32:  http://yo2loj.ro/containers/5_container_arm32.rsc
 ARM64:  http://yo2loj.ro/containers/5_container_arm64.rsc
 x86_64: http://yo2loj.ro/containers/5_container_x86_64.rsc
6 - final routing rules: http://yo2loj.ro/containers/6_rules.rsc

All available files are here: http://yo2loj.ro/containers/

(Details are coming...)

Rip Rip Hurray! de YO2LOJ