Setting up a gateway in a ROS7 Mikrotik router container on arm32 arm64 and x86-64

From 44Net Wiki
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Setting up a gateway in a ROS7 Mikrotik router running in a container on arm and arm64 models and x86-64 CHR

This is an experimental software build for the 'enthusiasts' out there.

Source code is available in git.ampr.org: https://git.ampr.org/yo2loj/ampr-ros7-container/-/tree/main


Info

These are the steps for setting up a fully functional AMPR gateway on an arm/arm64 MikroTik router Tested and found working on CRS2116 and RB3011 for now.

MikroTik ARM64 devices: 

Routers: CCR2004, CCR2116, CCR2216, RB5009
Switches: CRS520
Wireless & 5G: Netmetal ax, LHG-LTE6, ATL-LTE18
SOHO: hAP-ax2, cAP-ax, hAP-ax3, Chateau-ax
Others: AMPERE

MikroTik ARM32 devices: 

Routers: L009, RB3011, RB4011, RB1100AHx4, 
Switches: CRS305, CRS309, CRS310, CRS317, CRS320, CRS326, CRS328
Wireless & 5G: SXTsq-5ac, NetBox-5ax, LHGXL-5ac
SOHO: hAP-ax lite, hap-ac2, cAP-ac, wAP-ac, cAPXL-ac, hAP-ac3, Chateau
Routerboard: L11UG, L23UGSR, RB450Gx4

MikroTik x86-64 devices: 

Others: Cloud Hosted Router

Containers are not available on MIPSBE, MMIPS, SMIPS, TILE or PPC architectures.

General concept

Mikrotik routers running ROS 7 (7.15.3 being current at the time of writing) based on arm and arm64 processor, as well as CHR setups are able to run software containers (similar to docker). This opens the possibility to host a virtualized gateway in such a container, allowing a simple and efficient setup on modern systems.

The gateway will be hosted in a VRF on the router, providing gateway services using policy routing.

As a concept, the container has a single VETH interface which will decapsulate all incoming IPIP traffic from the tunnels, and encapsulate all outgoing traffic towards them. The container itself is isolated behind a bridge and offers some basic filtering function (e.g. restrict access from internet hosts). It will receive the RIPv2 broadcasts from the AMPR gateway and provide the obtained routes as RIP broadcasts to the router itself inside the mentioned VRF.

The container does not save anything to disk (which would be a bad idea on the router's flash memory), so the AMPR routes are lost on container or router restart, and you need to wait the now classical 5 minutes. But this should be no problem on a 24/7 on router.

Limitations

The router does not forward multicast frames at all, nor does it send out broadcasts. Incoming broadcasts are accepted and forwarded to the local VRF.

The container itself needs to sit behind a bridge due to a kernel bug in the version used by Mikrotik which sends out "Port unreachable" ICMP messages on incoming IPIP traffic if it is handled in user space (The same thing causing the need of a kernel filter in amprd. This is fixed in newer kernel releases but it will take a while for it to make its way into ROS). Bridge filtering is used to mask those messages.

New router 5 minutes set up

As a prerequisite, get your internet connection working based on the default mikrotik configuration. Basically set up your ISP uplink either via DHCP or by setting up a PPPoE or similar connection. Leave the default firewall rule as they are. Alternatively, you can start with a completely empty router, with only a active internet connection.

First you need to enable container support according to the info provided by Mikrotik. In a console type in: 

/system/device-mode/update container=yes

The system will want you to do a hard reset at this point to confirm the request. This means you need physical access to the device.

Next, you need to install the container package for your firmware version. Download the "extra" firmware package from MikroTik for your FW version and extract the "container-7.x.y-<arch>.npk" file. Upload it to your router and restart. This will install the package onto the router. After restart, you will have a new option available: /containers

Installation script

Next we need to install the container according to your hardware. Please chose the correct setup script variant: 

ARM32 -  ampr_arm32.rsc
ARM64 -  ampr_arm64.rsc
CHRx86 - ampr_x86_64.rsc

Unfortunately, containers are not available on Mips, Tile or PowerPC devices.

The example assumes you use an arm32 device. Please use the proper one...

Open a route console window.

1. Check is the remote server is available: 

[admin@MikroTik] > ping yo2loj.ro
 SEQ HOST                                     SIZE TTL TIME       STATUS                    
   0 89.33.44.100                               56  58 10ms574us 
   1 89.33.44.100                               56  58 9ms141us  
   2 89.33.44.100                               56  58 9ms5us    
   sent=3 received=3 packet-loss=0% min-rtt=9ms5us avg-rtt=9ms573us max-rtt=10ms574us

2. Download the configuration script 

[admin@MikroTik] > /tool fetch url="http://yo2loj.ro/containers/ampr_arm32.rsc"
     status: finished
 downloaded: 5KiB
      total: 5KiB
   duration: 1s

3. Run the configuration script 

[admin@MikroTik] > import ampr_arm32.rsc
AMPR: Creating bridge and VRF
AMPR: Setting up RIP
AMPR: Creating container envs
AMPR: Setting up firewall rules
AMPR: Creating container update script
AMPR: Creating routing rules
AMPR: Installing container
No container is installed
     status: finished
 downloaded: 366KiB
      total: 366KiB
   duration: 1s
AMPR: Script finished successful
AMPR: Now update your container envs and start the container

Your container is now installed. You need to configure its environment variables according to the description given below.

After configuration is complete, go to "containers" and star it up. It should show "running" and you should see it's messages in the log window.

After at most 5 minutes, you should get the tunnel routes in your vrf, and your gateway should be fully up and running.

If logging/debugging is not needed anymore, please disable it by clicking on the container and unchecking te logging box.

Container configuration parameters

You need to adapt the pre-existing container environment variables to your particular gateway before starting it again. The following ENV parameters are preset in Container-> Envs: 

AMPR_SUBNETS - holds your local subnets as defined in the portal, as comma separated list of <SUBNET>/<MASK> tupples, e.g. "44.128.0.0/24,44.128.1.0/24"
ALL_VIA_AMPRGW - enables forwarding of all AMPR destinations via AMPRGW, values are "0" or "1"
FORWARD_INTERNET - enables forward of traffic from/to internet hosts, values are "0" or "1"
IGNORED_SUBNETS - allows you to ignore specific subnets provided by RIP, by <SUBNET>/<MASK> or gateway address e.g. "44.128.0.0/16"
CALL_HOME - the classic string, <CALLSIGN>@<LOCATOR> to show up on the map. You will get a yellow dot. e.g. "YO2LOJ@KN05OR". Leaving the field empty disables call home.

Please note that the provided default will allow you to play around, but will not provide a working set up.

Next, you need to set up a local AMPR LAN on your router router, or, if you have only a single IP address assigned, add it to one of your router's interfaces with a /32 netmask Anyway, you need to add a src-nat rule to the router's IP address to get your traffic flowing (let's assume its 44.128.0.1). The address shall be set on an interface OUTSIDE OF THE VRF.

For a single address: 

/ip address add address=44.128.0.1 interface=bridge

or even on the loopback interface: For a single address: 

/ip address add address=44.128.0.1 interface=lo

For a subnet: 

/ip address add address=44.128.0.1/24 interface=<interface name>

And your src-nat NAT rules: 

/ip firewall nat add action=src-nat chain=srcnat out-interface=bridge-ampr-gw to-addresses=44.128.0.1
/ip firewall nat add action=src-nat chain=srcnat out-interface=vrf-ampr to-addresses=44.128.0.1

Of course you need to set up additional firewall rules & stuff, but if you do not enable internet forwarding, you should be pretty safe being exposed only to AMPR partners.

Please note that for your firewall rules the incoming interface from the tunnels is "vrf_ampr" and the outgoing interface is "bridge-ampr-gw" for forwarded and "vrf-ampr" for local outgoing data.

Additional optional configuration

You may notice that on an external traceroute your router's IP address will show up as 172.17.0.1. To fix this small glitch, you need to modify your existing "rip-ampr-in" RIP input filter rule to provide the correct preferred source address.

Modify the existing rule from 

accept;

to set your router's local AMPR IP as its preferred source 

set pref-src 44.128.0.1;
accept;

Also, if you want to access the AMPR network from a LAN not using AMPR addresses, you need to set up a forwarding rule and a SRC-NAT one: 

/ip firewall filter
add action=accept chain=forward comment="from LAN" in-interface=<YourLANInterface> dst-address-list=ampr_addr

and 

/ip firewall nat
add action=src-nat chain=srcnat comment="NAT to AMPR" dst-address-list=Ampr out-interface=bridge-ampr-gw \
   src-address-list=!ampr_addr to-addresses=44.128.0.1

Configuration on an existing working router - 6 steps

1 - Bridge, VETH, VRF and interface setup
2 - RIP setup
3 - Firewall rules, Filter, NAT and Mangle
4 - Container environment setup
5 - Container installation (architecture dependent)
6 - Container configuration and final touches

Preliminary: prepare the router to accept containers

First, you need to install container support on your router. In a console issue: 

/system/device-mode/update container=yes

The system will want you to do a hard reset at this point to confirm the request. This means you need physical access to the device.

Next, you need to install the container package for your firmware version. Download the "extra" firmware package from MikroTik for your FW version and extract the "container-7.x.y-<arch>.npk" file. Upload it to your router and restart. This will install the package onto the router. After restart, you will have a new option available: /containers

Step 1: Bridge, VETH, VRF and interface setup

First create a bridge which will be used for your containr. Let's call it 'bridge-ampr-gw': 

/interface bridge add comment="AMPR container" name=bridge-ampr-gw

Assign a network to it. The typical docker IP will be ok: 

/ip address add address=172.17.0.1/24 interface=bridge-ampr-gw

Create a virtual ethernet interface for the container itself (call it veth-ampr): 

/interface veth add name=veth-ampr address=172.17.0.2/24 comment="AMPR container interface" \
   gateway=172.17.0.1

Add the VETH port to the bridge we created above: 

/interface bridge port add bridge=bridge-ampr-gw interface=veth-ampr

Because of a kernel anomaly preventing proper userspace IPIP handling, we need to filter icmp messages on the bridge from the container itself: 

/interface bridge filter add action=drop chain=input in-interface=veth-ampr ip-protocol=icmp \
   mac-protocol=ip src-address=172.17.0.2/32

Now we create a vrf called "vrf-ampr" and add the bridge to it: 

/ip vrf add interfaces=bridge-ampr-gw name=vrf-ampr

All the above steps are available here as a rsc file: http://yo2loj.ro/containers/1_ampr_bridge_vrf.rsc

Step 2: RIP setup

First, create a simple accept routing filter to be used by RIP: 

/routing filter rule add chain=rip-ampr-in disabled=no rule="accept;"

Next, create a RIP instance for your VRF using the above filter and the defined VRF: 

/routing rip instance add afi=ipv4 in-filter-chain=rip-ampr-in name=rip-ampr vrf=vrf-ampr

And now add a passive (receive only) interface to our instance: 

/routing rip interface-template add instance=rip-ampr interfaces=bridge-ampr-gw mode=passive

All the above steps are available here as a rsc file: http://yo2loj.ro/containers/2_rip.rsc

Step 3: Firewall rules, Filter, NAT and Mangle

Now we need to forward out IPIP tunnels to the container, and extract our data from the VRF. For our convenience we will set up an address list to handle AMPR space as one entity (if you already have such a list on the router, you can use it) 

/ip firewall address-list
add address=44.0.0.0/9 list=ampr_addr
add address=44.128.0.0/10 list=ampr_addr

Also, we can use an interface list called WAN for the internet access interfaces (like the one in the default config). If you prefer individual interfaces, you can of course use them in your rules.

Filters are needed to allow data input and forward to/from the router. Accept RIP from the VRF: 

/ip firewall filter
add action=accept chain=input comment="RIP via VRF" dst-port=520 in-interface=vrf-ampr protocol=udp

Accept input from the AMPR address space to the router (important for ping and traceroute): 

/ip firewall filter
add action=accept chain=input comment="AMPR via Tunnels" dst-address-list=ampr_addr in-interface=vrf-ampr \
   src-address-list=ampr_addr

And we need to accept some forwarding for the IPIP tunnels, from VRF to our AMPR space and between AMPR hosts: 

/ip firewall filter
add action=accept chain=forward comment="IPIP Tunnels from ISP" in-interface-list=Internet protocol=ipencap
add action=accept chain=forward comment="IPIP Tunnels from VRF" in-interface=vrf-ampr protocol=ipencap
add action=accept chain=forward comment="VRF to AMPR" dst-address-list=ampr_addr in-interface=vrf-ampr
add action=accept chain=forward comment="AMPR to AMPR" dst-address-list=ampr_addr src-address-list=ampr_addr

Next, we need to forward incoming IPIP traffic to our container (note that WAN interface list, use your interface if you like): 

/ip firewall nat
add action=dst-nat chain=dstnat comment="NAT ENCAP" in-interface-list=WAN protocol=ipencap \
   to-addresses=172.17.0.2

Now to be able to traverse int and from the VRF, we need some mangle rules.

Incoming IPIP traffic will be marked with the vrf routing mark: 

/ip firewall mangle
add action=mark-routing chain=prerouting comment="AMPR IPIP incoming to VRF" in-interface-list=Internet \
   new-routing-mark=vrf-ampr passthrough=no protocol=ipencap

Outgoing IPIP traffic will be marked for the main routing table (or the one you need to reach your ISP) 

/ip firewall mangle
add action=mark-routing chain=prerouting comment="AMPR IPIP outgoing via ISP" in-interface=vrf-ampr \
   new-routing-mark=main passthrough=no protocol=ipencap

Traffic to our local router IPs will be directed via our bridge 

/ip firewall mangle
add action=route chain=prerouting comment="AMPR VRF route local" dst-address-type=local in-interface=vrf-ampr \
   passthrough=no route-dst=172.17.0.1

And finally, the incoming AMPR traffic will go to the main routing table 

/ip firewall mangle
add action=mark-routing chain=prerouting comment="AMPR VRF forward" in-interface=vrf-ampr new-routing-mark=main\
   passthrough=no

All the above steps are available here as a rsc file: http://yo2loj.ro/containers/3_firewall.rsc

Step 4 - Container environment setup

This step prepares the environment variables for the container.

/container envs 

add comment="My subnets, as defined in the portal" key=AMPR_SUBNETS name=ampr-cfg value=\
   44.128.0.0/24,44.128.1.0/24
add comment="Default gateway is AMPRGW instead of Internet" key=ALL_VIA_AMPRGW name=ampr-cfg value=0
add comment="Forward internet traffic" key=FORWARD_INTERNET name=ampr-cfg value=0
add comment="Call home callsign and locator" key=CALL_HOME name=ampr-cfg value=test@AA00aa
add comment="Ignored subnets in RIP" key=IGNORED_SUBNETS name=ampr-cfg value=44.128.0.0/16

Just paste them into the console or import the rsc script as it is. You will edit those later.

The rsc file is here: http://yo2loj.ro/containers/4_container_env.rsc

Step 5 - Container installation

Now you need to download the container which fits your architecture. The following rsc files will install a script which, when run will import and install the container. The same scripts will update and replace the container if run again. 

ARM32 -  http://yo2loj.ro/containers/5_container_arm32.rsc
ARM64 -  http://yo2loj.ro/containers/5_container_arm64.rsc
CHRx86 - http://yo2loj.ro/containers/5_container_x86_64.rsc

Of course, instead of the script, you can download the appropriate tar container files yourself: 

ARM32 -  http://yo2loj.ro/containers/ampr-arm32.tar
ARM64 -  http://yo2loj.ro/containers/ampr-arm64.tar
CHRx86 - http://yo2loj.ro/containers/ampr-x86-64.tar

and install them manually. Please set them to use the env variables "ampr-cfg".

Or, you can compile and pack the container yourself from source. At the time of writing, the current version is 1.2.0: 

http://yo2loj.ro/containers/ampr-container-1.2.0-release.tgz

Step 6 - Container configuration and final touches

You need to edit your env variables according to the description given in the new router setup.

Start your container and wait 5 min. You should see AMPR routes showing up in the VRF's routing table (Some 840 of them if ALL_VIA_AMPRGW is not enabled, otherwise you will get only 2 routes, 44.0.0.0/9 and 44.128.0.0/10).

You may notice that on an external traceroute your router's IP address will show up as 172.17.0.1. To fix this small glitch, you need to modify your existing "rip-ampr-in" RIP input filter rule to provide the correct preferred source address.

Modify the existing rule from 

accept;

to set your router's local AMPR IP as its preferred source 

set pref-src 44.128.0.1;
accept;

Now, activate the use of the whole system. Set up a routing rule set to force all outgoing AMPR traffic to do a lookup in the VRF routing table: 

/routing rule
add action=lookup disabled=yes dst-address=44.0.0.0/9 table=vrf-ampr
add action=lookup disabled=yes dst-address=44.128.0.0/10 table=vrf-ampr

If no route is found in that table, the lookup will continue via the main table towards your default route.

Your local AMPR network and additional routing will go into the main table and the lookup will be done AFTER passing through the VRF's routing table. This means that a matching route in the VRF, including a default 0.0.0.0/0 will take precedence over any route defined in the main table.

Also, if you want to access the AMPR network from a LAN not using AMPR addresses, you need to set up a forwarding rule and a SRC-NAT one: 

/ip firewall filter
add action=accept chain=forward comment="from LAN" in-interface=LAN dst-address-list=ampr_addr

and 

/ip firewall nat
add action=src-nat chain=srcnat comment="NAT to AMPR" dst-address-list=Ampr out-interface=bridge-ampr-gw \
   src-address-list=!ampr_addr to-addresses=<your router's AMPR ip>

Additional Info

All available files are here: http://yo2loj.ro/containers/

Rip Rip Hurray! de YO2LOJ

Retrieved from "https://wiki.ampr.org/w/index.php?title=Setting_up_a_gateway_in_a_ROS7_Mikrotik_router_container_on_arm32_arm64_and_x86-64&oldid=1177"