View source for Setting up a gateway in a ROS7 Mikrotik router container on arm32 arm64 and x86-64

Setting up a gateway in a ROS7 Mikrotik router running in a container on arm and arm64 models and x86-64 CHR

<span style="color: blue;">'''This is an experimental software build for the 'enthusiasts' out there.'''</span>

Source code is available in git.ampr.org: https://git.ampr.org/yo2loj/ampr-ros7-container/-/tree/main


<h1>Info</h1>

These are the steps for setting up a fully functional AMPR gateway on an arm/arm64 MikroTik router
Tested and found working on CRS2116 and RB3011 for now.

MikroTik <b>ARM64</b> devices:
 Routers: CCR2004, CCR2116, CCR2216, RB5009
 Switches: CRS520
 Wireless & 5G: Netmetal ax, LHG-LTE6, ATL-LTE18
 SOHO: hAP-ax2, cAP-ax, hAP-ax3, Chateau-ax
 Others: AMPERE

MikroTik <b>ARM32</b> devices:
 Routers: L009, RB3011, RB4011, RB1100AHx4, 
 Switches: CRS305, CRS309, CRS310, CRS317, CRS320, CRS326, CRS328
 Wireless & 5G: SXTsq-5ac, NetBox-5ax, LHGXL-5ac
 SOHO: hAP-ax lite, hap-ac2, cAP-ac, wAP-ac, cAPXL-ac, hAP-ac3, Chateau
 Routerboard: L11UG, L23UGSR, RB450Gx4

MikroTik <b>x86-64</b> devices:
 Others: Cloud Hosted Router

Containers are not available on MIPSBE, MMIPS, SMIPS, TILE or PPC architectures.

== General concept ==
Mikrotik routers running ROS 7 (7.15.3 being current at the time of writing) based on arm and arm64 processor, as well as CHR setups are able to run software containers (similar to docker).
This opens the possibility to host a virtualized gateway in such a container, allowing a simple and efficient setup on modern systems.

The gateway will be hosted in a VRF on the router, providing gateway services using policy routing.

As a concept, the container has a single VETH interface which will decapsulate all incoming IPIP traffic from the tunnels, and encapsulate all outgoing traffic towards them.
The container itself is isolated behind a bridge and offers some basic filtering function (e.g. restrict access from internet hosts).
It will receive the RIPv2 broadcasts from the AMPR gateway and provide the obtained routes as RIP broadcasts to the router itself inside the mentioned VRF.

The container does not save anything to disk (which would be a bad idea on the router's flash memory), so the AMPR routes are lost on container or router restart, and you need to wait the now classical 5 minutes. But this should be no problem on a 24/7 on router.

== Limitations ==

The router does not forward multicast frames at all, nor does it send out broadcasts.
Incoming broadcasts are accepted and forwarded to the local VRF.

The container itself needs to sit behind a bridge due to a kernel bug in the version used by Mikrotik which sends out "Port unreachable" ICMP messages on incoming IPIP traffic if it is handled in user space (The same thing causing the need of a kernel filter in amprd. This is fixed in newer kernel releases but it will take a while for it to make its way into ROS). Bridge filtering is used to mask those messages.

<h1>New router 5 minutes set up</h1>

As a prerequisite, get your internet connection working based on the default mikrotik configuration.
Basically set up your ISP uplink either via DHCP or by setting up a PPPoE or similar connection.
Leave the default firewall rule as they are.
Alternatively, you can start with a completely empty router, with only a active internet connection.

First you need to enable container support according to the info provided by Mikrotik.
In a console type in:
 /system/device-mode/update container=yes
The system will want you to do a hard reset at this point to confirm the request. This means you need physical access to the device.

Next, you need to install the container package for your firmware version. Download the "extra" firmware package from MikroTik for your FW version and extract the "container-7.x.y-<arch>.npk" file. Upload it to your router and restart. This will install the package onto the router. After restart, you will have a new option available: /containers

== Installation script ==
Next we need to install the container according to your hardware.
Please chose the correct setup script variant:

 <span style="color: blue;">ARM32</span> -  ampr_arm32.rsc
 <span style="color: blue;">ARM64</span> -  ampr_arm64.rsc
 <span style="color: blue;">CHRx86</span> - ampr_x86_64.rsc

Unfortunately, containers are not available on Mips, Tile or PowerPC devices.

The example assumes you use an arm32 device. Please use the proper one...

Open a route console window.

1. Check is the remote server is available:
 [admin@MikroTik] > ping yo2loj.ro
  SEQ HOST                                     SIZE TTL TIME       STATUS                    
    0 89.33.44.100                               56  58 10ms574us 
    1 89.33.44.100                               56  58 9ms141us  
    2 89.33.44.100                               56  58 9ms5us    
    sent=3 received=3 packet-loss=0% min-rtt=9ms5us avg-rtt=9ms573us max-rtt=10ms574us

2. Download the configuration script
 [admin@MikroTik] > /tool fetch url="http://yo2loj.ro/containers/<span style="color: red;">ampr_arm32.rsc</span>"
      status: finished
  downloaded: 5KiB
       total: 5KiB
    duration: 1s

3. Run the configuration script
 [admin@MikroTik] > import <span style="color: red;">ampr_arm32.rsc</span>
 AMPR: Creating bridge and VRF
 AMPR: Setting up RIP
 AMPR: Creating container envs
 AMPR: Setting up firewall rules
 AMPR: Creating container update script
 AMPR: Creating routing rules
 AMPR: Installing container
 No container is installed
      status: finished
  downloaded: 366KiB
       total: 366KiB
    duration: 1s
 <span style="color: blue;">AMPR: Script finished successful
 AMPR: Now update your container envs and start the container</span>

Your container is now installed.
You need to configure its environment variables according to the description given below.

After configuration is complete, go to "containers" and star it up.
It should show "running" and you should see it's messages in the log window.

After at most 5 minutes, you should get the tunnel routes in your vrf, and your gateway should be fully up and running.

If logging/debugging is not needed anymore, please disable it by clicking on the container and unchecking te logging box.

== Container configuration parameters ==
You need to adapt the pre-existing container environment variables to your particular gateway before starting it again.
The following ENV parameters are preset in Container-> Envs:
 <span style="color: blue;">AMPR_SUBNETS</span> - holds your local subnets as defined in the portal, as comma separated list of <SUBNET>/<MASK> tupples, e.g. "44.128.0.0/24,44.128.1.0/24"
 <span style="color: blue;">ALL_VIA_AMPRGW</span> - enables forwarding of all AMPR destinations via AMPRGW, values are "0" or "1"
 <span style="color: blue;">FORWARD_INTERNET</span> - enables forward of traffic from/to internet hosts, values are "0" or "1"
 <span style="color: blue;">IGNORED_SUBNETS</span> - allows you to ignore specific subnets provided by RIP, by <SUBNET>/<MASK> or gateway address e.g. "44.128.0.0/16"
 <span style="color: blue;">CALL_HOME</span> - the classic string, <CALLSIGN>@<LOCATOR> to show up on the map. You will get a yellow dot. e.g. "YO2LOJ@KN05OR". Leaving the field empty disables call home.

Please note that the provided default will allow you to play around, but will not provide a working set up.

Next, you need to set up a local AMPR LAN on your router router, or, if you have only a single IP address assigned, add it to one of your router's interfaces with a /32 netmask
Anyway, you need to add a src-nat rule to the router's IP address to get your traffic flowing (let's assume its 44.128.0.1).
<span style="color: red;">The address shall be set on an interface OUTSIDE OF THE VRF</span>.

For a single address:
 /ip address add address=<span style="color: red;">44.128.0.1</span> interface=bridge
or even on the loopback interface:
For a single address:
 /ip address add address=<span style="color: red;">44.128.0.1</span> interface=lo

For a subnet:
 /ip address add address=<span style="color: red;">44.128.0.1/24</span> interface=<span style="color: red;"><interface name></span>

And your src-nat NAT rules:
 /ip firewall nat add action=src-nat chain=srcnat out-interface=bridge-ampr-gw to-addresses=<span style="color: red;">44.128.0.1</span>
 /ip firewall nat add action=src-nat chain=srcnat out-interface=vrf-ampr to-addresses=<span style="color: red;">44.128.0.1</span>

Of course you need to set up additional firewall rules & stuff, but if you do not enable internet forwarding, you should be pretty safe being exposed only to AMPR partners.

<span style="color: blue;">Please note that for your firewall rules the incoming interface from the tunnels is "<span style="color: red;">vrf_ampr</span>" and the outgoing interface is "<span style="color: red;">bridge-ampr-gw</span>" for forwarded and "<span style="color: red;">vrf-ampr</span>" for local outgoing data.</span>

== Additional optional configuration ==

You may notice that on an external traceroute your router's IP address will show up as 172.17.0.1.
To fix this small glitch, you need to modify your existing "rip-ampr-in" RIP input filter rule to provide the correct preferred source address.

Modify the existing rule from
 accept;

to set your router's local AMPR IP as its preferred source
 set pref-src <span style="color: red;">44.128.0.1</span>;
 accept;

Also, if you want to access the AMPR network from a LAN not using AMPR addresses, you need to set up a forwarding rule and a SRC-NAT one:

 /ip firewall filter
 add action=accept chain=forward comment="from LAN" in-interface=<span style="color: red;"><YourLANInterface></span> dst-address-list=ampr_addr
and
 /ip firewall nat
 add action=src-nat chain=srcnat comment="NAT to AMPR" dst-address-list=Ampr out-interface=bridge-ampr-gw \
    src-address-list=!ampr_addr to-addresses=<span style="color: red;">44.128.0.1</span>

<h1>Configuration on an existing working router - 6 steps</h1>

 1 - Bridge, VETH, VRF and interface setup
 2 - RIP setup
 3 - Firewall rules, Filter, NAT and Mangle
 4 - Container environment setup
 5 - Container installation (architecture dependent)
 6 - Container configuration and final touches

== Preliminary: prepare the router to accept containers ==
First, you need to install container support on your router.
In a console issue:
 /system/device-mode/update container=yes
The system will want you to do a hard reset at this point to confirm the request.
This means you need physical access to the device.

Next, you need to install the container package for your firmware version.
Download the "extra" firmware package from MikroTik for your FW version and extract the "container-7.x.y-<arch>.npk" file.
Upload it to your router and restart. This will install the package onto the router.
After restart, you will have a new option available: /containers

== Step 1: Bridge, VETH, VRF and interface setup ==

First create a bridge which will be used for your containr. Let's call it 'bridge-ampr-gw':
 /interface bridge add comment="AMPR container" name=bridge-ampr-gw
Assign a network to it. The typical docker IP will be ok:
 /ip address add address=172.17.0.1/24 interface=bridge-ampr-gw
Create a virtual ethernet interface for the container itself (call it veth-ampr):
 /interface veth add name=veth-ampr address=172.17.0.2/24 comment="AMPR container interface" \
    gateway=172.17.0.1
Add the VETH port to the bridge we created above:
 /interface bridge port add bridge=bridge-ampr-gw interface=veth-ampr
Because of a kernel anomaly preventing proper userspace IPIP handling, we need to filter icmp messages on the bridge from the container itself:
 /interface bridge filter add action=drop chain=input in-interface=veth-ampr ip-protocol=icmp \
    mac-protocol=ip src-address=172.17.0.2/32
Now we create a vrf called "vrf-ampr" and add the bridge to it:
 /ip vrf add interfaces=bridge-ampr-gw name=vrf-ampr

All the above steps are available here as a rsc file: http://yo2loj.ro/containers/1_ampr_bridge_vrf.rsc

== Step 2: RIP setup ==

First, create a simple accept routing filter to be used by RIP:
 /routing filter rule add chain=rip-ampr-in disabled=no rule="accept;"
Next, create a RIP instance for your VRF using the above filter and the defined VRF:
 /routing rip instance add afi=ipv4 in-filter-chain=rip-ampr-in name=rip-ampr vrf=vrf-ampr
And now add a passive (receive only) interface to our instance:
 /routing rip interface-template add instance=rip-ampr interfaces=bridge-ampr-gw mode=passive

All the above steps are available here as a rsc file: http://yo2loj.ro/containers/2_rip.rsc

== Step 3: Firewall rules, Filter, NAT and Mangle ==

Now we need to forward out IPIP tunnels to the container, and extract our data from the VRF.
For our convenience we will set up an address list to handle AMPR space as one entity
(if you already have such a list on the router, you can use it)
 /ip firewall address-list
 add address=44.0.0.0/9 list=ampr_addr
 add address=44.128.0.0/10 list=ampr_addr
Also, we can use an interface list called WAN for the internet access interfaces (like the one in the default config).
If you prefer individual interfaces, you can of course use them in your rules.

Filters are needed to allow data input and forward to/from the router.
Accept RIP from the VRF:
 /ip firewall filter
 add action=accept chain=input comment="RIP via VRF" dst-port=520 in-interface=vrf-ampr protocol=udp
Accept input from the AMPR address space to the router (important for ping and traceroute):
 /ip firewall filter
 add action=accept chain=input comment="AMPR via Tunnels" dst-address-list=ampr_addr in-interface=vrf-ampr \
    src-address-list=ampr_addr
And we need to accept some forwarding for the IPIP tunnels, from VRF to our AMPR space and between AMPR hosts:
 /ip firewall filter
 add action=accept chain=forward comment="IPIP Tunnels from ISP" in-interface-list=Internet protocol=ipencap
 add action=accept chain=forward comment="IPIP Tunnels from VRF" in-interface=vrf-ampr protocol=ipencap
 add action=accept chain=forward comment="VRF to AMPR" dst-address-list=ampr_addr in-interface=vrf-ampr
 add action=accept chain=forward comment="AMPR to AMPR" dst-address-list=ampr_addr src-address-list=ampr_addr

Next, we need to forward incoming IPIP traffic to our container (note that WAN interface list, use your interface if you like):
 /ip firewall nat
 add action=dst-nat chain=dstnat comment="NAT ENCAP" in-interface-list=WAN protocol=ipencap \
    to-addresses=172.17.0.2

Now to be able to traverse int and from the VRF, we need some mangle rules.

Incoming IPIP traffic will be marked with the vrf routing mark:
 /ip firewall mangle
 add action=mark-routing chain=prerouting comment="AMPR IPIP incoming to VRF" in-interface-list=Internet \
    new-routing-mark=vrf-ampr passthrough=no protocol=ipencap
Outgoing IPIP traffic will be marked for the main routing table (or the one you need to reach your ISP)
 /ip firewall mangle
 add action=mark-routing chain=prerouting comment="AMPR IPIP outgoing via ISP" in-interface=vrf-ampr \
    new-routing-mark=main passthrough=no protocol=ipencap
Traffic to our local router IPs will be directed via our bridge
 /ip firewall mangle
 add action=route chain=prerouting comment="AMPR VRF route local" dst-address-type=local in-interface=vrf-ampr \
    passthrough=no route-dst=172.17.0.1
And finally, the incoming AMPR traffic will go to the main routing table
 /ip firewall mangle
 add action=mark-routing chain=prerouting comment="AMPR VRF forward" in-interface=vrf-ampr new-routing-mark=main\
    passthrough=no

All the above steps are available here as a rsc file: http://yo2loj.ro/containers/3_firewall.rsc

== Step 4 - Container environment setup ==
This step prepares the environment variables for the container.

/container envs
 add comment="My subnets, as defined in the portal" key=AMPR_SUBNETS name=ampr-cfg value=\
    44.128.0.0/24,44.128.1.0/24
 add comment="Default gateway is AMPRGW instead of Internet" key=ALL_VIA_AMPRGW name=ampr-cfg value=0
 add comment="Forward internet traffic" key=FORWARD_INTERNET name=ampr-cfg value=0
 add comment="Call home callsign and locator" key=CALL_HOME name=ampr-cfg value=test@AA00aa
 add comment="Ignored subnets in RIP" key=IGNORED_SUBNETS name=ampr-cfg value=44.128.0.0/16

Just paste them into the console or import the rsc script as it is. You will edit those later.

The rsc file is here: http://yo2loj.ro/containers/4_container_env.rsc

== Step 5 - Container installation ==
Now you need to download the container which fits your architecture.
The following rsc files will install a script which, when run will import and install the container.
The same scripts will update and replace the container if run again.

 <span style="color: blue;">ARM32</span> -  http://yo2loj.ro/containers/5_container_arm32.rsc
 <span style="color: blue;">ARM64</span> -  http://yo2loj.ro/containers/5_container_arm64.rsc
 <span style="color: blue;">CHRx86</span> - http://yo2loj.ro/containers/5_container_x86_64.rsc

Of course, instead of the script, you can download the appropriate tar container files yourself:
 <span style="color: blue;">ARM32</span> -  http://yo2loj.ro/containers/ampr-arm32.tar
 <span style="color: blue;">ARM64</span> -  http://yo2loj.ro/containers/ampr-arm64.tar
 <span style="color: blue;">CHRx86</span> - http://yo2loj.ro/containers/ampr-x86-64.tar
and install them manually. Please set them to use the env variables "ampr-cfg".

Or, you can compile and pack the container yourself from source. At the time of writing, the current version is 1.2.0:
 http://yo2loj.ro/containers/ampr-container-1.2.0-release.tgz

== Step 6 - Container configuration and final touches ==

You need to edit your env variables <span style="color: blue;">according to the description given in the new router setup</span>.

Start your container and wait 5 min. You should see AMPR routes showing up in the VRF's routing table
(Some 840 of them if ALL_VIA_AMPRGW is not enabled, otherwise you will get only 2 routes, 44.0.0.0/9 and 44.128.0.0/10).

You may notice that on an external traceroute your router's IP address will show up as 172.17.0.1.
To fix this small glitch, you need to modify your existing "rip-ampr-in" RIP input filter rule to provide the correct preferred source address.

Modify the existing rule from
 accept;

to set your router's local AMPR IP as its preferred source
 set pref-src <span style="color: red;">44.128.0.1</span>;
 accept;

Now, activate the use of the whole system.
Set up a routing rule set to force all outgoing AMPR traffic to do a lookup in the VRF routing table:
 /routing rule
 add action=lookup disabled=yes dst-address=44.0.0.0/9 table=vrf-ampr
 add action=lookup disabled=yes dst-address=44.128.0.0/10 table=vrf-ampr
If no route is found in that table, the lookup will continue via the main table towards your default route.

Your local AMPR network and additional routing will go into the main table and the lookup will be done AFTER passing through the VRF's routing table.
This means that a matching route in the VRF, including a default 0.0.0.0/0 will take precedence over any route defined in the main table.

Also, if you want to access the AMPR network from a LAN not using AMPR addresses, you need to set up a forwarding rule and a SRC-NAT one:

 /ip firewall filter
 add action=accept chain=forward comment="from LAN" in-interface=LAN dst-address-list=ampr_addr
and
 /ip firewall nat
 add action=src-nat chain=srcnat comment="NAT to AMPR" dst-address-list=Ampr out-interface=bridge-ampr-gw \
    src-address-list=!ampr_addr to-addresses=<your router's AMPR ip>

== Additional Info ==
All available files are here: http://yo2loj.ro/containers/

Rip Rip Hurray! de YO2LOJ