Virtual Extensible LAN (VXLAN) is a Layer 2 overlay scheme utilizing a Layer 3 network. VXLAN uses MAC Address-in-User Datagram Protocol (MAC-in-UDP) encapsulation to provide a means to extend Layer 2 segments across a Layer 3 segment. This basically means the Layer 2 packet gets a VXLAN header applied, then that frame gets encapsulated into a UDP IP packet and sent over to the Layer 3 network.
In later 5.4 FortiOS firmwares, VXLAN encapsulation was added. This is a great technology that can help connect to sites at Layer 2 over Layer 3. Something to take note of: FortiOS 5.6.2 has lots of improvements and enhancements to VXLAN encapsulation. For example, VLAN trunking works very well now, and multicast also will traverse the VXLAN!
I’ve set this up a couple of times and both were situations where we had to have Layer 2 stretched for a certain purpose, most recently to two different data centers. Learn more about choosing between stretching your Layer 2 or utilizing VXLAN.
Below is the aforementioned scenario of connecting two datacenters, as well as the config of the FortiGates and the ARP/MAC from the Cisco switch. Fortinet has some great documentation as well on this feature:
- Technical Note: Basic VXLAN over IPsec configuration
- Technical Note: Building a Layer-2 VPN with VxLAN over IPsec
Below you can see our simple layout. The red line indicates the VXLAN encapsulation path. Encapsulation only happens at the FortiGates.
What you need:
1. Create VXLAN VPN
- Local encap-local-gw4 is the public address on the local FW
- “encap-remote-gw4” is the peer address of the other side
- “remote-gw” is the peer address of the other side
2. Create a new switch interface
- Add both the local network and VXLAN-VPN interface to this switch.
- Create firewall policies allow traffic.
Recommendations
- Lowering the MTU of the VXLAN/internal interface might be a good idea as the VXLAN encapsulation can add around 50 bytes. Most Cisco documentation will suggest increasing the MTU, but since we are going over the net with this, increasing MTU might cause lots of fragmentation.
- No IP address on the switch interface is needed, and I have actually seen issues arise from putting an IP address on the interface.
- In CLI, use the commands below to help get broadcasts (but be careful) and ARP to go across:
- config sys int
- edit VXLAN
- set l2forward enable
- set broadcast-foward enable
- end
- end
- edit VXLAN
- config sys int
- In 5.6.2, VLANs tags will pass through the tunnel
Configuration for the FortiGates
SIDE 1 (60D)
config vpn ipsec phase1-interface
edit “VXLAN”
set interface “wan2”
set peertype any
set proposal aes256-sha1
set encapsulation vxlan
set encapsulation-address ipv4
set encap-local-gw4 1.1.1.1
set encap-remote-gw4 1.1.1.2
set remote-gw 1.1.1.2
set psksecret password
next
end
config vpn ipsec phase2-interface
edit “VXLAN_ph2”
set phase1name “VXLAN”
set proposal aes256-sha1
next
end
config system switch-interface
edit “VXLAN-SWITCH”
set vdom “root”
set member “internal1” “internal2” “VXLAN”
next
end
Here is the switch in the FortiOS GUI:
Check the firewall policies:
SIDE 2 (60E)
config vpn ipsec phase1-interface
edit “VXLAN”
set interface “wan1”
set peertype any
set proposal aes256-sha1
set encapsulation vxlan
set encapsulation-address ipv4
set encap-local-gw4 1.1.1.2
set encap-remote-gw4 1.1.1.1
set remote-gw 1.1.1.1
set psksecret password
next
end
config vpn ipsec phase2-interface
edit “VXLAN_ph2”
set phase1name “VXLAN”
set proposal aes256-sha1
next
end
Again, check the switch in the GUI:
And then the firewall policies:
Testing
A simple ping test between two devices on the same subnet will be enough to make sure things are working. TCP is always the best way to test. You can also check and make sure that the ARP/MAC address tables on each side show something on the remote side. For example, the below shows the ARP/MAC of the Cisco 3650 switch at the datacenter side (60D):
Datacenter-Stack#show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.19.21 0 000c.291c.b2a5 ARPA Vlan1
Internet 192.168.19.51 0 000c.2918.b8be ARPA Vlan1 – 19.51 lives behind the 60E
Datacenter-Stack#show mac address-table
Mac Address Table
——————————————-
Vlan Mac Address Type Ports
—- ———– ——– —–
1 000c.2918.b8be DYNAMIC Gi1/0/1 — Fortinet 60D is connected to gig 1/0/1
Wrapping Up
That’s it! VXLAN is an open source protocol that is a great datacenter technology. Fortinet makes it very easy to get this up quickly.