Overview
With SDDC version 1.18, VMware released the capability to add additional Tier-1 gateways inside of the Software Defined Data Center (SDDC) for VMware Cloud on AWS. Since inception of the service, customers have not had the choice to deploy different typologies inside of the SDDC. Prior to 1.18, SDDCs were deployed with a single compute gateway and a management gateway. Now, customers have additional capabilities to consider in designing their SDDC. Customers can now have overlapping IP space in the SDDC, use zone/perimeter firewalling, implement gateway specific VPNs, or other features. This post will focus on the various gateway types, functions, and a route validation inside of the SDDC.
The T1 gateways ultimately come in several varieties:
- Management Gateway
- Required
- Deployed by VMware
- Limited Policy Management by Customer
- Compute Gateway
- Required
- Deployed by VMware
- Full Policy Management by Customer
- Additional Compute Gateway – Routed/Isolated/NATed
- Optional
- Deployed by Customer
- Full Policy Management by Customer
- up to 200 additional Tier-1 Gateways Per SDDC
- Can be a VPN Endpoint for 1 VPN per Gateway
Note that I will use the NSX Manager for the examples. Some features of NSX are not available in the Networking & Security tab of the SDDC. Also, the VMware Configuration Maximums tool is updated to include multi CGW considerations.
Example Topology
The Tier-1 gateways can be configured from the NSX Manager, the VMware Cloud Service Portal, or API.
Validating Connectivity Between Networks
Each T1 gateway has a gateway firewall associated with it, logically filtering traffic between the T0 and the T1. Additionally, NATed tier-1 gateways need 1 or more NAT statements created to adequately allow the traffic flows. I will validate the inter-network connectivity considerations in the following sections.
Traffic Between Two Segments Connected to the Same Tier-1
A compute gateway firewall policy will never block this traffic. The distributed firewall is the way to go in order to filter traffic between two segments in many (MANY) cases. The traffic never hits the CGW firewall rule set, therefore cannot generate logs in vRealize Log Insight Cloud either.
root@cgw-42 [ ~ ]# tcpdump -n icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:52:05.896382 IP 192.168.41.11 > 192.168.42.10: ICMP echo request, id 1, seq 71, length 64 15:52:05.896410 IP 192.168.42.10 > 192.168.41.11: ICMP echo reply, id 1, seq 71, length 64 15:52:06.920286 IP 192.168.41.11 > 192.168.42.10: ICMP echo request, id 1, seq 72, length 64 15:52:06.920337 IP 192.168.42.10 > 192.168.41.11: ICMP echo reply, id 1, seq 72, length 64
After applying a compute gateway firewall rule
Traffic Between Two Segments Connected to Different Routed T1s
If two segments are connected to different T1s, the traffic by default is allowed.
root@prod-21 [ ~ ]# tcpdump -n icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:54:24.646609 IP 192.168.41.11 > 192.168.21.10: ICMP echo request, id 2, seq 33, length 64 15:54:24.646648 IP 192.168.21.10 > 192.168.41.11: ICMP echo reply, id 2, seq 33, length 64 15:54:25.670674 IP 192.168.41.11 > 192.168.21.10: ICMP echo request, id 2, seq 34, length 64 15:54:25.670712 IP 192.168.21.10 > 192.168.41.11: ICMP echo reply, id 2, seq 34, length 64
If I attempt to block the traffic with the default compute gateway firewall (CGW) I am not able to. This is due to the logical location of the CGW firewall being on the Tier-0 edge. For more information about that, refer to the VMC on AWS Network Security Techzone article.
root@cgw-41 [ ~ ]# tcpdump -n icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 16:00:16.376465 IP 192.168.41.11 > 192.168.21.10: ICMP echo request, id 3, seq 8, length 64 16:00:16.377013 IP 192.168.21.10 > 192.168.41.11: ICMP echo reply, id 3, seq 8, length 64 16:00:17.400451 IP 192.168.41.11 > 192.168.21.10: ICMP echo request, id 3, seq 9, length 64 16:00:17.400951 IP 192.168.21.10 > 192.168.41.11: ICMP echo reply, id 3, seq 9, length 64 16:00:18.424412 IP 192.168.41.11 > 192.168.21.10: ICMP echo request, id 3, seq 10, length 64
If I attempt to block the traffic with the default compute gateway firewall I am not able to. Now if I configure a firewall rule on the customer defined T1, the traffic can be blocked from a compute firewall. If a customer wanted to have multiple layers of gateway firewalling, additional T1s would help. Still, the distributed firewall generally is a more manageable solution in my experience.
root@cgw-41 [ ~ ]# tcpdump -n icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode 15:56:41.049098 IP 192.168.21.10 > 192.168.41.11: ICMP echo reply, id 2, seq 166, length 64 15:56:42.072438 IP 192.168.41.11 > 192.168.21.10: ICMP echo request, id 2, seq 167, length 64 15:56:42.072992 IP 192.168.21.10 > 192.168.41.11: ICMP echo reply, id 2, seq 167, length 64 15:56:43.096434 IP 192.168.41.11 > 192.168.21.10: ICMP echo request, id 2, seq 168, length 64 15:56:43.097018 IP 192.168.21.10 > 192.168.41.11: ICMP echo reply, id 2, seq 168, length 64 15:56:44.120440 IP 192.168.41.11 > 192.168.21.10: ICMP echo request, id 2, seq 169, length 64 15:56:45.144432 IP 192.168.41.11 > 192.168.21.10: ICMP echo request, id 2, seq 170, length 64 15:56:46.168451 IP 192.168.41.11 > 192.168.21.10: ICMP echo request, id 2, seq 171, length 64 15:56:47.192415 IP 192.168.41.11 > 192.168.21.10: ICMP echo request, id 2, seq 172, length 64
Now I can also generate logs
<99>1 2022-09-29T15:56:57.148Z NSX-Edge-1-10-50-192-8 NSX 5085 FIREWALL [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="firewallpkt" level="INFO"] <8 dd44b2191da84bca:b56040e929e722fe> INET reason-match REJECT 2032 OUT 56 PROTO 1 192.168.21.10->192.168.41.11
Working With a NATed Gateway
NATed gateways add an additional level of consideration. NATed gateways can be very useful for overlapping IP address ranges. I wanted to understand the differences between source NAT (SNAT) and destination NAT (DNAT), as well as the order of operations.
In this example, I am generating traffic from 192.168.41.11
root@cgw-41 [ ~ ]# ping 192.168.31.10 PING 192.168.31.10 (192.168.31.10) 56(84) bytes of data. --- 192.168.31.10 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 3063ms
If I create a NAT Statement, where as I want to NAT 172.16.31.0/24 -> 192.168.31.0/24. Initially I set this as a DNAT.
Now, the traffic is allowed
root@cgw-41 [ ~ ]# ping 172.16.31.10 PING 172.16.31.10 (172.16.31.10) 56(84) bytes of data. 64 bytes from 172.16.31.10: icmp_seq=1 ttl=61 time=0.664 ms 64 bytes from 172.16.31.10: icmp_seq=2 ttl=61 time=0.704 ms 64 bytes from 172.16.31.10: icmp_seq=3 ttl=61 time=0.618 ms --- 172.16.31.10 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2036ms
If I try the reverse traffic (IE logging into 192.168.31.10 and pinging 192.168.41.11
root@prod-n-31 [ ~ ]# ping 192.168.41.11 PING 192.168.41.11 (192.168.41.11) 56(84) bytes of data. --- 192.168.41.11 ping statistics --- 8 packets transmitted, 0 received, 100% packet loss, time 7173ms
Again, the traffic is not getting through. Lets add a source NAT (SNAT)
root@prod-n-31 [ ~ ]# ping 192.168.41.11 PING 192.168.41.11 (192.168.41.11) 56(84) bytes of data. 64 bytes from 192.168.41.11: icmp_seq=1 ttl=61 time=0.350 ms 64 bytes from 192.168.41.11: icmp_seq=2 ttl=61 time=0.606 ms 64 bytes from 192.168.41.11: icmp_seq=3 ttl=61 time=0.623 ms 64 bytes from 192.168.41.11: icmp_seq=4 ttl=61 time=0.561 ms 64 bytes from 192.168.41.11: icmp_seq=5 ttl=61 time=0.633 ms --- 192.168.41.11 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4094ms
NAT Gateways and the Gateway Firewall
Next I wanted to test the order of operations in relation to NATed Gateways. If I apply a firewall policy, should it be applied to the original source, or the NAT IP address?
First, I apply a firewall policy to the Tier-1 NATed Gateway
The traffic is blocked. Note that when i set the “Action” to Reject, an ICMP destination Unreachable (code 10) response is returned rather than drop, which returns nothing at all.
root@cgw-41 [ ~ ]# tcpdump -n icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 16:16:45.816426 IP 192.168.41.11 > 172.16.31.10: ICMP echo request, id 6, seq 87, length 64 16:16:45.817070 IP 172.16.31.10 > 192.168.41.11: ICMP host 192.168.31.10 unreachable - admin prohibited, length 36 16:16:46.840417 IP 192.168.41.11 > 172.16.31.10: ICMP echo request, id 6, seq 88, length 64 16:16:46.840893 IP 172.16.31.10 > 192.168.41.11: ICMP host 192.168.31.10 unreachable - admin prohibited, length 36 16:16:47.864464 IP 192.168.41.11 > 172.16.31.10: ICMP echo request, id 6, seq 89, length 64 16:16:47.864972 IP 172.16.31.10 > 192.168.41.11: ICMP host 192.168.31.10 unreachable - admin prohibited, length 36 16:16:48.888432 IP 192.168.41.11 > 172.16.31.10: ICMP echo request, id 6, seq 90, length 64 16:16:48.888907 IP 172.16.31.10 > 192.168.41.11: ICMP host 192.168.31.10 unreachable - admin prohibited, length 36 8 packets captured 8 packets received by filter 0 packets dropped by kernel
In validation, if I set the firewall policy to the NATed IP address range (172.16.31.0/24), the traffic is not blocked. The firewall policy default is to “Match Internal Address” but is configurable in the NAT configuration. If I set the firewall policy to “Match External IP Address” the fireall statement above would block the traffic. Using the internal IP address would make sense under certain circumstances such as reusing the same groups in a distributed firewall policy. It may make sense to use the external IP address firewall setting if I had overlapping IP addresses. Design and implementation is flexible for many customer use cases.
Isolated Gateways
Isolated gateways can be useful for building isolated disaster recovery environments, development pods, or any other use case where you may want to group networks together to talk together but not the external world.
Traffic between networks behind the same isolated gateway will be allowed, and there is no gateway firewall policy that will block the traffic (use the distributed firewall policy).
To validate the routing, I can use the API or the Python Client for VMware Cloud on AWS. From the Python Client, note the following lines:
- Default Compute Gateway Networks: Lines 59 and 159
- Routed-Production Networks: Lines 49 and 149
- NATed-Production: Lines 109 and 114
Note that there are no routes for 192.168.101.0/24 or 192.168.102.0/24 for the isolated T1 Gateway.
laptop:pyVMC user$ pyVMC.py show-t0-routes --------------------------------------- Route type: t1s Network: 10.50.192.3/32 Admin distance: 3 Next hop: 100.64.0.3 --------------------------------------- Route type: t0c Network: 100.64.0.4/31 Admin distance: 0 Next hop: --------------------------------------- Route type: t0s Network: 0.0.0.0/0 Admin distance: 200 Next hop: 10.50.160.1 --------------------------------------- Route type: t1s Network: 10.50.192.2/32 Admin distance: 3 Next hop: 100.64.0.3 --------------------------------------- Route type: t1c Network: 10.50.224.0/19 Admin distance: 3 Next hop: 100.64.0.3 --------------------------------------- Route type: t1c Network: 10.50.32.0/20 Admin distance: 3 Next hop: 100.64.0.3 --------------------------------------- Route type: t1c Network: 100.96.1.0/30 Admin distance: 3 Next hop: 100.64.0.1 --------------------------------------- Route type: t0c Network: 169.254.10.0/24 Admin distance: 0 Next hop: --------------------------------------- Route type: t1s Network: 10.50.192.4/32 Admin distance: 3 Next hop: 100.64.0.3 --------------------------------------- Route type: t1c Network: 192.168.21.0/24 Admin distance: 3 Next hop: 100.64.0.5 --------------------------------------- Route type: t0c Network: fccc:64d6:a1ec:b400::/64 Admin distance: 0 Next hop: --------------------------------------- Route type: t1c Network: 192.168.42.0/24 Admin distance: 3 Next hop: 100.64.0.1 --------------------------------------- Route type: t0c Network: 100.64.0.6/31 Admin distance: 0 Next hop: --------------------------------------- Route type: t0s Network: 10.50.112.0/20 Admin distance: 1 Next hop: 10.50.176.1 --------------------------------------- Route type: t0c Network: fccc:64d6:a1ec:e800::/64 Admin distance: 0 Next hop: --------------------------------------- Route type: t0c Network: 10.50.176.0/21 Admin distance: 0 Next hop: --------------------------------------- Route type: t0n Network: 172.31.7.248/32 Admin distance: 2 Next hop: --------------------------------------- Route type: t0s Network: 10.50.192.11/32 Admin distance: 1 Next hop: 100.64.0.3 --------------------------------------- Route type: b Network: 10.50.0.0/16 Admin distance: 200 Next hop: --------------------------------------- Route type: t0c Network: 169.254.11.0/24 Admin distance: 0 Next hop: --------------------------------------- Route type: t1c Network: 169.254.101.0/30 Admin distance: 3 Next hop: 100.64.0.3 --------------------------------------- Route type: t1n Network: 172.16.31.0/24 Admin distance: 3 Next hop: 100.64.0.7 --------------------------------------- Route type: t1n Network: 172.16.32.0/24 Admin distance: 3 Next hop: 100.64.0.7 --------------------------------------- Route type: t1s Network: 10.50.192.6/32 Admin distance: 3 Next hop: 100.64.0.3 --------------------------------------- Route type: t1s Network: 10.50.0.0/17 Admin distance: 3 Next hop: 100.64.0.3 --------------------------------------- Route type: t1n Network: 172.16.32.0/32 Admin distance: 3 Next hop: 100.64.0.7 --------------------------------------- Route type: t1s Network: 10.50.128.0/19 Admin distance: 3 Next hop: 100.64.0.3 --------------------------------------- Route type: t0c Network: 34.218.151.110/32 Admin distance: 0 Next hop: --------------------------------------- Route type: t0c Network: 169.254.0.0/24 Admin distance: 0 Next hop: --------------------------------------- Route type: t1c Network: 192.168.22.0/24 Admin distance: 3 Next hop: 100.64.0.5 --------------------------------------- Route type: t0c Network: fccc:64d6:a1ec:b401::/64 Admin distance: 0 Next hop: --------------------------------------- Route type: t1c Network: 192.168.41.0/24 Admin distance: 3 Next hop: 100.64.0.1 --------------------------------------- Route type: t0c Network: 169.254.13.0/24 Admin distance: 0 Next hop: --------------------------------------- Route type: t0c Network: fe80::/64 Admin distance: 0 Next hop: --------------------------------------- Route type: t0c Network: 100.64.0.0/31 Admin distance: 0 Next hop: --------------------------------------- Route type: t0c Network: fccc:64d6:a1ec:5c00::/64 Admin distance: 0 Next hop: --------------------------------------- Route type: t0s Network: 10.50.192.12/32 Admin distance: 1 Next hop: 100.64.0.1 --------------------------------------- Route type: t1s Network: 10.50.192.5/32 Admin distance: 3 Next hop: 100.64.0.3 --------------------------------------- Route type: t0c Network: 10.50.168.0/21 Admin distance: 0 Next hop: --------------------------------------- Route type: t0s Network: 172.31.0.0/16 Admin distance: 1 Next hop: 10.50.168.1 --------------------------------------- Route type: t0c Network: 10.50.192.13/32 Admin distance: 0 Next hop: --------------------------------------- Route type: t0c Network: 100.64.0.2/31 Admin distance: 0 Next hop: --------------------------------------- Route type: t0c Network: 169.254.12.0/24 Admin distance: 0 Next hop: --------------------------------------- Route type: t0c Network: 10.50.160.0/21 Admin distance: 0 Next hop:
Routing Outside of the SDDC
Most of my customers are leveraging the VMware Transit Connect service to connect their SDDCs to AWS services, their on premises data centers, and/or the internet. Transit Connect (VTGW) is a VMware managed AWS Transit Gateway. Typically when segments are created in the SDDC, the routes are automatically added to the VTGW route tables.
After building out the environment above, I went ahead and created a VTGW to validate the route table. Upon inspection:
Only the infrastructure segment and the segments connected to the default compute gateway is advertised to the VTGW. In my next post, I will work to get the other routes into the VTGW so the rest of my enviroment can access the resources behind the routed and NATed gateway.