Route and traffic engineering in Azure

Everything in Azure networking is easy until you connect something to it. It's the joke I open with every time I run a training session on how Azure networking actually behaves behind the scenes, and routing is no exception. I've already written about routing in Azure with UDRs and the risk of building asymmetric paths (Asymmetrical routing in Azure network). This post goes one level deeper: modifying the BGP advertisements themselves. That's traffic engineering. It's routine practice in on-premises networking and a new one in Azure. Two features carry most of the weight today: advertised gateway prefixes on a hub VNET, and Route-maps in Virtual WAN and Azure Route Server (in public preview). Advertised gateway prefixes Azure virtual networks expose a property called summarizedGatewayPrefixes (public preview at the time of writing). It lets you hand the gateway a list of CIDR blocks to advertise instead of its default advertisement. The default behaviour is straightforward: an ExpressRoute Gateway or VPN Gateway advertises every address space of its own VNet, plus every address space of each peered VNet using gateway transit. In a hub-and-spoke topology, that means every spoke prefixes are advertised to the gateway. So instead of advertising 10.0.0.0/24 , 10.0.1.0/24 and 10.0.2.0/24 separately, you configure 10.0.0.0/22 on the hub VNet and the gateway advertises that single prefix. For each peered spoke, Azure checks whether the spoke's address space is covered by a summary - if it is, the spoke prefix is suppressed. If it isn't, it keeps being advertised individually. ExpressRoute private peering caps the number of prefixes that can be advertised on a single connection to 1,000 IPv4 and 100 IPv6. Cross the ceiling and the session drops. That is not a graceful degradation; it's an outage. In a Hub VNet with a few hundred spokes and multiple address spaces per spoke, you get there faster than you'd like. - The property only takes effect on a VNet that has a GatewaySubnet and a deployed gateway. You can set it earlier; it just does nothing. - It applies to both VPN Gateway and ExpressRoute Gateway. Route Maps This used to be the only BGP manipulation available outside Virtual WAN. That changed: Route Maps for Azure Route Server entered public preview recently. Route-maps is a policy engine on the virtual hub. It inspects, modifies and filters the routing information exchanged between the hub and a connection, in the inbound direction, the outbound direction, or both. It applies to ExpressRoute connections, site-to-site VPN connections, point-to-site connections, and VNet connections. One route-map per direction, per connection. Worth knowing up front: you cannot apply a route-map to the connection between on-premises and an SD-WAN or firewall NVA deployed in the hub, other connections in that hub still work normally. A route-map is an ordered list of rules. Each rule has match conditions and actions, plus a Next step setting that decides whether a matched route continues to the following rules or terminates there. Match conditions - Route-prefix , AS-Path , Community , each with an equals or contains criterion: | Property | Criterion | Meaning | |---|---|---| | Route-prefix | equals | Matches those exact routes only | | Route-prefix | contains | Matches those routes and everything more specific underneath | | AS-Path | equals | AS-PATH must contain the listed ASNs in that order | | AS-Path | contains | AS-PATH contains one or more of the listed ASNs, order irrelevant | | Community | equals | Route must carry all the listed communities | | Community | contains | Route carries one or more of them | Multiple match conditions in a rule are evaluated as a logical AND, all of them have to be satisfied. A rule with no match condition matches every route on the connection. If no rule matches, the default is allow, not deny. Actions - drop, or modify: | Property | Action | Effect | |---|---|---| | Route-prefix | Drop | Matched routes are filtered out of the advertisement | | Route-prefix | Replace | Matched routes replaced by the prefix(es) in the rule (this is summarization) | | AS-Path | Add | Prepend the listed ASNs | | AS-Path | Replace | Set AS-PATH to the listed ASNs - with no value, clear it | | Community | Add / Replace / Remove | Add, overwrite, or remove specific communities | A single rule can carry several modifications; the order in which they're written doesn't matter. Use cases Summarization Same problem as Part 1, this time on a Virtual WAN hub. You're heading toward the ExpressRoute advertised-prefix ceiling and you need to cut the count. - Match: Route-prefix, contains ,10.2.0.0/16 - Action: Route-prefix, Replace ,10.2.0.0/16 - Applied: Outbound, on the ExpressRoute connection Instead of a pile of /24s, the circuit sees one /16. Two caveats. Summarization strips the BGP community and AS-PATH from the resulting route - inbound and outbound alike. And Route-maps only summarizes; you cannot use it to generate more specific routes. Filtering - Match: Route-prefix, contains ,10.2.0.0/24 - Action: Drop The prefix is no longer advertised to that peer. This is not a firewall. You've removed a route, not blocked traffic. If the peer has another path to that destination, a default route, a summary from elsewhere, a User Defined Route, the packets still flow. Filtering is a routing tool. Use Azure Firewall or NSGs for security. BGP communities A community is a tag attached to a route so that the receiving side can make policy decisions on it. Virtual WAN preserves communities generated by BGP peers, within a hub and across inter-hub connections. The natural use is classification. Say you tag by workload tier: | Prefix | Community | |---|---| 10.0.0.0/21 | 65001:100 database services | 10.0.8.0/21 | 65001:200 production applications | 10.0.16.0/21 | 65001:300 test | 10.0.24.0/21 | 65001:400 dev | - Match: Route-prefix, contains ,10.0.0.0/21 - Action: Community, Add ,65001:100 The connected network can then apply local preference, filtering or QoS policy on the tag rather than maintaining a prefix list that goes stale every time someone deploys a spoke. Do not remove the Azure-reserved communities in the 65515:* , 65517:* and 65518:* ranges. They are used internally and stripping them breaks routing in a way that is unpleasant to debug. AS-PATH manipulation This is the part where most designs go wrong, so it's worth being precise. The scenario A Virtual WAN with two hubs: Hub-Paris (France Central) and Hub-London (UK South). A branch office runs a site-to-site VPN with BGP to both hubs, so it learns Azure routes from both. An application VNet attached to Hub-Paris advertises 10.2.100.0/24. Every Virtual WAN hub and gateway uses ASN 65515, and that is not configurable. Inter-hub propagation doesn't add an ASN either. So the branch receives 10.2.100.0/24 from two neighbours with an identical AS-PATH. Equal length, equal everything, you have no say in which hub it picks. Goal: the branch should prefer Paris and keep London as a warm backup. The fix Apply the route-map outbound, on the branch's VPN connection in Hub-London: - Match: Route-prefix, equals ,10.2.100.0/24 - Action: AS-Path, Add , your ASN (repeat it two or three times for margin) - Direction: Outbound - Connection: the site-to-site VPN connection to the branch, in Hub-London only The branch now receives the prefix with a one-ASN path from Paris and a longer path from London. Paris wins on AS-PATH length; London stays in the RIB as a backup and takes over if Paris drops. Why not inbound at Hub-Paris Because inbound route-maps are applied before the route enters the hub's defaultRouteTable - and that table feeds every propagating connection, including the inter-hub link to London. Prepend inbound at Paris and both hubs advertise the prepended route to the branch. You've made the prefix uniformly less attractive and changed nothing about which hub gets chosen. Inbound is the wrong direction for this problem. It's the right direction for a different one. Choosing an ASN - 2-byte ASNs only. If your network runs 4-byte ASNs, you can't use them in a route-map action. - Don't use reserved numbers: public 8074 ,8075 ,12076 ; private65515 ,65517 ,65518 ,65519 ,65520 . - For Express Route only Don't use private ASNs for prepending. the Microsoft Enterprise Edge router that connect to your network locally removes private ASNs from the AS-PATH, so your prepend never reaches the your routers and you spend an afternoon wondering why nothing changed. This stripping doesn't apply to a VPN-connected branch. - Never prepend the receiving router's own ASN. It will reject the route as a loop. Same reason you can't prepend 12076 over ExpressRoute. - 64496 -64511 are RFC 5398 documentation ASNs and are explicitly not permitted in the path over ExpressRoute. The distinction that matters: - Inbound route-maps can influence Azure's own best-path selection. The modified AS-PATH is what lands in defaultRouteTable , and with hub routing preference set to AS Path, the hub compares those lengths. This is how you make Azure prefer ExpressRoute over VPN, or one hub over another, for return traffic. - Outbound route-maps cannot. Best-path selection happens before outbound route-maps run. Outbound only changes what the peer sees. things to remind One line to remember: inbound influences Azure's choice; outbound influences your peer's choice. Pick the direction that matches whose decision you're trying to change. - Route-maps aren't applied to the virtual hub address space. - A prefix can be modified by Route-maps or by NAT, not both. - Your spoke VNet connections must propagate to the Default route table for their prefixes to reach on-premises at all. - Set Terminate on the last rule, or you'll spend time debugging a rule you thought had stopped processing. - Use the Route Map dashboard to verify. It shows the route before and after the map, with AS-PATH and communities - which turns "I thi