The topic of this post is Layer 3 VPN (L3VPN or VPRN as we call it in SROS) configuration, and I decided to kill two birds with one stone by inviting Juniper vMX to our cozy SROS environment.
The BGP/MPLS VPN (RFC 4364) configuration will undergo the following milestones:
- PE-PE relationship configuration with VPN IPv4 address family introduction
- PE-CE routing configuration with both BGP and OSPF as routing protocols
- Export policy configuration for advertising VPN routes on PE routers
- AS override configuration
- and many more
We’ll wrap it up with the Control Plane/Data Plane evaluation diagrams which help a lot with understanding the whole BGP VPN mechanics. Take your seats, and buckle up!
The topology I use throughout this tutorial consists of two customers (namely Alcatel and Juniper) which have two remote sites and want to get connectivity between them by means of an L3VPN service:
We start off with ISIS (any IGP would suffice) running smoothly in our provider core so every router can reach every other’s loopback address. Another prerequisite is to have an MPLS enabled core, since L3VPN uses MPLS encapsulation for dataplane communication. I configured RSVP-TE tunnels between PE routers for this tutorial in the way that PE1_ALU can resolve PE2_JUN (and vice versa) loopback address via RSVP-TE tunnel.
Lets have a look at the relevant configuration blocks on the three routers PE1_ALU, P1_ALU and PE2_JUN
Before we dive deep into the BGP L3VPN configuration it is necessary to refresh on some basic theory. To get a deeper and broader knowledge on the following topic please consider Juniper’s JUNOS MPLS and VPNs student guide and Alcatel-Lucent’s Service Routing Architect guide.
In order to maintain different customer’s routes independently PE routers use separate logical routing tables called Virtual Routing and Forwarding (VRF).
Each PE router maintains a number of separate forwarding tables. One of the forwarding tables is the “default forwarding table”. The others are “VPN Routing and Forwarding tables”, or “VRFs”.
3.1. VRFs and Attachment Circuits
Every PE/CE attachment circuit is associated, by configuration, with one or more VRFs. An attachment circuit that is associated with a VRF is known as a “VRF attachment circuit”.
In the simplest case and most typical case, a PE/CE attachment circuit is associated with exactly one VRF. When an IP packet is received over a particular attachment circuit, its destination IP address is looked up in the associated VRF. The result of that lookup determines how to route the packet.
Provider Edge routers must have a VRF configured for each connected site. VRFs are totally separated in routers control plane by default, so we can depict VRFs as the routers on their own caged in a single hardware unit:
VRFs also remain local to the corresponding hosting PE routers and their number representation or names are never propagated to the other PEs. In our example we have four VRFs in total, two VRFs (VRF Alcatel and VRF Juniper) per PE.
Since one router can have many routing instances (VRFs) inside, it is necessary to help a router to distinct between the different routes in the different VRFs. It is highly likely that customers connected to a single PE will have overlapping IP addresses and this will potentially lead to troubles as the router won’t know which customer a route belongs to.
I emulated this situation to help you better understand the problem; see, Juniper’s loopback address for the emulated customers CE1/CE2 overlaps with Alcatel’s customer loopback addresses. How will a PE router PE1_ALU distinct between these routes?
Route Distinguisher (RD) comes to the rescue.
RFC 4364. Route Distinguisher definition
An RD is simply a number, and it does not contain any inherent information; it does not identify the origin of the route or the set of VPNs to which the route is to be distributed. The purpose of the RD is solely to allow one to create distinct routes to a common IPv4 address prefix.
RD can be written in several forms, but it is handy to use the IP address in the Administrator subfield and VPN number in the Assigned number subfield:
A combination of a Route Distinguisher and an IPv4 route effectively produces what is called the VPN-IPv4 route. VPN-IPv4 routes are 12 byte length (8b RD + 4b IPv4) addresses exchanged by MP-BGP speakers. PE routers compose VPN-IPv4 addresses and allocate MPLS labels for the routes before sending them to the MP-BGP neighbors. Consider the picture above to get a visual representation of an VPN-IPv4 route.
So Route Distinguishers make every VPN-IPv4 route unique in a providers core, but we still need a mechanism to tell what VRF a single VPN-IPv4 route belongs to? We need a way to extend the VPN-IPv4 route with the information about which routing instance this route should be put into.
Every VRF is associated with one or more Route Target (RT) attributes. When a VPN-IPv4 route is created (from an IPv4 route that the PE has learned from a CE) by a PE router, it is associated with one or more Route Target attributes. These are carried in BGP as attributes of the route.
Any route associated with Route Target T must be distributed to every PE router that has a VRF associated with Route Target T. When such route is received by a PE router, it is eligible to be installed in those of the PE’s VRFs that are associated with Route Target T. (Whether it actually gets installed depends upon the outcome of the BGP decision process, and upon the outcome of the decision process of the IGP (i.e., the intra-domain routing protocol) running on the PE/CE interface.)
A Route Target attribute can be thought of as identifying a set of sites. (Though it would be more precise to think of it as identifying a set of VRFs.) Associating a particular Route Target attribute with a route allows that route to be placed in the VRFs that are used for routing traffic that is received from the corresponding sites.
There is a set of Route Targets that a PE router attaches to a route received from site S; these may be called the “Export Targets”. And there is a set of Route Targets that a PE router uses to determine whether a route received from another PE router could be placed in the VRF associated with site S; these may be called the “Import Targets”. The two sets are distinct, and need not be the same. Note that a particular VPN-IPv4 route is only eligible for installation in a particular VRF if there is some Route Target that is both one of the route’s Route Targets and one of the VRF’s Import Targets.
Usually the RTs are represented as
<AS Number of a client network>:<VRF ID>:
First thing to accomplish in the L3VPN configuration is the BGP peering inside the provider’s core network. We have two Provider Edge routers (PE) and one core provider (P) router in our simple network. Our business goal is to provide the L3VPN service to our beloved JUN and ALU customers. To do so, we need to configure BGP peering between all the PE routers involved in the L3VPN service setup, these two routers are PE1_ALU and PE2_JUN.
The BGP configuration part for PE1_ALU and PE2_JUN routers follows a simple iBGP configuration routine (check BGP configuration tutorial to grab the basics), the only part which is different is a need of a new BGP address family. We need to enable this address family to deal with the VPN routes, which are different from the IPv4 routes.
In Juniper this family is called
inet-vpn, in SROS it is
vpn-ipv4, but nonetheless it is just an address family which enables communication of VPN routes between the peers. We will see later how this family differs from a classic IPv4, but for now just look at the BGP configuration part for both PE routers:
As you see, the only part which is related to L3VPN is this new VPN address family.
Support for the additional address families transforms a classical BGP to a fancy Multi-Protocol BGP (RFC 4760). Lets see how this family is communicated in the BGP messages:
Both routers announces the capability to exchange VPN Unicast IPv4 routes in the
BGP OPEN messages. If a BGP peer sees this capability in an incoming OPEN message, it assumes that the neighbor speaks VPN IPv4 routes.
So far we have configured PE-PE relationship which is a foundation for a working L3VPN service. Our next step is a VRF configuration which can be seen as a customers facing dedicated routers inside a singel PE router hardware unit. We will start with PE1_ALU and configure VRFs 20 and 30.
At first we should ensure that customer facing ports operate in
SROS uses the concept of the
customers which is similar to the tenants in a virtualization world. I will create two new customers (
Customer 1 is a default one) to map them to the customers we have in our network:
After that I create a VPRN service (which is a fancy SROS name for a L3VPN) for each customer:
VRF 30 configuration repeats the same steps:
Ok, our VRFs 20 and 30 are configured on PE1_ALU router and we have customers interfaces attached. What we need to do next is to configure a routing protocol which will propagate customers routes to the PE router. On PE1_ALU router we will use BGP as a routing protocol towards the CE routers, consequently CE routers will use BGP as well. Lets configure BGP instances for VRFs 20 and 30:
BGP configuration for VRF 20:
as-override command under the BGP section is used to resolve the issue with AS-PATH loop prevention mechanism.
When a BGP UPDATE message goes from CE1_JUN over eBGP to PE1_ALU it has AS-PATH value of
200. Then this UPDATE message traverses Service Provider’s network and as it goes over eBGP session to CE2_JUN its AS-PATH value becomes
"100 200". But CE2_JUN is a part of AS
200 itself, so it will silently discard a route update with AS-PATH value containing its AS number (AS PATH loop prevention mechanism makes it so).
as-override command placed under the BGP context of the receiving VRF on a PE router replaces the customers AS number with Service Providers own AS number, so AS-PATH string of
"100 200" will become
"100 100" and will be accepted by the CE router residing in AS 200 since no loop will be detected.
Note, that it is mandatory to create an export policy on SROS PE routes for incoming BGP-VPN routes to leave the VRF over the PE -> CE routing protocol to the CE router:
Now add this policy under the BGP context of the VRF:
Super, now repeat the steps for
VRF 30. The complete service configuration part on PE1_ALU should look as follows:
Juniper JUNOS does not use concept of network/access ports, thats why you deal with CE-facing interfaces just like you do with the normal ones:
Now the VRF part; VRF is called a routing-instance in JUNOS.
VRF configuration on Juniper looks almost identical to Nokia. The major difference here is that you don’t have to tell JUNOS to resolve VRF’s next-hop address via MPLS tunnel. And you don’t have to configure an export policy in case you are using eBGP as a PE-CE protocol. Juniper defaults to that behavior.
Note, that with Juniper we omit the explicit AS number configuration under the BGP configuration. In that case the globally configured AS number will be used.
Configuration portion for VRF 20 will look as follows:
So far we’ve played with BGP as a PE-CE routing protocol, but frankly speaking OSPF is not a stranger for this task as well. Lets see how to configure the OSPF adjacency between the Juniper PE and a CE2_ALU router.
A key piece in this configuration block is the export policy which lets vpn-ipv4 routes imported into VRF 30 to be exported to CE2_ALU over OSPF.
Configure an export policy first:
Then configure PE-CE OSPF protocol with the export policy applied:
Complete VRF configuration for PE2_JUN goes like this:
Now it is time to connect our customers to the service provider’s network via VRFs created earlier and finally add some VPN routes. CE routers completely unaware of a complex L3VPN configuration on the PE routers, what they need to do is just setup a routing protocol over which customers routes could be delivered to (and received from) the Service Provider.
Starting with Juniper CE1_JUN and CE2_JUN that run eBGP with PE routers:
Now its Nokia time. Pay attention to the CE2_ALU router, since we are using OSPF on CE2-PE2 link configuration it is a little bit different from other CE’s configs.
We are done with the configuration, all in all it was not a complex task, what is more important is to understand what’s going on with control and data planes. I believe you will like this step-by-step walkthrough via every node in the network.
I will start with dissection of a control plane operation from the point where MP-BGP session between PE routers has been already established and we are enabling VRFs on customers routers. Refer to this overview chart and see how an information about CE1_JUN’s loopback interface propagates through the entire network to CE2_JUN counterpart:
All the pictures are clickable, to see the full sized pics choose “open an image in a separate tab” option in your browser.
CE1_JUN router has an export policy
export_loopback configured which is used by BGP to construct the BGP UPDATE message with
lo0 prefix as an NLRI.
CE1_JUN sends a regular BGP UPDATE message to its eBGP peer PE1_ALU.
PE1_ALU router receives this update via its interface
toCE1 configured in
vprn 20 context. PE1_ALU populates its VRF 20 with a route to
PE1_ALU router has an established MP-iBGP session with PE2_JUN so it takes a BGP route from VRF 20 and automatically sends an MP-BGP UPDATE message to its peer. Note, that ALU routers will send MP-BGP update automatically only for the connected to VRF routes and the routes received via BGP. If we had OSPF between CE1 and PE1, we would need to configure an export policy to propagate this update over MP-BGP session.
Since PE1_ALU router wants to send an update for a route in the VRF it should construct an MP-BGP Update message which has a specific Path attribute - MP_REACH_NLRI - to communicate this routing information. And PE1_ALU will transform the
220.127.116.11/32 IPv4 prefix to an VPN-IPv4 one.
Take a closer look at this BGP message. See how PE1_ALU router added some valuable additional information to correctly pass CE1_ALU’s loopback address via MP-BGP. First of all examine how NLRI has been transformed in MP-BGP: it now has a Route Distinguisher which we configured for VRF 20 earlier, it has the IPv4 prefix itself and it has the MPLS label
PE1_ALU router allocated a VPN label which it associated with the VRF 20. This label tells PE1_ALU router that if it ever receives a data packet with this label it should associate the data encapsulated within it with VRF 20! This way ingress PE routers tell others PEs what label should be used as a VPN label for the routes residing in a particular VRF.
There are two methods of allocating the VPN labels (they are also called Service labels):
- per VRF: all routes originated from a single VRF will have the same VPN label. SROS routers default to this.
- per VRF per next-hop: If a VRF has >1 CE interfaces, PE router will allocate different labels for different CE interfaces inside one VRF. Juniper routers default to this.
If we zoom over the Extended Community attribute of the BGP UPDATE message, we can spot the Route Target
200:20 value there.
Important things happened to the Next-Hop, not only it looks now like a VPN-IPv4 route with a Route Distinguisher value of
0:0 and without MPLS label, but Next-Hop IPv4 address has been changed to PE1_ALU’s system (loopback) interface
10.10.10.1. This is how PE1 router tells PE2 that it can reach VRF 20 routes via PE1.
In the end of the day, PE1_ALU’s update reaches PE2_JUN since it has the IP destination address of 10.10.10.3.
Notice, that BGP updates traverse Service Provider’s network in a form of the simple IP packets, MPLS is out of the picture at this moment. Service Provider’s core router - P1_ALU - simply routes IP packets and has no take in BGP at all.
PE2_JUN receives the BGP UPDATE with VPN-IPv4 route. Once this route passes validation checks (Nexhop resolvable, no AS Path loop) PE2 submits this route to a specific table named
bgp.l3vpn.0. This table stores all BGP VPN routes, refer to this figure to examine some of its content:
PE2 extracts the routing information from this update an based on the Route Target value installs the IPv4 route
18.104.22.168/32 into the VRF 20 table -
20.inet.0. PE2 resolves the next-hop address of the fellow PE1_ALU (10.10.10.1) via MPLS Label Switched Path (LSP) and stores this information in the
Remember that it is mandatory to have an active LSP to the remote PE, since we have to have an MPLS transport to the remote end to carry the data packets.
Since we installed the route for the
22.214.171.124/32 IPv4 prefix into VRF 20 and we have an active eBGP peer in VRF 20, we should send an update for this IPv4 prefix to the CE2_JUN router to let the CE2 site to be aware of the remote prefix. This update goes as an ordinary eBGP update.
CE2_JUN receives the BGP UPDATE and installs a route into the only table it has for IPv4 routes -
This completes Control Plane operation regarding the prefix
126.96.36.199/32, same process goes for the other loopbacks and connected to VRFs link addresses for both Alcatel and Juniper customers.
To complete this post we should examine the data plane operations. We will see how data packets destined to
188.8.131.52 propagate through the network using the labels allocated during the control plane operations.
CE2_JUN wants to send a data packet to CE1_JUN via L3VPN service provided by our network. CE2 has an active route in its route table
inet.0 that says that it can reach
10.20.99.2 address via the
ge-0/0/0 interface. CE2 has a MAC address for
10.20.99.2 so it constructs the whole frame and puts it on the wire.
PE2_JUN receives the Ethernet frame on its interface
ge-0/0/1 which belongs to VRF 20, that is how PE2 decides to associate this packet with VRF 20. PE2 consults with the VRF 20 routing table and sees that it has to use the LSP
toPE1 to send the incoming data packet further.
Then PE2 gets MPLS label which it received earlier from its RSVP neighbor P1_ALU during the LSP signalization process.
But this was just a transport MPLS label, it helps PE2_JUN to reach PE1_ALU, but PE2 needs one label more - the VPN Label - to tell PE1_ALU to which VRF this data belongs. This label was signalled earlier (see Control Plane operation section) via MP-BGP.
Now PE2 has everything it needs:
- MPLS VPN label to encapsulate the data packets from its VRF 20 destined to the VRF 20 on PE1_ALU
- Transport MPLS Label to get to PE1_ALU via MPLS core
and thus it constructs a packet with two labels stacked and fires it off.
P1_ALU is totally unaware of the whole services and customers mess, it just switches MPLS packets by replacing the incoming transport label with the outgoing one.
PE1_ALU receives an MPLS packet from P1_ALU. It pops out the transport label (fig. 4.1) and examines the enclosed MPLS label. This label value
131068 was signalled by PE1_ALU via MP-BGP during the Control Plane operation. So PE1 knows that it has to pop this label and associate the enclosed packet with the VPRN 20 (VRF 20) (fig. 4.2)
VRF’s 20 routing table says that packets destined to
184.108.40.206 should be forwarded to
10.20.99.3 address (fig. 4.3), which is a connected network leading to CE1_JUN (fig. 4.4). PE1_ALU constructs the packet and moves it via Ethernet out of the 1/1/2 port (fig. 4.5).
CE2_JUN receives an ordinary IP packet with a destination address matching its interface. It decapsulates ICMP echo request and sends back the echo reply.
This concludes the control and data plane operations walk through. If you followed along the explanations and practiced the configuration steps, you should be in a good shape to implement the basic L3VPN services and also should have a pretty solid understanding of the service establishment mechanics.