Das Blinken Lichten

Understanding BGP – Establishing peering

Jon Langemak — Thu, 07 Nov 2024 04:13:37 +0000

After some reflection I’ve realized that while I’ve spent a lot of time talking about BGP in it’s many forms I haven’t really ever done a deep dive on it. To be clear – Im not aiming to talk about how to configure BGP , or how path selection works, or even how to troubleshoot BGP. What I want to examine is what BGP is doing on the wire. How it communicates with peers, when it sends updates, and what kind of things are in the updates. Im hoping to write several blogs starting with the basics and then diving deeper as we go. That said, let’s get started!

To start things out with – I think it makes sense to start with a simple lab consisting of two BGP nodes that are peering together. Something like this…

I don’t want to spend a lot of time focusing on the configuration syntax and basic configuration parameters so let’s just run BIRD on both of the nodes so we can get off the ground with minimal effort. Let’s assume that both BGP Peers shown above are just normal Ubuntu VMs and both have a single interface on common 169.254.10.0/31 network.

Note: All of the links will come out of the /24 and be /31 IPv4 networks.

To install BIRD we’ll simply….

apt install bird2

Once that completes we should be up and running with a default BIRD configuration. You can validate this by simply typing birdc show proto…

root@vm0:~# birdc show proto
BIRD 2.0.7 ready.
Name       Proto      Table      State  Since         Info
device1    Device     ---        up     18:12:43.178  
direct1    Direct     ---        down   18:12:43.178  
kernel1    Kernel     master4    up     18:12:43.178  
kernel2    Kernel     master6    up     18:12:43.178  
static1    Static     master4    up     18:12:43.178  
root@vm0:~#

Perfect! Now the default configuration that comes with BIRD is meant to be more of an example of what’s possible rather than a meaningful configuration. That said – let’s add our own basic configuration to the BIRD-0 host. You can simply overwrite the configuration in /etc/bird/bird.conf with this…

log syslog all;

router id 169.254.10.0;

debug protocols {states,interfaces,events};

protocol bgp bird_1_iBGP_peering {
    local as 65000;
    neighbor 169.254.10.1 as 65000;
    ipv4 {
        import all;
        export all;
    };
}

And then restart the bird service (systemctl restart bird). Now let’s do something similar on the BIRD-1 host…

log syslog all;

router id 169.254.10.1;

debug protocols {states,interfaces,events};

protocol bgp bird_0_iBGP_peering {
    local as 65000;
    neighbor 169.254.10.0 as 65000;
    ipv4 {
        import all;
        export all;
    };
}

After you have this configuration in place on both nodes, and have restarted the BIRD service, you should be able to run the show proto command again and see something like this…

root@vm0:~# birdc show proto
BIRD 2.0.7 ready.
Name       Proto      Table      State  Since         Info
bird_1_iBGP_peering BGP        ---        up     18:53:51.344  Established   
root@vm0:~#

Wohoo! That might be the quickest Zero to BGP peering we’ve ever done here! Now at this point – all we have is a BGP session. But that by itself required a lot of communication to happen. I’d like to spend the rest of this post talking about what actually happened for both of these BGP peers to actually decide it was OK for them to form a peering relationship. So let’s kill the BIRD service on the BIRD-0 host, start a packet capture on the wire, and then restart the service so we can see the communication that happens when the hosts try to BGP peer…

Now this is hard to see but above is the total output of our BGP session startup. So let’s walk through these packets one at a time to make sure we know what’s going on…

Packet 1 – Packet one shows the host BIRD-1 (169.254.10.1) attempting to setup a TCP session to the host BIRD-0 (169.254.10.0) by sending a SYN packet. This makes sense as we only stopped the BGP service on BIRD-0 so BIRD-1 will happily try to kick things off.

Packet 2 – The host BIRD-0 sends a reset to BIRD-1. It does this because it’s not actually listening on the TCP 179 for BGP yet as the service is still down.

Packet 3 – We restart the BIRD service on the host BIRD-0 and it immediately attempts to setup a TCP connection to the host BIRD-1 by sending a SYN packet. Let’s use this TCP session setup as a means to remind ourselves how the TCP three way handshake works. You’ll note that the sequence number in this SYN packet is 2120355052 which is a random 32 bit number chosen by the client. This initial sequence value is sometimes referred to as an “Initial Sequence Number” or ISN for short. Also worth noting is that the acknowledgement number in this packet is set to 0 which it should always be in the initial setup of a TCP session.

Note: Keep in mind that Wireshark is trying to be helpful which is why the “Sequence number” field is listed as 0. It’s 0 relative to the first TCP segment seen. To see the real sequence number you should be looking at the “raw” version of this field.

Packet 4 – In packet 4 we see BIRD-1’s response to the SYN sent by BIRD-0. Note that both the SYN and ACK flags are set and that the acknowledgement number is 2120355053 which is the sequence number received in the SYN packet from BIRD-0 (2120355052) plus 1. In addition BIRD-1 has sent it’s own ISN of 147274070 in the reply to BIRD-0.

Packet 5 – In packet 5 we see the final ACK of the TCP session setup. Note that the raw sequence number is the ISN show in packet 3 incremented by 1. Also the acknowledgment number is the ISN sent in packet 4 incremented by 1. In terms of TCP options we see that the ACK flag is also set. While there’s a lot more going on at the TCP level (windowing etc) that we arent covering here we now have a setup TCP session.

Note: It’s worth mentioning here that there are generally 6 phases of BGP session establishment. While checking on the status of a BGP peer you might see either Idle, Connect, Active, Open Sent, Open Confirm, or Established. The first 3 typically refer to the TCP session setup while the last 3 refer to the BGP session setup. I’ll note here that the “Active” state has led to more than one conversation around “Well I don’t know why you’re end is down – my end says it’s active” when turning up BGP peerings.

Packet 6 – Our first BGP protocol packet! In this packet we see host BIRD-0’s attempt to start a BGP session. To do this the host originates a BGP open message. The BGP open message includes a surprising amount of information about the peers BGP configuration. Let’s talk through it line by line…

Marker – This one deserves a paragraph on it’s own which we’ll cover in a later post. For now just know that in a standard BGP open message this field is just filled with all 1s.

Length: The Length of the BGP options which is 53 bytes. In other words the TCP payload of this frame is 53 bytes.

Type: There are defined types of BGP message types. 1 happens to be the BGP Open message.

Version: The BGP version being used. While many folks assume that version 4 implies that this is an IPv4 peering – that’s not actually true. It’s the actual version of the BGP protocol which means the previous version was in fact BGP-3. I would be stretching my memory if I took a guess at when BGP-4 became the defacto standard but Im thinking it was in the early 2000s.

My AS – Just what it sounds like. The Autonomous system this BGP speaker lives in.

Hold time – This describes the hold time for the BGP session. There is often confusion around this with BGP. There’s always a misconception around hold time and keep alive timers and if they need to match. The reality is that they dont need to match and the only thing that’s negotiated is the hold time. The hold time negotiation is really just the two peers picking the lower of the two values in the exchanged open messages. The keepalive is not negotiated and is assumed to be 1/3 of the hold time by default. That said – the keepalive can be manually set on each peer as well although I’d argue you’d have to have a very good reason to manually change the keep-alive timers. But it does mean that you can configure one peer to send keepalives every second and let the other use the default keepalive (which in BIRD is a hold time of 240 seconds yielding a keepalive of 80 seconds). The summary here being that hold times are negotiated and keepalives are assumed based on the hold time by default. They can be overriden in the peer configuration so each peer sends them at different rates.

BGP Identifier – This is the peers router-ID. There is also general confusion around this. In most cases – a router-ID is configured to align with the peering IP address. This makes things easy to comprehend. The reality though is that a router-ID can be pretty much anything you want. Some operating systems mandate that the router-ID is one of the defined IP addresses on the routing system. They even go so far as to have logic that dynamically picks a router-ID based on the available IPs. That should be your first hint that there isn’t a strong binding between the peering relationship and the router-ID. They are only mandating that you have it configured as “a” IP address on the system. That could be anything! Now you might be wondering – what happens if the IPs overlap between peers? Turns out so long as it’s an eBGP peering that works! The only rule is that a router-ID can not overlap inside of a given AS. At the end of the day – the best approach is to align the router-ID with something that’s significant. For iBGP peerings likely the source transport loopback – for eBGP the peering interface IP address in the case of a single peer.

Optional Parameters – These define the optional parameters which are typically a list of capabilities the peer can support. There can be other things than capabilities here – but the most common type of optional parameter is type 2 (capabilities). We won’t dive into each capability listed in this open (there’s a complete reference here) – but it’s important to know that peers need to align on the available capabilities. That is – if one peer supports something the other peer doesnt – that capability wont work across the peering. The capability negotiation (I use that term very very loosely here) is actually sort of a mess. Since the exchange of capabilities only occurs as part of the BGP open some BGP implementations will forcibly restart a session if you change your local capabilities. Others might expect you to know that you have to restart the session and will happily keep running without the new capability until you manually restart the session. Ivan Pepelnjak wrote an interesting post about this awhile back and mentioned the draft RFC for capability exchange which appears to have still gone nowhere. Since we’re using BIRD I’ll note that if you change the capabilities and ask BIRD to refresh the configuration (birdc configure) the host will actually send a BGP notification message (type 3) major code 6 minor code 6. Which if we reference our type codes from the links you’ll see it’s a BGP notification message for Cease because of configuration change. Put another way – BIRD will forcibly restart the session if the peer capabilities change. That said if there’s not consensus on a capability it’s just not used.

Packet 7 – Packet 7 is simply host BIRD-1 ACKing the BGP open packet it got from BIRD-0. If you’re still playing the TCP game with me you’ll notice that the ACK number is the length of the open packet TCP segment (53) plus the initial 1 from the TCP session setup.

Packet 8 – Now it’s host BIRD-1’s turn to send it’s BGP open message to BIRD-0. These packets should look shockingly similar as they share almost identical configuration.

Packet 9 – Host BIRD-0 ACKing the open from BIRD-1. Again you can follow the TCP seq/ack numbers if you’re interested. A fun exercise if you’re so inclined on these smaller more manageable captures.

Packet 10 – BIRD-1 sends BIRD-0 a keepalive message. While keep alive messages have a distinct use case in terms of keeping a BGP session alive this one is actually super important. It signals to BIRD-0 that BIRD-1 has agreed to the BGP session (agreed to the terms in the open message) and acts as a sort of final ACK that we’re good to establigh the BGP peering. After this initial keepalive future keepalive messages are used to keep the session open.

Packet 11 – BIRD-0 sends BIRD-1 the keepalive ACK

Packet 12 – BIRD-0 ACKing the keepalive from BIRD-1

Packet 13 – BIRD-1 ACKing the keepalive from BIRD-0. The BGP session is now fully established!

Packet 14 – The next thing we see right after the session comes up is what I would describe as an empty update packet. We can see it’s a type 2 BGP message which is an update. But it looks sort of desperately empty. If you’re thinking this makes sense because we aren’t advertising any routes yet you’d be mostly correct. But the reality is this is a very special kind of BGP update message that’s referred to as an “End of RIB marker”. While initially defined for use as part of the BGP graceful restart capability (we’ll talk about that in an upcoming post because it is some truly interesting signaling that takes place) it was proposed to be used as part of general BGP convergence and used in some other capabilities such route target constraints (another interesting use case we’ll eventually get to). In either case – you might have noticed that one of the default capabilities our peers have is – you guessed it – graceful restart! Since we negotiated that capability we use it which means that following the RFC which says….

The Receiving Speaker MUST send the End-of-RIB marker once it
   completes the initial update for an address family (including the
   case that it has no routes to send) to the peer.

So in our case we see in both packet 14 and 16 that each BGP speaker sends this End of RIB marker to other peer to indicate that it has no more updates to send. But what makes this an End of RIB marker? Specifically the fact that it’s an update that has 0 withdrawn paths and 0 path attribute length.

Packet 15 – ACKing the End of RIB marker from BIRD-0

Packet 16 – End of RIB marker from BIRD-1

Packet 17 – ACKing the End of RIB marker from BIRD-1

So there you have it – a walk through of basic BGP session establishment. In the next post we’re going to cover some other use cases around this basic BGP Peering before we move into the exciting world of actually exchanging routing information!

Packet Actions – Python and Scapy

Jon Langemak — Tue, 01 Jun 2021 12:58:28 +0000

Hello and welcome to the “Packet Actions” series of blog posts. I’d like to spend a few posts talking through how you can programmatically integrate with a network dataplane. I had thrown around the idea of calling this series “Doing things with packets” but that seemed a bit long and also could mean just about anything. So what does Packet Actions mean? Well – its the shortest way I could come up with to say “Looking at packets on the wire and doing things based on what you see in the packet”. To discuss this further I’d like to talk about the often made analogy of network engineers being plumbers – an analogy that makes fairly good sense in most cases. For instance, network engineers create the paths for data to flow – plumbers make paths for water to flow. Additionally both need to make sure that there are no blockages or issues with handling the amount of data or water that needs to flow through the pipes. Going a step further – plumbers might use a diagnostic tool like a scope to physically look inside the pipes if theres a blockage or issue so they can see what’s going on. One could quite easily translate this to a network engineer using a network diagnostic tool such as TCPDump to inspect the flow of bits to see if we can spot an issue. So all in all – the analogy makes pretty good sense.

In this series of posts – I’d like to take things one further step and not only look at the packets – but also take actions based on them. Im sure at this point someone will comment “We already do this!”. Certainly what Im referring to is heading in the direction of network applications and that is a space that has seen an explosion in recent years. But let’s back up a second – a network engineer certainly already does do this – after all we manage network applications like routing protocols, link management protocols, traffic monitoring tooling, and the list goes on. We even have applications that look deeply into the packets and can do a myriad of things for us based on that information. The problem is that this has largely been a feature of vendor provided products and tooling. There’s nothing wrong with that – I have been happily watching these tools and products mature over the last few years. But at the same time – there has been an explosion of functionality that is very (very, very, very) accessible to you as well. What was once something that lived in the realm of vendors who did magical things in secret ASICs (or at least that’s what we were told) is now largely there for you to do yourself. And that’s the point of this series – to show you how you can do some interesting things based on real packets on a real wire. In each post in this series we’ll use different tools to build the same application in the hopes of exploring all of our options for interacting with real packets.

But before we get into the weeds – I’ll take a moment to pause here and say two more things. First – just because you can do something doesn’t mean you should. There are a couple of “packet actions” I will discuss that are really neat – but won’t scale to the point you need them to. Are there ways to make them scale? Sure – but that’s sort of the fun part about all of this – figuring out how things work and how you can apply them to real world problems. The point of these posts is to just show you what’s possible. The application we build will be largely useless but should give you and idea of what’s possible and what level of effort is required to make something that can scale and be resilient. Second – Some of this is going to be totally new to me so you’ll need to bear with me. As we progress through the posts I’ll be learning some of this as we go. This includes new programming languages and new Linux kernel functions that I have only been playing with recently (sometimes because the functionality is so recent too!). As always – if you spot something that’s wrong or could be better just let me know! Feedback is much appreciated.

So now that we got the intro out of the way – let’s talk about what our packet action is going to be. In this series we’re going to be building an application Im going to called “ping_return”. What ping return does is simple. If you ping an IP – it will do what’s required to facilitate that ping to work. Something that looks like this…

That seems like an entirely useless application, which it is, but again the point here is to look at different ways in which we can look at the real packets on the wire and do “something” based on them. So in our case – we want this flow to happen….

Client has a default route pointed at the server address.
Client pings a random IP address which matches the default route
The server receives these ICMP request packets
Our application on the server see these ICMP request packets and adds an IP address matching the destination to a local interface
The client gets a ICMP return and things work!

Should be easy right? So let’s take a look at how to do this. First thing first – let’s look at what our lab is going to look like for this…

Perhaps the easiest lab we’ve ever had, but that’s OK – we want to focus on the packet actions not the lab topology. The two servers we’re going to use here are straight Ubuntu 20 VMs. Base installs – nothing fancy – we’ll work through the dependancies as we come across them. So let’s get right into it!

As the title suggest, we’re going to use Scapy to accomplish our goals in this post. Many of you may be familiar with Scapy, but for those who aren’t, it’s an open source Python library that lets you do all sorts of interesting things with packets. While Im not an everyday user, I suspect that the majority of the use cases for Scapy are more around packet creation. It seems to be a pretty popular tool to use in the security space for this very reason. In any case, it has a rather unique function called sniff which can be used much like TCPDump in order to see real traffic on the wire. But the first thing we need to do is to install it. Since this is a blank fresh install of Ubuntu 20 – the first thing we need to do is to install PIP3 so we can install Python packages…

apt-get update
apt-get install python3-pip

Now that we have PIP installed we can install the scapy module as follows…

pip3 install scapy

Awesome, with the prereqs out of the way we can write a super simple script that shows us how the sniffing capability can work. Create a new file called ping_return.py and put this code in it…

import scapy.all as scapy
ens6_traffic = scapy.sniff(iface="ens6", count=5)
ens6_traffic.nsummary()

Now on your client box – start a ping to the server IP address. Something like ping 10.10.10.1 and just leave it running. Now go back on the server and run the script you just created (python3 ping_return.py1) and wait a few seconds…

root@server:~# python3 ping_return.py 
0000 Ether / IP / ICMP 10.10.10.0 > 10.10.10.1 echo-request 0 / Raw
0001 Ether / IP / ICMP 10.10.10.1 > 10.10.10.0 echo-reply 0 / Raw
0002 Ether / IP / ICMP 10.10.10.0 > 10.10.10.1 echo-request 0 / Raw
0003 Ether / IP / ICMP 10.10.10.1 > 10.10.10.0 echo-reply 0 / Raw
0004 Ether / IP / ICMP 10.10.10.0 > 10.10.10.1 echo-request 0 / Raw
root@server:~#

So that was easy! Our second line of code simply told it to sniff on interface ens6 and that it should only sniff 5 packets. Then the third line of code said to simply print out a summary of those 5 packets. Pretty cool huh? Let’s look at a couple of other ways we can look at this data though before we move on. To do that we’re going to use the prn function of the sniff to call a function for each packet the sniffer receives. So let’s modify our script to be something like this…

import scapy.all as scapy

def process_packet(packet):
    packet.show()

ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet, count=1)

So what’s changed here is that we’ve lowered our packet count to 1 (count=1) and we’ve also added a prn flag which references a function (prn=process_packet).

Note: If you’re like me and you’re wondering what the heck prn stands for there’s an actual StackOverFlow question about it. At least Im not the only one who was wondering…

The prn flag tells the sniffer to call the defined function every time it receives a packet. This may seem sort of counterintuitive considering we’ve also set a the count equal to 1 but all this does is tells Scapy to just take in one packet and then end. So let’s give it a run…

root@server:~# python3 ping_return.py 
###[ Ethernet ]### 
  dst       = 52:ab:54:cd:02:01
  src       = 52:ab:54:cd:01:01
  type      = IPv4
###[ IP ]### 
     version   = 4
     ihl       = 5
     tos       = 0x0
     len       = 84
     id        = 48642
     flags     = DF
     frag      = 0
     ttl       = 64
     proto     = icmp
     chksum    = 0x5492
     src       = 10.10.10.0
     dst       = 10.10.10.1
     \options   \
###[ ICMP ]### 
        type      = echo-request
        code      = 0
        chksum    = 0xdaae
        id        = 0x2
        seq       = 0x613b
        unused    = ''
###[ Raw ]### 
           load      = 'м\\xaf`\x00\x00\x00\x00o#\x0e\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./01234567'

root@server:~#

Very cool! So we got a packet and we printed it out. Looking at the packet we can see it is clearly split into 4 sections. Ethernet, IP, ICMP, and RAW. The relevant fields for all of these sections are also called out. What’s cool about this is that we can actually reference all of these fields in our code! There are two ways to do this. First, we can do it by index. For instance, say that we wanted to print out the destination ethernet address and destination IP address we could change our code to look like this…

import scapy.all as scapy

def process_packet(packet):
    packet.show()
    destination_mac_address = packet[0][0].dst
    destination_ip_address = packet[0][1].dst
    print("Received a packet with a destination MAC address of %s and a destination IP address of %s" % (destination_mac_address,destination_ip_address))

ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet, count=1)

Notice that in addition to showing the packet summary, we are now also defining variables for the destination ethernet and IP address as well. What’s interesting is how we’re pulling those values out of the packet variable. Sort of looks like we’re referring to list indexes or something right? What we’re actually doing here is referencing the header we want to look at. The syntax packet[0][0] means look at the first packet (index 0) and the first header (in our case Ethernet).

Note: You could for instance sniff 10 packets and then assign them to a variable. In that case, you could reference that set of packets by index 0-9. In our case since the prn function gets called each time we get a packet we’ll always be looking at index 0.

That said – our packet summary looks a bit like this…

That makes pretty good sense – but it turns out you can also reference the sections by name. For instance we could change the program to look like this…

import scapy.all as scapy

def process_packet(packet):
    packet.show()
    destination_mac_address = packet[0][scapy.Ether].dst
    destination_ip_address = packet[0][scapy.IP].dst
    print("Received a packet with a destination MAC address of %s and a destination IP address of %s" % (destination_mac_address,destination_ip_address))

ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet, count=1)

Notice how we’re now able to refer to the fields by name instead of by index? Running this version of the program would provide the same results. I myself dislike this approach for two reasons. First, the variable names aren’t the same that as what shows up in the packet display. AKA – Ether is not Ethernet. Second – I like to import Scapy a certain way. Notice how I do an import scapy.all as scapy? If you look at lots of other examples you’ll see that it’s common to just do from scapy.all import *. This allows you to just reference the different Scapy types just by name. I tend to prefer to be rather explicit in the code so I can tell what comes from where. This is truly a matter of “to each their own” so you’ll need to figure out what works best for you. So for now, we’ll revert to using the index based approach.

So now we know how to get traffic off the wire and look at the values from each packet. Now let’s talk about how to put some of this together to work toward our goal of ping return. The first thing we need to do is figure out what kind of datagram we have coming in. Let me show you what I mean. Let’s update our program to look like this…

import scapy.all as scapy

def process_packet(packet):
    packet.show()
    destination_mac_address = packet[0][0].dst
    destination_ip_address = packet[0][1].dst
    icmp_type_code = packet[0][2].type
    print("Received a packet with a destination MAC address of %s and a destination IP address of %s and ICMP type code of %s" % (destination_mac_address,destination_ip_address, icmp_type_code))

ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet)

We’ve made two big changes. First, we removed the count command from the sniff instantiation. This just means that our prn function will get every packet it picks up with no limit. Second we added a new field to grab the ICMP type code from the packet. Let’s give that a run….

root@server:~# python3 ping_return.py 
###[ Ethernet ]### 
  dst       = 52:ab:54:cd:02:01
  src       = 52:ab:54:cd:01:01
  type      = IPv4
###[ IP ]### 
     version   = 4
     ihl       = 5
     tos       = 0x0
     len       = 84
     id        = 22559
     flags     = DF
     frag      = 0
     ttl       = 64
     proto     = icmp
     chksum    = 0xba75
     src       = 10.10.10.0
     dst       = 10.10.10.1
     \options   \
###[ ICMP ]### 
        type      = echo-request
        code      = 0
        chksum    = 0x6124
        id        = 0x2
        seq       = 0x6881
        unused    = ''
###[ Raw ]### 
           load      = 'Cį`\x00\x00\x00\x00s`\t\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./01234567'

Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
###[ Ethernet ]### 
  dst       = 52:ab:54:cd:01:01
  src       = 52:ab:54:cd:02:01
  type      = IPv4
###[ IP ]### 
     version   = 4
     ihl       = 5
     tos       = 0x0
     len       = 84
     id        = 15055
     flags     = 
     frag      = 0
     ttl       = 64
     proto     = icmp
     chksum    = 0x17c6
     src       = 10.10.10.1
     dst       = 10.10.10.0
     \options   \
###[ ICMP ]### 
        type      = echo-reply
        code      = 0
        chksum    = 0x6924
        id        = 0x2
        seq       = 0x6881
        unused    = ''
###[ Raw ]### 
           load      = 'Cį`\x00\x00\x00\x00s`\t\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./01234567'

Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
###[ Ethernet ]### 
  dst       = 52:ab:54:cd:01:01
  src       = 52:ab:54:cd:02:01
  type      = ARP
###[ ARP ]### 
     hwtype    = 0x1
     ptype     = IPv4
     hwlen     = 6
     plen      = 4
     op        = who-has
     hwsrc     = 52:ab:54:cd:02:01
     psrc      = 10.10.10.1
     hwdst     = 00:00:00:00:00:00
     pdst      = 10.10.10.0

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/scapy/packet.py", line 428, in __getattr__
    fld, v = self.getfield_and_val(attr)
  File "/usr/local/lib/python3.8/dist-packages/scapy/packet.py", line 423, in getfield_and_val
    raise ValueError
ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ping_return.py", line 10, in 
    ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet)
  File "/usr/local/lib/python3.8/dist-packages/scapy/sendrecv.py", line 1263, in sniff
    sniffer._run(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/scapy/sendrecv.py", line 1210, in _run
    session.on_packet_received(p)
  File "/usr/local/lib/python3.8/dist-packages/scapy/sessions.py", line 108, in on_packet_received
    result = self.prn(pkt)
  File "ping_return.py", line 6, in process_packet
    destination_ip_address = packet[0][1].dst
  File "/usr/local/lib/python3.8/dist-packages/scapy/packet.py", line 430, in __getattr__
    return self.payload.__getattr__(attr)
  File "/usr/local/lib/python3.8/dist-packages/scapy/packet.py", line 428, in __getattr__
    fld, v = self.getfield_and_val(attr)
  File "/usr/local/lib/python3.8/dist-packages/scapy/packet.py", line 1828, in getfield_and_val
    raise AttributeError(attr)
AttributeError: dst
root@server:~#

Aha! Can you sort out what happened? We were assuming that every packet would have an ICMP header and we ended up capturing an ARP request. The packet summary displayed fine, but when we tried to look at packet[0][2].type it bombed because in an ARP frame the 3rd header section doesn’t exist. So we can fix this a couple of ways. Let’s explore both of them. First, we can add a filter as part of the sniff instantiation. For instance, we could change our program to look like this…

import scapy.all as scapy

def process_packet(packet):
    packet.show()
    destination_mac_address = packet[0][0].dst
    destination_ip_address = packet[0][1].dst
    icmp_type_code = packet[0][2].type
    print("Received a packet with a destination MAC address of %s and a destination IP address of %s and ICMP type code of %s" % (destination_mac_address,destination_ip_address, icmp_type_code))

ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet, filter="icmp")

All we did was add on a filter variable and pass in icmp. For those of you that are familiar with TCPDump filtering – this syntax will be largely the same. This means that we’ll only be sniffing for ICMP packets which means our prn function will never receive any ARP or non-ICMP packets to start with. While this is OK (it works), I sort of want to see the other packets we’re getting too so let’s try another approach…

import scapy.all as scapy

def process_packet(packet):
    if (packet.haslayer(scapy.ICMP)):
        destination_mac_address = packet[0][0].dst
        destination_ip_address = packet[0][1].dst
        icmp_type_code = packet[0][2].type
        print("Received a packet with a destination MAC address of %s and a destination IP address of %s and ICMP type code of %s" % (destination_mac_address,destination_ip_address, icmp_type_code))
    else:
        print("We received a non-ICMP packet")

ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet)

Here we check the packet for a certain header (or as Scapy refers to them “layer”). We check to see if the packet has an ICMP layer and if it does we run our logic. If it does not – we just print a message saying we don’t have it. Our output would look like this…

root@server:~# python3 ping_return.py 
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
We received a non-ICMP packet
We received a non-ICMP packet
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8

Cool! But Im not satisfied yet. I want to see what that packet was that wasn’t ICMP…

import scapy.all as scapy

def process_packet(packet):
    if (packet.haslayer(scapy.ICMP)):
        destination_mac_address = packet[0][0].dst
        destination_ip_address = packet[0][1].dst
        icmp_type_code = packet[0][2].type
        print("Received a packet with a destination MAC address of %s and a destination IP address of %s and ICMP type code of %s" % (destination_mac_address,destination_ip_address, icmp_type_code))
    else:
        print("We received a non-ICMP packet")
        print(packet.summary())

ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet)

With this slight change we can get a quick view into the packet that wasn’t ICMP…

^Croot@server:~# python3 ping_return.py 
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
We received a non-ICMP packet
Ether / ARP who has 10.10.10.0 says 10.10.10.1
We received a non-ICMP packet
Ether / ARP is at 52:ab:54:cd:01:01 says 10.10.10.0
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0
Received a packet with a destination MAC address of 52:ab:54:cd:02:01 and a destination IP address of 10.10.10.1 and ICMP type code of 8
Received a packet with a destination MAC address of 52:ab:54:cd:01:01 and a destination IP address of 10.10.10.0 and ICMP type code of 0

And as expected – the non-ICMP packet was indeed an ARP request. As one last tweak, what if we only want to look at ICMP echo requests? That is, we don’t want to process the ICMP echo reply as well. Simple, just check the type code! We are already looking at it – so let’s add some logic to only look for ICMP type code of 8…

import scapy.all as scapy

def process_packet(packet):
    if (packet.haslayer(scapy.ICMP)):
        icmp_type_code = packet[0][2].type
        if icmp_type_code == 8:
            src_ip_address = packet[0][1].src
            dst_ip_address = packet[0][1].dst
            print("Received an ICMP echo request from source %s with a destination of %s" % (src_ip_address, dst_ip_address))
    else:
        print("We received a non-ICMP packet")
        print(packet.summary())

ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet)

Quite a few changes here – but we’ve already talked about all of them. We are checking to make sure the packet is ICMP, checking that the ICMP type is an echo request, and then grabbing the source and destination IP of out the IP layer (1). Pretty straight forward right? There actually isnt much left to do here in order to make this a ping return program. All we need to do is configure the IP address that the client is trying to ping on an interface on the server. To do that we’ll use pyroute2 so let’s get that installed…

pip3 install pyroute2

Now we can add a new function that will add an IP address to an interface…

import scapy.all as scapy
import pyroute2

ipr = pyroute2.IPRoute()

def add_interface_ip(interface_name, ip_address):
    interface_index_id = ipr.link_lookup(ifname=interface_name)[0]
    ipr.addr("add", index=interface_index_id, address=ip_address, prefixlen=32)

def process_packet(packet):
    if (packet.haslayer(scapy.ICMP)):
        icmp_type_code = packet[0][2].type
        if icmp_type_code == 8:
            src_ip_address = packet[0][1].src
            dst_ip_address = packet[0][1].dst
            print("Received an ICMP echo request from source %s with a destination of %s" % (src_ip_address, dst_ip_address))
            add_interface_ip("ens6", dst_ip_address)
    else:
        print("We received a non-ICMP packet")
        print(packet.summary())

ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet)

The biggest change in the above is our new function add_interface_ip. Here we’re using pyroute2 to send netlink commands to the kernel to configure a specific IP address on a specific interface. Note that in the function we first need to get the link index of the interface we wish to add the IP address to. Also note that the function returns a list so we really just want to grab the first (and only) item in that list. This is then used as the index in the next command in which we add the IP address to the interface that matches the index. Since these IP’s will all be individual I can safety hard code the mask to a /32. So let’s run this new version on the server and see what happens…

root@server:~# python3 ping_return.py 
Received an ICMP echo request from source 10.10.10.0 with a destination of 10.10.10.1
Traceback (most recent call last):
  File "ping_return.py", line 22, in 
    ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet)
  File "/usr/local/lib/python3.8/dist-packages/scapy/sendrecv.py", line 1263, in sniff
    sniffer._run(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/scapy/sendrecv.py", line 1210, in _run
    session.on_packet_received(p)
  File "/usr/local/lib/python3.8/dist-packages/scapy/sessions.py", line 108, in on_packet_received
    result = self.prn(pkt)
  File "ping_return.py", line 17, in process_packet
    add_interface_ip("ens6", dst_ip_address)
  File "ping_return.py", line 8, in add_interface_ip
    ipr.addr("add", index=interface_index_id, address=ip_address, prefixlen=32)
  File "/usr/local/lib/python3.8/dist-packages/pr2modules/iproute/linux.py", line 1518, in addr
    ret = self.nlm_request(msg,
  File "/usr/local/lib/python3.8/dist-packages/pr2modules/netlink/nlsocket.py", line 391, in nlm_request
    return tuple(self._genlm_request(*argv, **kwarg))
  File "/usr/local/lib/python3.8/dist-packages/pr2modules/netlink/nlsocket.py", line 882, in nlm_request
    for msg in self.get(msg_seq=msg_seq,
  File "/usr/local/lib/python3.8/dist-packages/pr2modules/netlink/nlsocket.py", line 394, in get
    return tuple(self._genlm_get(*argv, **kwarg))
  File "/usr/local/lib/python3.8/dist-packages/pr2modules/netlink/nlsocket.py", line 719, in get
    raise msg['header']['error']
pr2modules.netlink.exceptions.NetlinkError: (17, 'File exists')
root@server:~#

Aha! If you had left your client ping running – you should immediately see this error message. The kernel doesnt want to let you add the same IP address to the same interface since it’s already there. So how should we fix this? The easiest thing to do would be to just not ping that IP. So let’s stop the ping on the client and run the script again on the server. Now on the client – try and ping a new IP – let’s say 1.1.1.1…

Note: I assume that the client server has a default route pointing at the server

root@client:~# ping 1.1.1.1 -c 5
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=0.509 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=64 time=0.299 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=64 time=0.180 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=64 time=0.252 ms

--- 1.1.1.1 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4102ms
rtt min/avg/max/mdev = 0.180/0.310/0.509/0.122 ms
root@client:~#

Woohhoo it worked! Note that we missed the first sequence number. That’s to be expected in this case as the server needs to receive the first ICMP echo request and then add the IP address to it’s interface. Unfortunately, if we look at the server we see we crashed again…

root@server:~# python3 ping_return.py 
Received an ICMP echo request from source 10.10.10.0 with a destination of 1.1.1.1
Received an ICMP echo request from source 10.10.10.0 with a destination of 1.1.1.1
Traceback (most recent call last):
  File "ping_return.py", line 22, in 
    ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet)
  File "/usr/local/lib/python3.8/dist-packages/scapy/sendrecv.py", line 1263, in sniff
    sniffer._run(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/scapy/sendrecv.py", line 1210, in _run
    session.on_packet_received(p)
  File "/usr/local/lib/python3.8/dist-packages/scapy/sessions.py", line 108, in on_packet_received
    result = self.prn(pkt)
  File "ping_return.py", line 17, in process_packet
    add_interface_ip("ens6", dst_ip_address)
  File "ping_return.py", line 8, in add_interface_ip
    ipr.addr("add", index=interface_index_id, address=ip_address, prefixlen=32)
  File "/usr/local/lib/python3.8/dist-packages/pr2modules/iproute/linux.py", line 1518, in addr
    ret = self.nlm_request(msg,
  File "/usr/local/lib/python3.8/dist-packages/pr2modules/netlink/nlsocket.py", line 391, in nlm_request
    return tuple(self._genlm_request(*argv, **kwarg))
  File "/usr/local/lib/python3.8/dist-packages/pr2modules/netlink/nlsocket.py", line 882, in nlm_request
    for msg in self.get(msg_seq=msg_seq,
  File "/usr/local/lib/python3.8/dist-packages/pr2modules/netlink/nlsocket.py", line 394, in get
    return tuple(self._genlm_get(*argv, **kwarg))
  File "/usr/local/lib/python3.8/dist-packages/pr2modules/netlink/nlsocket.py", line 719, in get
    raise msg['header']['error']
pr2modules.netlink.exceptions.NetlinkError: (17, 'File exists')
root@server:~#

Uggh. Well at least it’s for the same reason! We can see that when the first request came in all seemed to be well. However when the second request came in we ran into the same issue. So now that we know that the base premise works – let’s think about a better way of doing this.

The base problem can be addressed by having the server keep track of what IPs it has already configured. That seems easy enough so let’s tackle that first…

import scapy.all as scapy
import pyroute2

configured_ips = []
ipr = pyroute2.IPRoute()

def add_interface_ip(interface_name, ip_address):
    interface_index_id = ipr.link_lookup(ifname=interface_name)[0]
    ipr.addr("add", index=interface_index_id, address=ip_address, prefixlen=32)

def process_packet(packet):
    if (packet.haslayer(scapy.ICMP)):
        icmp_type_code = packet[0][2].type
        if icmp_type_code == 8:
            src_ip_address = packet[0][1].src
            dst_ip_address = packet[0][1].dst
            print("Received an ICMP echo request from source %s with a destination of %s" % (src_ip_address, dst_ip_address))
            if dst_ip_address not in configured_ips:
                configured_ips.append(dst_ip_address)
                add_interface_ip("ens6", dst_ip_address)
    else:
        print("We received a non-ICMP packet")
        print(packet.summary())

ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet)

To fix that we simply add a list of IPs we’ve configured and make sure that we don’t process the same IP twice. Easy! However the problem still remains even with this code that the IPs we configured in previous runs will still be there and cause the same problem for us. We should probably write a clean up routine that checks for extra IPs and cleans them up before we start sniffing for traffic. As part of this, I’d suggest we also move the configuration of the IPs we sniff to the loopback interface rather than the physical interface. This will keep things much cleaner and not pollute the actual dataplane interfaces with extraneous IPs. So let’s start with a cleanup routine. We’ll create a function like this…

def cleanup_ips(interface_name):
    interface_index_id = ipr.link_lookup(ifname=interface_name)[0]
    interface_addresses = ipr.get_addr(index=interface_index_id, family=2)
    for address in interface_addresses:
        ip = address.get_attrs("IFA_ADDRESS")[0]
        if ip != "127.0.0.1":
            print("Cleanup - Deleting IP address %s from interface %s" % (ip, interface_name))
            delete_interface_ip(interface_name, ip)

So all we’re doing here is taking in an interface name and then using pyroute2 to grab all of the IP addresses from that interface. As before, we need to do this by link index so we have to grab that first. Also by specifying family=2 as part of the call we tell it to only return IPv4 addresses.

Note: If you’re wondering how 2 equals the IPv4 family you need to look a little deeper at Linux. The best place I can find this defined is here. Here you can see that AF_INET which is the IPv4 address family matches to a type of 2.

The pyroute2 call will return a list of IP addresses. We then need to iterate through them and delete the ones that we no longer want. Specifically we want to avoid deleting 127.0.0.1. The line ip = address.get_attrs("IFA_ADDRESS")[0] might be slightly confusing. If it is – it might help to look at the data that is returned from pyroute2. You can see this yourself by simply adding a print statement for address at the beginning of the for loop. If we did that – we might see this in the output…

{'family': 2, 'prefixlen': 32, 'flags': 128, 'scope': 0, 'index': 1, 'attrs': [('IFA_ADDRESS', '1.2.3.4'), ('IFA_LOCAL', '1.2.3.4'), ('IFA_LABEL', 'lo'), ('IFA_FLAGS', 128), ('IFA_CACHEINFO', {'ifa_preferred': 4294967295, 'ifa_valid': 4294967295, 'cstamp': 23959062, 'tstamp': 23959062})], 'header': {'length': 76, 'type': 20, 'flags': 2, 'sequence_number': 256, 'pid': 3992, 'error': None, 'target': 'localhost', 'stats': Stats(qsize=0, delta=0, delay=0)}, 'event': 'RTM_NEWADDR'}

And now formatted more nicely….

{
    "family": 2,
    "prefixlen": 32,
    "flags": 128,
    "scope": 0,
    "index": 1,
    "attrs": [
        ("IFA_ADDRESS", "1.2.3.4"),
        ("IFA_LOCAL", "1.2.3.4"),
        ("IFA_LABEL", "lo"),
        ("IFA_FLAGS", 128),
        (
            "IFA_CACHEINFO",
            {
                "ifa_preferred": 4294967295,
                "ifa_valid": 4294967295,
                "cstamp": 23959062,
                "tstamp": 23959062,
            },
        ),
    ],
    "header": {
        "length": 76,
        "type": 20,
        "flags": 2,
        "sequence_number": 256,
        "pid": 3992,
        "error": None,
        "target": "localhost",
        "stats": Stats(qsize=0, delta=0, delay=0),
    },
    "event": "RTM_NEWADDR",
}

You can see that there’s a lot of information here. pyroute2 has these handy functions such as get_attr to help you grab some of this data. So in our case – when we say address.get_attrs("IFA_ADDRESS")[0] we’re saying that we want to parse through the attributes for the base item (this address) and return the value for the key "IFLA_ADDRESS". And while it doesn’t look like it – that function will actually return a list hence the need to say [0] at the end. It’s interesting to note that if you want the full address, that is the IP and the netmask, you need to grab the netmask from elsewhere. The netmask is called prefixlen and is in the base object and not in the attribute section.

Once we have that info – we call a function called delete_interface_ip which looks suspicously like the add_interface_ip function we already wrote…

def delete_interface_ip(interface_name, ip_address):
    interface_index_id = ipr.link_lookup(ifname=interface_name)[0]
    ipr.addr("delete", index=interface_index_id, address=ip_address, prefixlen=32)

Now if we put this all together our final script will look like this….

import scapy.all as scapy
import pyroute2

configured_ips = []
ipr = pyroute2.IPRoute()

def add_interface_ip(interface_name, ip_address):
    interface_index_id = ipr.link_lookup(ifname=interface_name)[0]
    ipr.addr("add", index=interface_index_id, address=ip_address, prefixlen=32)

def delete_interface_ip(interface_name, ip_address):
    interface_index_id = ipr.link_lookup(ifname=interface_name)[0]
    ipr.addr("delete", index=interface_index_id, address=ip_address, prefixlen=32)

def cleanup_ips(interface_name):
    interface_index_id = ipr.link_lookup(ifname=interface_name)[0]
    interface_addresses = ipr.get_addr(index=interface_index_id, family=2)
    for address in interface_addresses:
        ip = address.get_attrs("IFA_ADDRESS")[0]
        if ip != "127.0.0.1":
            print("Cleanup - Deleting IP address %s from interface %s" % (ip, interface_name))
            delete_interface_ip(interface_name, ip)

def process_packet(packet):
    if (packet.haslayer(scapy.ICMP)):
        icmp_type_code = packet[0][2].type
        if icmp_type_code == 8:
            src_ip_address = packet[0][1].src
            dst_ip_address = packet[0][1].dst
            if dst_ip_address not in configured_ips:
                configured_ips.append(dst_ip_address)
                print("Received an ICMP echo request from source %s with a destination of %s" % (src_ip_address, dst_ip_address))
                add_interface_ip("lo", dst_ip_address)
    else:
        print("We received a non-ICMP packet")
        print(packet.summary())


cleanup_ips("lo")
ens6_traffic = scapy.sniff(iface="ens6", prn=process_packet)

Beyond the check to make sure we don’t process an IP twice the only other new piece is calling our cleanup function before we start sniffing. So let’s give this a try…

Note: Make sure to go and manually clean up any IPs you have on ens6 from earlier testing

root@client:~# ping 1.1.1.1 -c 5
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=0.550 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=64 time=0.254 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=64 time=0.239 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=64 time=0.277 ms

--- 1.1.1.1 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4082ms
rtt min/avg/max/mdev = 0.239/0.330/0.550/0.127 ms
root@client:~# ping 2.2.2.2 -c 5
PING 2.2.2.2 (2.2.2.2) 56(84) bytes of data.
64 bytes from 2.2.2.2: icmp_seq=2 ttl=64 time=0.320 ms
64 bytes from 2.2.2.2: icmp_seq=3 ttl=64 time=0.256 ms
64 bytes from 2.2.2.2: icmp_seq=4 ttl=64 time=0.271 ms
64 bytes from 2.2.2.2: icmp_seq=5 ttl=64 time=0.241 ms

--- 2.2.2.2 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4081ms
rtt min/avg/max/mdev = 0.241/0.272/0.320/0.029 ms
root@client:~# ping 3.3.3.3 -c 5
PING 3.3.3.3 (3.3.3.3) 56(84) bytes of data.
64 bytes from 3.3.3.3: icmp_seq=2 ttl=64 time=0.309 ms
64 bytes from 3.3.3.3: icmp_seq=3 ttl=64 time=0.256 ms
64 bytes from 3.3.3.3: icmp_seq=4 ttl=64 time=0.258 ms
64 bytes from 3.3.3.3: icmp_seq=5 ttl=64 time=0.316 ms

--- 3.3.3.3 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4089ms
rtt min/avg/max/mdev = 0.256/0.284/0.316/0.027 ms
root@client:~#

The client looks happy! It seems it can ping any IP address it wants. On the server side we see the log messages for things the sniffer picked up…

Received an ICMP echo request from source 10.10.10.0 with a destination of 1.1.1.1
We received a non-ICMP packet
Ether / ARP who has 10.10.10.1 says 10.10.10.0
We received a non-ICMP packet
Ether / ARP is at 52:ab:54:cd:02:01 says 10.10.10.1
We received a non-ICMP packet
Ether / ARP who has 10.10.10.0 says 10.10.10.1
We received a non-ICMP packet
Ether / ARP is at 52:ab:54:cd:01:01 says 10.10.10.0
Received an ICMP echo request from source 10.10.10.0 with a destination of 2.2.2.2
Received an ICMP echo request from source 10.10.10.0 with a destination of 3.3.3.3

And now if we restart the server we should see it clean up those IPs as well…

root@server:~# python3 ping_return.py 
Cleanup - Deleting IP address 1.1.1.1 from interface lo
Cleanup - Deleting IP address 2.2.2.2 from interface lo
Cleanup - Deleting IP address 3.3.3.3 from interface lo

Nice! Now this still really isnt a great solution. The only way this is working right now is because the source IP address of the ping is on a directly connected route for the server. A better solution would be to add a static route pointing back out the sniffer interface for each source IP we see. This would mean that the client could be many hops away and it should still work. I’m not going to write that code in this blog, but I can assure you it’s easy to do using pyroute2 (wow – that rhymed. That should be their slogan or something).

So I hope you can see that performing actions based on packets on the wire isn’t hard to do. This script is less than 40 lines of Python and we were already able to fulfill our requirements for the ping return application. In the next post we’ll attempt the same thing but in a different programming language! Stay tuned!

Working with Linux VRFs

Jon Langemak — Wed, 28 Apr 2021 02:02:50 +0000

The concept of VRFs is likely one that you’re familiar with. They are the de facto standard when we talk about isolating layer 3 networks. As we’ve talked about previously, they are used extensively in applications such as MPLS VPNs and really provide the foundation for layer 3 network isolation. They do this by allowing the creation of multiple routing tables. Any layer 3 construct can then be mapped into the VRF. For instance, I could assign an IP address to an interface and then map that interface into the VRF. Likewise, I could configure a static route and specify that the route is part of a given VRF. Going one step further I could establish a BGP session off of one of the VRF interfaces and receive remote BGP routes into the VRF. VRFs are to layer 3 like VLANs are to layer 2.

So while we’ve talked about how they are typically used and implemented on networking hardware like routers and switches – we haven’t talked about how they’re implemented in Linux. Actually – they’re fairly new to the Linux space. The functionality was actually written by Cumulus Networks and then contributed to the Linux kernel (kudos to them for doing that). VRFs were introduced in kernel version 4.3 which I believe came to be around 2015. That said – they were, in my opinion, a long time coming. So what did folks do before VRFs were introduced? Let’s talk about that for a moment.

There were really 2 options available to you to create a VRF like construct in Linux prior to VRFs landing. One option was to leverage multiple routing tables and policy routing to make something that looked VRF like. I tried this once – I wouldn’t advise it – the shortcomings far outweighed the advantages and I’d argue you’d have a heck of a hard time getting that solution past any sort of audit around network isolation. The far more popular option was to leverage network namespaces. This become incredibly popular when containers came on the scene and began using network namespaces to manage container isolation. But while network namespaces certainly work well – they are overkill for what we want. Cumulus talks about this at length in their article about VRFs so I won’t reiterate their points. However it is worth calling out the major drawback at least once. Network namespaces provide total and complete isolation for “all the things”. In our case, “all the things” includes the entire network stack – devices, interfaces, ARP tables, route tables, etc. Those of you familiar with how VRFs work know that’s not what we normally get with VRF functionality in a “normal” (I guess that’s what Im calling vendor provided) router. This is obvious to see if we look at an example VRF configuration on a normal router. For instance let’s look at a setup that looks like this…

Above we have a single router that has two interfaces. Interface ge-0/0/0 is a member of vrf vrf-1 while interface ge-0/0/1 is a member of vrf-2. We also have some static routes defined in each VRF. Pretty straight forward right? The configuration on the router might look something like this…

set interfaces ge-0/0/0 unit 0 family inet address 192.168.10.1/24
set interfaces ge-0/0/1 unit 0 family inet address 192.168.10.1/24

set routing-instances vrf-1 instance-type virtual-router
set routing-instances vrf-1 interface ge-0/0/0.0
set routing-instances vrf-1 routing-options static route 0.0.0.0/0 next-hop 192.168.10.254
set routing-instances vrf-1 routing-options static route 172.64.32.0/24 next-hop 192.168.10.101

set routing-instances vrf-2 instance-type virtual-router
set routing-instances vrf-2 interface ge-0/0/1.0
set routing-instances vrf-2 routing-options static route 0.0.0.0/0 next-hop 192.168.10.10
set routing-instances vrf-2 routing-options static route 192.168.128.0/24 next-hop 192.168.10.20

Note: In JunOS parlance a VRF that is local (Cisco used to call this VRF-lite) AKA one that is not used as part of a MPLS VPN and does not have a RD/RT assigned is of type “virtual-router” instead of type “vrf”.

Again – nothing crazy here in the configuration. But let’s take a look at the routing tables etc.

root@vmx1.lab> show route table vrf-1.inet.0    

vrf-1.inet.0: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 00:00:02
                    > to 192.168.10.254 via ge-0/0/0.0
172.64.32.0/24     *[Static/5] 00:00:02
                    > to 192.168.10.101 via ge-0/0/0.0
192.168.10.0/24    *[Direct/0] 00:01:10
                    > via ge-0/0/0.0
192.168.10.1/32    *[Local/0] 00:01:10
                      Local via ge-0/0/0.0

root@vmx1.lab> show route table vrf-2.inet.0    

vrf-2.inet.0: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 00:00:06
                    > to 192.168.10.10 via ge-0/0/1.0
192.168.10.0/24    *[Direct/0] 00:01:14
                    > via ge-0/0/1.0
192.168.10.1/32    *[Local/0] 00:01:14
                      Local via ge-0/0/1.0
192.168.128.0/24   *[Static/5] 00:00:06
                    > to 192.168.10.20 via ge-0/0/1.0

root@vmx1.lab>

The important thing to call out here is that I need to tell the router I want to look at a specific VRFs routing table. But what I don’t need to do is look inside the VRF context to see things that are common to the platform. The most obvious case of this is the configuration I pasted above. I have one global configuration that is used for all the VRFs. Sure, I map things into specific VRFs but I have a single pane of glass view into all of that. Even though my interfaces are now mapping into a VRF – I can still see them on the platform without any specific command to look inside the routing instance…

root@vmx1.lab> show interfaces terse | grep ge-0/0/ 
ge-0/0/0                up    up
ge-0/0/0.0              up    up   inet     192.168.10.1/24 
ge-0/0/1                up    up
ge-0/0/1.0              up    up   inet     192.168.10.1/24 
ge-0/0/2                up    down
ge-0/0/3                up    down
ge-0/0/4                up    down
ge-0/0/5                up    down
ge-0/0/6                up    down
ge-0/0/7                up    down
ge-0/0/8                up    down
ge-0/0/9                up    down

root@vmx1.lab>

And perhaps more importantly – services that run on the router that aren’t L3 aware can just run once on the platform as a whole. That is, I don’t need to run all of these services in all of the VRFs. A great example of this is something like LLDP. I’ve mapped my two ‘up’ interfaces into VRFs, but LLDP is still just running on the platform and I can still see all of the LLDP neighbors etc without having to run the software or command in a given VRF…

root@vmx1.lab> show lldp neighbors 
Local Interface    Parent Interface    Chassis Id          Port info          System Name
ge-0/0/1           -                   2c:6b:f5:4a:23:c0   ge-0/0/0           vmx3.lab            
ge-0/0/0           -                   2c:6b:f5:f9:08:c0   ge-0/0/1           vmx2.lab            

root@vmx1.lab>

If this still comes across as a weak argument to you, let’s consider the same scenario if we were to use network namespaces on Linux services to provide L3 isolation. Let’s say we have a Linux server that looks like this..

Configuring something like this might look like this…

ip netns add namespace-1
ip link set dev ens6 netns namespace-1
ip netns exec namespace-1 ip link set dev ens6 up
ip netns exec namespace-1 ip address add 192.168.10.1/24 dev ens6
ip netns exec namespace-1 ip route add 0.0.0.0/0 via 192.168.10.254
ip netns exec namespace-1 ip route add 172.64.32.0/24 via 192.168.10.101

ip netns add namespace-2
ip link set dev ens7 netns namespace-2
ip netns exec namespace-2 ip link set dev ens7 up
ip netns exec namespace-2 ip address add 192.168.10.1/24 dev ens7
ip netns exec namespace-2 ip route add 0.0.0.0/0 via 192.168.10.10
ip netns exec namespace-2 ip route add 192.168.128.0/24 via 192.168.10.20

Not too far off from where we were with the router. We configure the namespace, add an interface to it, and then add other routes inside the namespace. And we end up with something that looks pretty close to what we had before…

root@vm1:~# ip netns exec namespace-1 ip route show
default via 192.168.10.254 dev ens6 
172.64.32.0/24 via 192.168.10.101 dev ens6 
192.168.10.0/24 dev ens6 proto kernel scope link src 192.168.10.1 
root@vm1:~# ip netns exec namespace-2 ip route show
default via 192.168.10.10 dev ens7 
192.168.10.0/24 dev ens7 proto kernel scope link src 192.168.10.1 
192.168.128.0/24 via 192.168.10.20 dev ens7 
root@vm1:~#

Now here’s where things begin to diverge quickly. If I wanted to run something like LLDP on this server – how would I do that? Let’s start by installing it and seeing what it sees…

root@vm1:~# apt -y install lldpd
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  snmpd
The following NEW packages will be installed:
  lldpd
0 upgraded, 1 newly installed, 0 to remove and 126 not upgraded.
Need to get 0 B/154 kB of archives.
After this operation, 511 kB of additional disk space will be used.
Selecting previously unselected package lldpd.
(Reading database ... 107391 files and directories currently installed.)
Preparing to unpack .../lldpd_1.0.4-1build2_amd64.deb ...
Unpacking lldpd (1.0.4-1build2) ...
Setting up lldpd (1.0.4-1build2) ...
Processing triggers for libc-bin (2.31-0ubuntu9.1) ...
Processing triggers for systemd (245.4-4ubuntu3.3) ...
Processing triggers for man-db (2.9.1-1) ...
root@vm1:~# 
root@vm1:~# lldpcli show interfaces
-------------------------------------------------------------------------------
LLDP interfaces:
-------------------------------------------------------------------------------
Interface:    ens3, via: unknown, Time: 0 day, 00:19:09
  Chassis:     
    ChassisID:    mac 52:ab:54:ab:01:01
    SysName:      vm1
    SysDescr:     Ubuntu 20.04.1 LTS Linux 5.9.4-050904-generic #202011042130 SMP Wed Nov 4 21:36:10 UTC 2020 x86_64
    MgmtIP:       192.168.127.1
    MgmtIP:       fe80::50ab:54ff:feab:101
    Capability:   Bridge, off
    Capability:   Router, off
    Capability:   Wlan, off
    Capability:   Station, on
  Port:        
    PortID:       mac 52:ab:54:ab:01:01
    PortDescr:    ens3
  TTL:          120
-------------------------------------------------------------------------------
root@vm1:~# lldpcli show neighbors
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
root@vm1:~#

Notice that it doesn’t detect ens6 or ens7, the two interfaces we moved into the namespaces. But this makes total sense. The interfaces are no longer in the global namespaces and simply can’t be seen. Again – compare this to our router where we could still see the interfaces – we had just isolated the L3 constructs. To get this to work, we’d actually need to run lldpd in each namespace. Since by default lldpd uses the same socket we’d need to create unique sockets for each instance. Something like this…

root@vm1:~# ip netns exec namespace-1 /usr/sbin/lldpd -u /var/run/lldpd-namespace-1.socket
root@vm1:~# ip netns exec namespace-1 lldpcli -u /var/run/lldpd-namespace-1.socket show interfaces
-------------------------------------------------------------------------------
LLDP interfaces:
-------------------------------------------------------------------------------
Interface:    ens6, via: unknown, Time: 18739 days, 20:45:55
  Chassis:     
    ChassisID:    mac 52:ab:54:cd:01:01
    SysName:      vm1
    SysDescr:     Ubuntu 20.04.1 LTS Linux 5.9.4-050904-generic #202011042130 SMP Wed Nov 4 21:36:10 UTC 2020 x86_64
    MgmtIP:       192.168.10.1
    MgmtIP:       fe80::50ab:54ff:fecd:101
    Capability:   Bridge, off
    Capability:   Router, off
    Capability:   Wlan, off
    Capability:   Station, on
  Port:        
    PortID:       mac 52:ab:54:cd:01:01
    PortDescr:    ens6
  TTL:          120
-------------------------------------------------------------------------------
root@vm1:~# ip netns exec namespace-1 lldpcli -u /var/run/lldpd-namespace-1.socket show neighbors
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    ens6, via: LLDP, RID: 1, Time: 0 day, 00:00:39
  Chassis:     
    ChassisID:    mac 52:ab:54:ab:02:01
    SysName:      vm2
    SysDescr:     Ubuntu 20.04.1 LTS Linux 5.9.4-050904-generic #202011042130 SMP Wed Nov 4 21:36:10 UTC 2020 x86_64
    MgmtIP:       192.168.127.2
    MgmtIP:       fe80::50ab:54ff:feab:201
    Capability:   Bridge, off
    Capability:   Router, off
    Capability:   Wlan, off
    Capability:   Station, on
  Port:        
    PortID:       mac 52:ab:54:cd:02:02
    PortDescr:    ens7
    TTL:          120
-------------------------------------------------------------------------------
root@vm1:~#

So we can run another instance of lldpd in the namespace which allows us to see the local namespace interfaces and neighbors that are available out of those interfaces. The same could be done for namespace-2…

root@vm1:~# ip netns exec namespace-2 /usr/sbin/lldpd -u /var/run/lldpd-namespace-2.socket
root@vm1:~# ip netns exec namespace-2 lldpcli -u /var/run/lldpd-namespace-2.socket show interfaces
-------------------------------------------------------------------------------
LLDP interfaces:
-------------------------------------------------------------------------------
Interface:    ens7, via: unknown, Time: 18739 days, 20:47:14
  Chassis:     
    ChassisID:    mac 52:ab:54:cd:01:02
    SysName:      vm1
    SysDescr:     Ubuntu 20.04.1 LTS Linux 5.9.4-050904-generic #202011042130 SMP Wed Nov 4 21:36:10 UTC 2020 x86_64
    MgmtIP:       192.168.10.1
    MgmtIP:       fe80::50ab:54ff:fecd:102
    Capability:   Bridge, off
    Capability:   Router, off
    Capability:   Wlan, off
    Capability:   Station, on
  Port:        
    PortID:       mac 52:ab:54:cd:01:02
    PortDescr:    ens7
  TTL:          120
-------------------------------------------------------------------------------
root@vm1:~# ip netns exec namespace-2 lldpcli -u /var/run/lldpd-namespace-2.socket show neighbors
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    ens7, via: LLDP, RID: 1, Time: 0 day, 00:00:05
  Chassis:     
    ChassisID:    mac 52:ab:54:ab:03:01
    SysName:      vm3
    SysDescr:     Ubuntu 20.04.1 LTS Linux 5.9.4-050904-generic #202011042130 SMP Wed Nov 4 21:36:10 UTC 2020 x86_64
    MgmtIP:       192.168.127.3
    MgmtIP:       fe80::50ab:54ff:feab:301
    Capability:   Bridge, off
    Capability:   Router, off
    Capability:   Wlan, off
    Capability:   Station, on
  Port:        
    PortID:       mac 52:ab:54:cd:03:02
    PortDescr:    ens7
    TTL:          120
-------------------------------------------------------------------------------
root@vm1:~#

Ok – but now we have 3 lldpd processes running…

root@vm1:~# ps -fC lldpd
UID          PID    PPID  C STIME TTY          TIME CMD
_lldpd      2304       1  0 12:53 ?        00:00:00 lldpd: monitor. 
_lldpd      2306    2304  0 12:53 ?        00:00:00 lldpd: no neighbor.
_lldpd      2501       1  0 12:53 ?        00:00:00 lldpd: monitor. 
_lldpd      2503    2501  0 12:53 ?        00:00:00 lldpd: connected to vm2.
_lldpd      2511       1  0 12:56 ?        00:00:00 lldpd: monitor. 
_lldpd      2513    2511  0 12:56 ?        00:00:00 lldpd: connected to vm3.
root@vm1:~#

There’s totally no need to isolate things that don’t need to be isolated – the only thing we gain by doing that is creating more overhead on the router. That said – I hope this shows that while namespaces do provide the necessary isolation – they are overkill for what we’re trying to achieve. As Cumulus said in their article “Network Namespace as a VRF? Just say No”.

So now that we’ve looked at the other possible options for solving this problem – let’s talk about using actual VRFs in Linux. So same setup we’ve done above – but now with VRFs!

The initial implementation of VRFs might initially feel a little different than what you’re used to – but once you start using it I’d argue it makes good sense. The nice thing about the VRF implementation in Linux is that it’s all done through existing tooling. That’s huge! So all we need is the existing iproute2 packages which in most cases should already be on your system. Once there – all you need to do to create a VRF is this…

root@vm1:~# ip link add vrf-1 type vrf table 1
root@vm1:~# ip link set dev vrf-1 up
root@vm1:~# ip vrf
Name              Table
-----------------------
vrf-1                1
root@vm1:~#

Easy! So now we have a VRF – what do with it? Well – we can add interfaces to the VRF quite easily…

root@vm1:~# ip link set dev ens6 master vrf-1
root@vm1:~# ip addr add 192.168.10.1/24 dev ens6
root@vm1:~# 
root@vm1:~# ip route show vrf vrf-1
192.168.127.0/24 dev ens6 proto kernel scope link src 192.168.127.1 
root@vm1:~#

Above we add ens6 to the VRF vrf-1 and then we add an IP address to it. Then we can look at the vrf-1 routing table using the ip route show vrf syntax. I’ll also point out that since the interface ens6 is now in the VRF it does not mean it’s gone from the global namespace as was the case with namespaces…

root@vm1:~# ip -br link
lo               UNKNOWN        00:00:00:00:00:00  
ens3             UP             52:ab:54:ab:01:01  
ens6             UP             52:ab:54:cd:01:01  
ens7             UP             52:ab:54:cd:01:02  
vrf-1            UP             4e:19:81:5c:ad:ee  
root@vm1:~#

So now that we have an interface in the VRF – we can add static routes…

root@vm1:~# ip route add 0.0.0.0/0 via 192.168.127.254 vrf vrf-1
root@vm1:~# ip route add 172.64.32.0/24 via 192.168.127.101 vrf vrf-1
root@vm1:~# ip route show vrf vrf-1
default via 192.168.127.254 dev ens6 
172.64.32.0/24 via 192.168.127.101 dev ens6 
192.168.127.0/24 dev ens6 proto kernel scope link src 192.168.127.1 
root@vm1:~#

Let’s now configure vrf-2 all in one shot…

ip link add vrf-2 type vrf table 2
ip link set dev vrf-2 up
ip link set dev ens7 master vrf-2
ip addr add 192.168.127.1/24 dev ens7
ip route add 0.0.0.0/0 via 192.168.127.10 vrf vrf-2
ip route add 192.168.128.0/24 via 192.168.127.20 vrf vrf-2

So now that we have the full configuration – lets back up for a second. When we created the VRF – it looked a lot like we were creating another network device. In fact we were and the device that was created is sometimes called a “layer 3 master device” (l3mdev for short). This device acts like a sort of central point that you can connect all of the VRF interfaces to. The process of adding interfaces to the VRF might have looked familiar to you if you’ve worked with Linux bridges before. If so – it’s because it’s done in the exact same way! When they designed the VRF functionality they modeled it in the same way that you add devices to a bridge domain. The mapping even looks the same if we look at the interfaces…

root@vm1:~# ip -d link show dev ens6
3: ens6:  mtu 1500 qdisc mq master vrf-1 state UP mode DEFAULT group default qlen 1000
    link/ether 52:ab:54:cd:01:01 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    vrf_slave table 1 addrgenmode eui64 numtxqueues 2 numrxqueues 2 gso_max_size 65536 gso_max_segs 65535 
    altname enp0s6
root@vm1:~#

See how it shows that the master is vrf-1 and in the details we can the VRF table number as well? Not only does this make it feel more natural to folks already familiar with the concepts – but it also means that you can do some interesting things with the VRF device. For instance, what if you want to see traffic in your VRF? No problem – just do a TCPDUMP on the VRF interface…

root@vm1:~# tcpdump -nnel -i vrf-1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vrf-1, link-type EN10MB (Ethernet), capture size 262144 bytes
21:04:59.140107 42:f6:86:4a:a6:17 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 98: 192.168.10.1 > 192.168.10.254: ICMP echo request, id 2, seq 466, length 64
21:04:59.140575 52:ab:54:cd:02:02 > 52:ab:54:cd:01:01, ethertype IPv4 (0x0800), length 98: 192.168.10.254 > 192.168.10.1: ICMP echo reply, id 2, seq 466, length 64
21:05:00.164085 42:f6:86:4a:a6:17 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 98: 192.168.10.1 > 192.168.10.254: ICMP echo request, id 2, seq 467, length 64
21:05:00.164390 52:ab:54:cd:02:02 > 52:ab:54:cd:01:01, ethertype IPv4 (0x0800), length 98: 192.168.10.254 > 192.168.10.1: ICMP echo reply, id 2, seq 467, length 64
21:05:01.188086 42:f6:86:4a:a6:17 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 98: 192.168.10.1 > 192.168.10.254: ICMP echo request, id 2, seq 468, length 64
21:05:01.188320 52:ab:54:cd:02:02 > 52:ab:54:cd:01:01, ethertype IPv4 (0x0800), length 98: 192.168.10.254 > 192.168.10.1: ICMP echo reply, id 2, seq 468, length 64

Same thing works for ping – we can just set the source interface to the VRF device name and…

root@vm1:~# ping -I vrf-1 192.168.10.254
ping: Warning: source address might be selected on device other than: vrf-1
PING 192.168.10.254 (192.168.10.254) from 192.168.10.1 vrf-1: 56(84) bytes of data.
64 bytes from 192.168.10.254: icmp_seq=1 ttl=64 time=0.250 ms
64 bytes from 192.168.10.254: icmp_seq=2 ttl=64 time=0.253 ms
64 bytes from 192.168.10.254: icmp_seq=3 ttl=64 time=0.444 ms

We can also add IP addresses to the VRF interface itself…

root@vm1:~# ip addr add 1.1.1.1/32 dev vrf-1
root@vm1:~# ip route get vrf vrf-1 1.1.1.1
local 1.1.1.1 dev vrf-1 table 1 src 1.1.1.1 uid 0 
    cache  
root@vm1:~#

This IP can be used as you would a loopback interface in the VRF.

So at this point we’ve shown how you can create and use VRFs – but we haven’t really talked about how we actually land traffic into the VRF. We know that when we created the VRF we had to assign it a routing table number. When dealing with routing tables we generally need to make some rule sets to handle them so let’s take a look at the existing IP rules…

root@vm1:~# ip rule show
0:	from all lookup local
1000:	from all lookup [l3mdev-table]
32766:	from all lookup main
32767:	from all lookup default
root@vm1:~#

Notice rule 1000. Now look at a different box that you haven’t configured any VRFs on…

root@vm2:~# ip rule show
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default
root@vm2:~#

Notice that rule 1000 isn’t present in the box with no VRFs defined. Rule 1000 is created when a VRF is configured and it’s what allows the VRF lookups to magically work. But let’s dig in here for a minute. What does the IP rule set on vm1 actually do? More importantly, what does rule 0 do that is listed before our magic VRF lookup rule?

Rule 0 exists by default everywhere and provides local lookups. So what does that mean?

root@vm1:~# ip route show table local
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 
broadcast 192.168.127.0 dev ens3 proto kernel scope link src 192.168.127.1 
local 192.168.127.1 dev ens3 proto kernel scope host src 192.168.127.1 
broadcast 192.168.127.255 dev ens3 proto kernel scope link src 192.168.127.1 
root@vm1:~#

This table includes all of the local lookups. That is – routes for directly connected or local interfaces. You’ll note a lack of any routes for prefixes or interfaces that exist in either of our VRFs. Makes sense – VRFs are a layer 3 construct so we wouldn’t want routes , even local ones, for the VRF in a table they don’t belong. The catch here though is how this rule processing happens. We have a total of 4 rules…

Rule 0 – Local Lookups
Rule 1000 – L3MDEV (Our magic VRF lookup rule)
Rule 32766 – The main table lookup (default table)
Rule 32767 – The “default” table lookup

Rules are processed in order lower to higher. So any time we need to do a route lookup – this processing happens until we find a match regardless of where the lookup is happening. This is why it’s important that the kernel creates rule 1000 to do VRF lookups before we do a lookup in the main table. Some of you might already be seeing a problem with this – but let’s table that for now while we talk about how rule 1000 works.

So what would happen if rule 1000 wasn’t there? Currently if we do a ping in our VRF out of one our VRF interfaces this works…

root@vm1:~# ping -I vrf-1 192.168.10.254
ping: Warning: source address might be selected on device other than: vrf-1
PING 192.168.10.254 (192.168.10.254) from 192.168.10.1 vrf-1: 56(84) bytes of data.
64 bytes from 192.168.10.254: icmp_seq=1 ttl=64 time=0.437 ms
64 bytes from 192.168.10.254: icmp_seq=2 ttl=64 time=0.260 ms
64 bytes from 192.168.10.254: icmp_seq=3 ttl=64 time=0.186 ms

Now let’s delete rule 1000 and try again…

root@vm1:~# ip rule del pref 1000
root@vm1:~# ping -I vrf-1 192.168.10.254 -c 2
ping: Warning: source address might be selected on device other than: vrf-1
PING 192.168.10.254 (192.168.10.254) from 192.168.127.1 vrf-1: 56(84) bytes of data.

--- 192.168.10.254 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1017ms

root@vm1:~#

No dice. So let’s do a route lookup in the VRF and see what’s happening…

root@vm1:~# ip route get vrf vrf-1 192.168.10.254
192.168.10.254 via 192.168.127.100 dev ens3 src 192.168.127.1 uid 0 
    cache 
root@vm1:~# ip route
default via 192.168.127.100 dev ens3 proto static 
10.20.30.0/24 via 192.168.127.100 dev ens3 proto static 
192.168.127.0/24 dev ens3 proto kernel scope link src 192.168.127.1 
root@vm1:~#

So as expected – the route lookup is falling into the main table and coming back with a route that isn’t in our VRF. Rule 1000, the l3mdev lookup rule, solved this for us by telling the kernel to always do VRF route table lookups for us. However – rule 1000 is strictly a convenience that covers all of the VRF table lookups for us. If we wanted to we could manually add the rules for a given VRF as well…

root@vm1:~# ip rule add oif vrf-1 table 1
root@vm1:~# ip rule add iif vrf-1 table 1
root@vm1:~# ip route get vrf vrf-1 192.168.10.254
192.168.10.254 dev ens6 table 1 src 192.168.10.1 uid 0 
    cache 
root@vm1:~# ping -I vrf-1 192.168.10.254 -c 2
ping: Warning: source address might be selected on device other than: vrf-1
PING 192.168.10.254 (192.168.10.254) from 192.168.10.1 vrf-1: 56(84) bytes of data.
64 bytes from 192.168.10.254: icmp_seq=1 ttl=64 time=0.447 ms
64 bytes from 192.168.10.254: icmp_seq=2 ttl=64 time=0.275 ms

--- 192.168.10.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.275/0.361/0.447/0.086 ms
root@vm1:~#

Above we add two rules that say “if out of interface vrf1 then lookup in table 1″ as well as “if in interface vrf-1 then lookup in table 1″. So while you could certainly add these rules for each VRF you create it’s easier to just let the default l3mdev rule take care of this for you. When you have a rule that points the lookup to l3mdev-table it covers the bases for all of the l3mdev (VRF interfaces). So let’s go ahead and put that rule back and clean up our other two rules…

root@vm1:~# ip rule show
0:	from all lookup local
32764:	from all iif vrf-1 lookup 1
32765:	from all oif vrf-1 lookup 1
32766:	from all lookup main
32767:	from all lookup default
root@vm1:~# ip rule del pref 32764
root@vm1:~# ip rule del pref 32765
root@vm1:~# ip rule add l3mdev pref 1000
root@vm1:~# ip rule show
0:	from all lookup local
1000:	from all lookup [l3mdev-table]
32766:	from all lookup main
32767:	from all lookup default
root@vm1:~#

Now this all seems to be working as expected – but let’s get back to the possible issues I hinted at earlier. As I mentioned, the rule lookup happens from lower to higher preference. So the first rule that will be evaluated will be rule 0. Remember – rule 0 is our table for all of the local or directly connected interfaces. So what happens when our VRF has a route it’s trying to reach which is also directly connected say in the global table?…

root@vm1:~# ping -I vrf-1 192.168.10.254 -c 2
ping: Warning: source address might be selected on device other than: vrf-1
PING 192.168.10.254 (192.168.10.254) from 192.168.10.1 vrf-1: 56(84) bytes of data.
64 bytes from 192.168.10.254: icmp_seq=1 ttl=64 time=0.368 ms
64 bytes from 192.168.10.254: icmp_seq=2 ttl=64 time=0.322 ms

--- 192.168.10.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1031ms
rtt min/avg/max/mdev = 0.322/0.345/0.368/0.023 ms
root@vm1:~# ip addr add 192.168.10.254/32 dev lo
root@vm1:~# ping -I vrf-1 192.168.10.254 -c 2
ping: Warning: source address might be selected on device other than: vrf-1
PING 192.168.10.254 (192.168.10.254) from 192.168.10.254 vrf-1: 56(84) bytes of data.

--- 192.168.10.254 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1030ms
root@vm1:~#

No worky. And if we check the routes…

root@vm1:~# ip route get vrf vrf-1 192.168.10.254
local 192.168.10.254 dev lo table local src 192.168.10.254 uid 0 
    cache  
root@vm1:~#

As expected, the route lookup is getting caught in the table local lookup and not making it’s way to the l3mdev rule. To fix this – it’s usually recommended to reorder the rules like so…

root@vm1:~# ip -4 rule add pref 32765 table local
root@vm1:~# ip -4 rule del pref 0
root@vm1:~# ip rule show
1000:	from all lookup [l3mdev-table]
32765:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default
root@vm1:~# 
root@vm1:~# ping -I vrf-1 192.168.10.254 -c 2
ping: Warning: source address might be selected on device other than: vrf-1
PING 192.168.10.254 (192.168.10.254) from 192.168.10.1 vrf-1: 56(84) bytes of data.
64 bytes from 192.168.10.254: icmp_seq=1 ttl=64 time=0.564 ms
64 bytes from 192.168.10.254: icmp_seq=2 ttl=64 time=0.293 ms

--- 192.168.10.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1030ms
rtt min/avg/max/mdev = 0.293/0.428/0.564/0.135 ms
root@vm1:~#

This moves the local lookup to after rule 1000 so the VRF lookups can always occur first. Problem solved!

Now that we’ve gone through how they work, let’s talk about a couple of extra sort of handy commands to use when working with VRFs…

root@vm1:~# ip -br addr show master vrf-1
ens6             UP             192.168.10.1/24 fe80::50ab:54ff:fecd:101/64 
root@vm1:~#

If you want to see all of the IP interfaces in a given VRF you can use the show master version of the ip addr command to just list the addresses for a given VRF. Likewise to see all of the links in a given VRF you can use the same syntax with the ip link command…

root@vm1:~# ip link show master vrf-1
3: ens6:  mtu 1500 qdisc mq master vrf-1 state UP mode DEFAULT group default qlen 1000
    link/ether 52:ab:54:cd:01:01 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
root@vm1:~#

This should give you a good idea on how to get up and running with Linux VRFs. If nothing else, it should be a good reminder as to why VRFs are valuable. They provide a super lightweight construct for L3 isolation while still allowing commons services to run at other layers which helps keep the overhead down.

Python Pieces: Decorators

Jon Langemak — Sun, 01 Nov 2020 15:11:29 +0000

As some of you know – Im a big believer that we all learn differently. You may read something the first time and immediately grasp the topic whereas I may read it and miss the point entirely. For me, decorators have been one of those things that I felt like I was always close to understanding but still not quite getting it. Sure – some of the examples I read made sense but then I’d find another one that didn’t. In my quest to understand them, I spent a lot of time reviewing a lot of examples and asking a lot of very patient friends for help. At this point, I feel like I know enough to try and explain the topic in a manner that might hopefully help someone else who was having a hard time with the concept. With my learning philosophy out of the way, let’s jump right in….

I want to jump right into a real (albeit not super useful) example of decorators using the full decorator (or shorthand) syntax. Let’s start with this…

def a_decorator(a_function):
    print("You've been decorated!")
    return a_function

@a_decorator
def print_name_string(your_name):
    name_string = "Your name is: " + your_name
    return name_string

print(print_name_string("Jon"))

The code has two functions – a_decorator and print_name_string. The @a_decorator syntax directly above the print_name_string definition decorates that function with the function a_decorator (doing it this way is what I was referring to as the shorthand syntax). Lastly – we call our decorated function. Before we look at the output – let’s define what a decorator is. From all my reading the definition I like the most comes from this article where they define a decorator as…

A function that takes another function as an argument, generates a new function, augmenting the work of the original function, and returning the generated function so we can use it anywhere

With that said – and knowing that we are decorating the print_name_string function – any guesses as to what the output might be?

You've been decorated!
Your name is: Jon

That perhaps wasn’t hard to guess. If we apply our definition to this exact code example it might say…

a_decorator takes the print_name_string function as an argument, generates a new function, augmenting the work of print_name_string, and returns the generated function so we can use it anywhere

That said, you may look at the code above and think to yourself that when we call print_name_string Python sees the decorator and then runs the decorator function. The output we’re seeing certainly supports this. But now – let’s remove the print statement and try again…

def a_decorator(a_function):
    print("You've been decorated!")
    return a_function

@a_decorator
def print_name_string(your_name):
    name_string = "Your name is: " + your_name
    return name_string

#print(print_name_string("Jon"))

Our output is now….

You've been decorated!

Well – that’s discouraging. Why is the decorator function being called when we aren’t calling a function that’s been decorated with it? The answer is that decorators are rendered at runtime. The instant that the Python interpreter reads lines 5-8 above it runs them. So what does “run” mean in this context? I can boil it down to say that the new function based on the decorator is being created the instant the code is processed. This is not unlike any other function we create. If you’re in the Python CLI you have to create a function before you can call it. When the Python file is processed by the interpreter it’s done so top down. In our case, the interpreter sees the function is decorated and does the work to create a decorated function at that time so later in the code we can call it. So let’s look a little harder at what the decorator does…

def a_decorator(a_function):
    print("You've been decorated!")
    return a_function

It looks like it takes in a variable, does a print, and then returns the same variable. Based on the naming, we can discern that the variable being taken in, as well as returned, is a function. In this case a_function is the function that was being decorated (print_name_string). All that our decorator is doing is returning the same function. Seems like a waste returning the same function doesn’t it? That said – the decorator does need to return a function otherwise this will all fall apart. For instance…

def a_decorator(a_function):
    print("You've been decorated!")
    #return a_function

@a_decorator
def print_name_string(your_name):
    name_string = "Your name is: " + your_name
    return name_string

print(print_name_string("Jon"))

Commenting out the decorator function return gives us this…

You've been decorated!
Traceback (most recent call last):
  File "test.py", line 10, in 
    print(print_name_string("Jon"))
TypeError: 'NoneType' object is not callable

The message is pretty clear – but since our decorator is returning nothing (None) we can’t actually call the function since None is not callable. Again – it’s important to think of the decorator as replacing or augmenting the decorated function. Having the decorator not return a usable function is sort of like taking your car for an oil change and finding out afterwards your car is gone (Im surprised that’s the best example I can come up with). We pass the decorator a function, allow it to do some work, but in the end we expect it to return us a usable function. So it seems we have a hard rule defined here – decorators must return a function. But what function can they return if it’s not the one it is inherently passed as part of the decoration? Well we can create sub functions of our decorator function. A simple example might look like this…

def a_decorator(a_function):
    def add_some_flair(a_name):
        return "******* " + a_name + " *******"
    return add_some_flair

@a_decorator
def print_name_string(your_name):
    name_string = "Your name is: " + your_name
    return name_string

print(print_name_string("Jon"))

Here we add a sub-function to a_decorator called add_some_flair. This function adds a series of asterisks around the variable it receives and returns that string. Any ideas what the output of this might look like?

******* Jon *******

This is a good example of a decorator replacing the functionality of an existing function. Notice that our output doesn’t list the “Your name is: Jon” line. In this case, we decorated the function print_name_string but the decorator itself decided to return a function that does something entirely different. Again – this seems like it has little use if we are just fully overriding the functions that we are decorating. But it does go to show you that the decorator has the ability to completely override the initial function call if it wants to. We can prove that we can still get to the decorated function just by calling it as part of the add_some_flair function…

def a_decorator(a_function):
    def add_some_flair(a_name):
        print(a_function(a_name))
        return "******* " + a_name + " *******"
    return add_some_flair

@a_decorator
def print_name_string(your_name):
    name_string = "Your name is: " + your_name
    return name_string

print(print_name_string("Jon"))

Now our output would look like…

Your name is: Jon
******* Jon *******

You may have noticed that our function add_some_flair is taking in a single variable called a_name. This is not optional. Since we are decorating a function that takes in a single variable (your_name) our decorator return function must also take in a single variable. For instance, let’s try adding another variable to our print_string_name function…

def a_decorator(a_function):
    def add_some_flair(a_name):
        print(a_function(a_name))
        return "******* " + a_name + " *******"
    return add_some_flair

@a_decorator
def print_name_string(your_name, your_age):
    name_string = "Your name is: " + your_name
    return name_string

print(print_name_string("Jon"))

Now when we run this we get…

Traceback (most recent call last):
  File "test.py", line 12, in 
    print(print_name_string("Jon"))
  File "test.py", line 3, in add_some_flair
    print(a_function(a_name))
TypeError: print_name_string() missing 1 required positional argument: 'your_age'

The traceback points out our problem. We are trying to call the function print_your_name but aren’t giving it the right amount of variables. This is another case that we have to step back and think about this for a moment. When we call print_name_string what are we actually doing? Since it’s decorated with a_decorator which returns add_some_flair we are really calling add_some_flair. In doing so, we are passing it a variable a_name. If we wish to call the original function print_string_name and wish for it to have access to both your_name and your_age then we need to pass those to print_name_string which is really calling add_some_flair. You still with me? It’s important to think of the decorator being the main function being called. If you wish to call print_name_string again from within the decorator, you need to have the variables available to do so which means passing them into the decorator. Now – since we aren’t currently doing that, we can certainly make up a variable to meet the requirement…

def a_decorator(a_function):
    def add_some_flair(a_name):
        print(a_function(a_name,"bah"))
        return "******* " + a_name + " *******"
    return add_some_flair

@a_decorator
def print_name_string(your_name, your_age):
    name_string = "Your name is: " + your_name + " Your age is: " + your_age
    return name_string

print(print_name_string("Jon"))

In this case we are still only passing a_name to the add_some_flair function and when we go to call print_name_string(a_function) we simply pass it “bah” as the your_age parameter…

Your name is: Jon Your age is: bah
******* Jon *******

The right thing to do here though is to pass the variable through. If we just start passing a second variable on line 12 let’s see what happens…

def a_decorator(a_function):
    def add_some_flair(a_name):
        print(a_function(a_name,"bah"))
        return "******* " + a_name + " *******"
    return add_some_flair

@a_decorator
def print_name_string(your_name, your_age):
    name_string = "Your name is: " + your_name + " Your age is: " + your_age
    return name_string

print(print_name_string("Jon","99"))

Gives us…

Traceback (most recent call last):
  File "test.py", line 12, in 
    print(print_name_string("Jon","99"))
TypeError: add_some_flair() takes 1 positional argument but 2 were given

This output should help solidify the point I was trying to make above. We are now passing two variables to print_name_string and if we look at that function we see that it certainly does take two variables on line 8. However, since our decorator is replacing print_name_string with add_some_flair which does not take two arguments, the interpreter doesn’t know what to do. To fix this, we need to accept both arguments in the decorator return function add_some_flair. Once we do that we can start consuming it…

def a_decorator(a_function):
    def add_some_flair(a_name, a_age):
        print(a_function(a_name, a_age))
        return "******* " + a_name + " *******"
    return add_some_flair

@a_decorator
def print_name_string(your_name, your_age):
    name_string = "Your name is: " + your_name + " Your age is: " + your_age
    return name_string

print(print_name_string("Jon","99"))

Which gives us the output…

Your name is: Jon Your age is: 99
******* Jon *******

Now that we’ve talked about the basics – lets talk about the long hand way of decorating a function. We’ve already said that a decorator is a function that takes another function as an argument. That being said, instead of using the decorator shorthand syntax we could probably just do something like this…

def a_decorator(a_function):
    def add_some_flair(a_name, a_age):
        print(a_function(a_name, a_age))
        return "******* " + a_name + " *******"
    return add_some_flair

#@a_decorator
def print_name_string(your_name, your_age):
    name_string = "Your name is: " + your_name + " Your age is: " + your_age
    return name_string

decorated_print_name_string = a_decorator(print_name_string)
print(decorated_print_name_string("Jon","99"))

You can see on line 12 that we are creating a decorated function called decorated_print_name_string. So it’s a function that takes another function as an argument, possibly does some modification to it, and then returns the new function. This will work just like the shorthand syntax did…

Your name is: Jon Your age is: 99
******* Jon *******

Now – you might be wondering if we could shorten this up like this…

def a_decorator(a_function):
    def add_some_flair(a_name, a_age):
        print(a_function(a_name, a_age))
        return "******* " + a_name + " *******"
    return add_some_flair

#@a_decorator
def print_name_string(your_name, your_age):
    name_string = "Your name is: " + your_name + " Your age is: " + your_age
    return name_string

print(a_decorator(print_name_string("Jon","99")))

Unfortunately, running this provides us this output…

.add_some_flair at 0x102268430>

Why is that? By calling the function including the () we’re telling Python to execute that function. So in this case, we’re passing the return value of print_name_string("Jon") to the function a_decorator. We can see this by just making the same call ourselves…

def a_decorator(a_function):
    def add_some_flair(a_name, a_age):
        print(a_function(a_name, a_age))
        return "******* " + a_name + " *******"
    return add_some_flair

#@a_decorator
def print_name_string(your_name, your_age):
    name_string = "Your name is: " + your_name + " Your age is: " + your_age
    return name_string

print(a_decorator(print_name_string("Jon","99")))
print(a_decorator("Your name is: Jon Your age is: 99"))

The output is now…

.add_some_flair at 0x103f8a430>
.add_some_flair at 0x103f8a430>

The key here is that we aren’t so much worried about the function input at this point. In fact, we’re not interested in it at all when we create decorated functions. The goal of decorating a function is to create a new function. This is something that’s done totally outside of the realm of consuming the function (aka, passing variables to it and expecting a result) and is done only once when Python imports the script.

It probably makes sense at this point to provide a couple of examples of how decorators might be used in real life. Here’s a quick example of one you might use to time a given function call…

import time

def timing(a_function):
    def time_func_call(x_counter):
        print("DECORATOR -> Doing a loop for " + str(x_counter) + " iterations")
        start_time = time.perf_counter()
        a_function(x_counter)
        end_time = time.perf_counter()
        total_run_time = end_time - start_time
        print("DECORATOR -> Total time was: " + str(total_run_time))
    return time_func_call

@timing
def print_name_string(x_counter):
    print("The function beginith....")
    while x_counter > 0:
    	time.sleep(.1)
    	x_counter = x_counter - 1
    print("The function endith....")

print_name_string(10)

The output would look something like this…

DECORATOR -> Doing a loop for 10 iterations
The function beginith....
The function endith....
DECORATOR -> Total time was: 1.034948312

Ideally you’d use something besides print statements – but I think it’s enough to make the point. Decorators can be extremely useful and there are lots of Python projects that make great use of them. There are also lots of other problems to solve in regards to decorators but I’ll save those for another blog post. Im hoping this is enough to at least get folks to a base level understanding of how they work. As always – comments are welcome and appreciated!

Python Pieces: PyEnv and Venvs

Jon Langemak — Wed, 30 Sep 2020 13:03:35 +0000

In my last post, we talked about PyEnv and how it can help manage your local Python environments. As it turns out it can also help you manage virtual environments as well! However – pursuing this functionality took me down a rabbit hole that was a bit deeper than expected. The way that PyEnv works causes some behaviors (and on my end assumptions) to change which made me start questioning some of the things that I’ve always just taken for granted. In other words – prepare yourself to go down the rabbit hole with me.

At first glance PyEnv promised the same sort of awesome automagically context switching craziness that we saw previously work with Python versions. However – the virtual environment management implementation with PyEnv felt rather foreign (and maybe a little clunky?) to me. Most notably, as I pointed out in my last post, the .zshrc alias provided to make the auto activation piece work slows down my terminal immensely which is why I omitted using it. A slow terminal is about the worst thing I can think of…

That said – I still think it’s worth reviewing what it can offer so you can make your own decision. But before we dive into that – I want to point out something about PyEnv that I should have mentioned in my last post. As some folks have pointed out, PyEnv is just creating another layer of misdirection on your laptop. Admittedly it took me sometime to fully grasp the “shims” concept and understand what was actually going on. However – at least for me – I was happy to trade this misdirection for what looks like a very clean Python installation and management approach. Instead of having Python versions thrown about my filesystem and trying to figure out if I had the right $PATH to actually use them – PyEnv neatly installs all of the versions I want to use in one place. Discussing this now actually dovetails quite nicely into how PyEnv handles virtual environments so let me back track for a second and let’s look at what PyEnv is doing with Python versions…

user@users-MacBook-Pro ~ % pyenv versions                              
  system
  2.7.18
  3.5.1
* 3.8.5 (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro ~ % 
user@users-MacBook-Pro ~ % which python
/Users/user/.pyenv/shims/python
user@users-MacBook-Pro ~ % pyenv which python
/Users/user/.pyenv/versions/3.8.5/bin/python
user@users-MacBook-Pro ~ %

As we can see above – I have my system Python installed as well as 3 unique Python versions installed using PyEnv. If we use the standard which command we can see the shim as expected, but by using the pyenv which command we can see the real Python being used. Notice the file path – let’s take a look there…

user@users-MacBook-Pro ~ % cd /Users/user/.pyenv/versions
user@users-MacBook-Pro versions % ls
2.7.18	3.5.1	3.8.5
user@users-MacBook-Pro versions %

Let’s say we’ve finally decided we’re done with that 2.7.18 version and just want it off our laptop. How do we do that? Well PyEnv does have a handy uninstall command – but to prove my point let’s just delete that folder manually…

user@users-MacBook-Pro versions % rm -rf 2.7.18 
user@users-MacBook-Pro versions % pyenv versions
  system
  3.5.1
* 3.8.5 (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro versions %

Gonesies. This is perhaps one of the key features of PyEnv for me. It puts all of the things in one place. I no longer have to worry about lingering Python pieces laying around my filesystem causing issues. We can also prove the uninstall command works as well…

user@users-MacBook-Pro versions % pyenv uninstall 3.5.1
pyenv: remove /Users/user/.pyenv/versions/3.5.1? [y|N]y
pyenv: 3.5.1 uninstalled
user@users-MacBook-Pro versions % pyenv versions       
  system
* 3.8.5 (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro versions %

Alright – so now that we’ve shown how and where the PyEnv based Python installs actually live (and why I think that’s awesome) let’s talk about how PyEnv handles venvs.

Note1: As we start talking about PyEnv and virtual environments do keep in mind what we just saw and how PyEnv handles installs and versions. It will come in handy during the rest of this blog post.

Note2: I apparently can’t stop calling them different things so please keep in mind that a venv, a virtualenv, and a virtual environment are the same thing. Not familiar with what they are and why to use them? Check this out!

PyEnv has handy built in functions to create venvs but starting to use them felt, at least for me, a little bit weird. After understanding how PyEnv was handling the venvs the model did make more sense to me – but if you’re used to working with venvs in a more traditional sense you might find this approach clunky or at the very least uncomfortable. That said let’s dive right into creating one. First Im going to reinstall some Python versions so we have some stuff to play with…

pyenv install 2.7.18
pyenv install 3.5.1

Cool. Now let’s make some folders to house our potential projects…

user@users-MacBook-Pro ~ % mkdir venv_projects
user@users-MacBook-Pro ~ % cd venv_projects 
user@users-MacBook-Pro venv_projects % mkdir test_project_1
user@users-MacBook-Pro venv_projects % mkdir test_project_2

So let’s make a venv in test_project_1 with PyEnv…

user@users-MacBook-Pro venv_projects % cd test_project_1
user@users-MacBook-Pro test_project_1 % 
user@users-MacBook-Pro test_project_1 % pyenv virtualenv 2.7.18 test_project_1
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting virtualenv
  Using cached https://files.pythonhosted.org/packages/12/51/36c685ff2c1b2f7b4b5db29f3153159102ae0e0adaff3a26fd1448232e06/virtualenv-20.0.31-py2.py3-none-any.whl
Collecting appdirs<2,>=1.4.3 (from virtualenv)
  Using cached https://files.pythonhosted.org/packages/3b/00/2344469e2084fb287c2e0b57b72910309874c3245463acd6cf5e3db69324/appdirs-1.4.4-py2.py3-none-any.whl
Collecting importlib-metadata<2,>=0.12; python_version < "3.8" (from virtualenv)
  Using cached https://files.pythonhosted.org/packages/8e/58/cdea07eb51fc2b906db0968a94700866fc46249bdc75cac23f9d13168929/importlib_metadata-1.7.0-py2.py3-none-any.whl
Collecting six<2,>=1.9.0 (from virtualenv)
  Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting filelock<4,>=3.0.0 (from virtualenv)
  Using cached https://files.pythonhosted.org/packages/14/ec/6ee2168387ce0154632f856d5cc5592328e9cf93127c5c9aeca92c8c16cb/filelock-3.0.12.tar.gz
Collecting importlib-resources>=1.0; python_version < "3.7" (from virtualenv)
  Using cached https://files.pythonhosted.org/packages/ba/03/0f9595c0c2ef12590877f3c47e5f579759ce5caf817f8256d5dcbd8a1177/importlib_resources-3.0.0-py2.py3-none-any.whl
Collecting distlib<1,>=0.3.1 (from virtualenv)
  Using cached https://files.pythonhosted.org/packages/f5/0a/490fa011d699bb5a5f3a0cf57de82237f52a6db9d40f33c53b2736c9a1f9/distlib-0.3.1-py2.py3-none-any.whl
Collecting pathlib2<3,>=2.3.3; python_version < "3.4" and sys_platform != "win32" (from virtualenv)
  Using cached https://files.pythonhosted.org/packages/e9/45/9c82d3666af4ef9f221cbb954e1d77ddbb513faf552aea6df5f37f1a4859/pathlib2-2.3.5-py2.py3-none-any.whl
Collecting configparser>=3.5; python_version < "3" (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv)
  Using cached https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whl
Collecting contextlib2; python_version < "3" (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv)
  Using cached https://files.pythonhosted.org/packages/85/60/370352f7ef6aa96c52fb001831622f50f923c1d575427d021b8ab3311236/contextlib2-0.6.0.post1-py2.py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata<2,>=0.12; python_version < "3.8"->virtualenv)
  Using cached https://files.pythonhosted.org/packages/96/0a/67556e9b7782df7118c1f49bdc494da5e5e429c93aa77965f33e81287c8c/zipp-1.2.0-py2.py3-none-any.whl
Collecting typing; python_version < "3.5" (from importlib-resources>=1.0; python_version < "3.7"->virtualenv)
  Using cached https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl
Collecting singledispatch; python_version < "3.4" (from importlib-resources>=1.0; python_version < "3.7"->virtualenv)
  Using cached https://files.pythonhosted.org/packages/c5/10/369f50bcd4621b263927b0a1519987a04383d4a98fb10438042ad410cf88/singledispatch-3.4.0.3-py2.py3-none-any.whl
Collecting scandir; python_version < "3.5" (from pathlib2<3,>=2.3.3; python_version < "3.4" and sys_platform != "win32"->virtualenv)
  Using cached https://files.pythonhosted.org/packages/df/f5/9c052db7bd54d0cbf1bc0bb6554362bba1012d03e5888950a4f5c5dadc4e/scandir-1.10.0.tar.gz
Installing collected packages: appdirs, configparser, contextlib2, scandir, six, pathlib2, zipp, importlib-metadata, filelock, typing, singledispatch, importlib-resources, distlib, virtualenv
  Running setup.py install for scandir ... done
  Running setup.py install for filelock ... done
Successfully installed appdirs-1.4.4 configparser-4.0.2 contextlib2-0.6.0.post1 distlib-0.3.1 filelock-3.0.12 importlib-metadata-1.7.0 importlib-resources-3.0.0 pathlib2-2.3.5 scandir-1.10.0 singledispatch-3.4.0.3 six-1.15.0 typing-3.7.4.3 virtualenv-20.0.31 zipp-1.2.0
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
created virtual environment CPython2.7.18.final.0-64 in 592ms
  creator CPython2Posix(dest=/Users/user/.pyenv/versions/2.7.18/envs/test_project_1, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/Users/user/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.2, setuptools==44.1.1, wheel==0.35.1
  activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
Looking in links: /var/folders/sb/1xk4f6tj1wq20ngdjw1cwdwm0000gn/T/tmpBBtVKE
Requirement already satisfied: setuptools in /Users/user/.pyenv/versions/2.7.18/envs/test_project_1/lib/python2.7/site-packages (44.1.1)
Requirement already satisfied: pip in /Users/user/.pyenv/versions/2.7.18/envs/test_project_1/lib/python2.7/site-packages (20.2.2)
user@users-MacBook-Pro test_project_1 %

Alright – that was a lot of output – but it sure looks like it installed all the things we need. For those of you familiar with working with venvs more traditionally, you might think to yourself that now is the time to do a source test_project_1/bin/activate. So let’s see if thats possible…

user@users-MacBook-Pro test_project_1 % ls -al
total 0
drwxr-xr-x  2 user  staff   64 Sep 23 08:47 .
drwxr-xr-x  4 user  staff  128 Sep 23 08:48 ..
user@users-MacBook-Pro test_project_1 %

Hrmmm. Ok – well maybe there’s some other magic going on let’s see what Python we’re using…

pyenv which python
/Users/user/.pyenv/versions/3.8.5/bin/python
user@users-MacBook-Pro test_project_1 %

More hrmmm. So it’s still using our global version. So what’s going on here? This is where things get a little different. PyEnv seems to treat venvs as just another version of Python. And when you step back and think about how they’re accomplishing this – it makes sense given the model they’ve defined. All the same, it feels different and can definitely make you think you’re doing it wrong if you don’t understand how PyEnv works. So what is going on? Let’s go and take a look at our versions directory again…

user@users-MacBook-Pro test_project_1 % cd /Users/user/.pyenv/versions
user@users-MacBook-Pro versions % ls
2.7.18		3.5.1		3.8.5		test_project_1
user@users-MacBook-Pro versions %

Aha! That looks familiar. But why is that there? Sure looks to me like the virtual environment is being created as another PyEnv version. But if we look closer…

user@users-MacBook-Pro versions % ls -al    
total 0
drwxr-xr-x   6 user  staff  192 Sep 23 08:49 .
drwxr-xr-x  26 user  staff  832 Sep 22 10:20 ..
drwxr-xr-x   7 user  staff  224 Sep 23 08:49 2.7.18
drwxr-xr-x   6 user  staff  192 Sep 23 08:40 3.5.1
drwxr-xr-x   7 user  staff  224 Sep 22 10:20 3.8.5
lrwxr-xr-x   1 user  staff   54 Sep 23 08:49 test_project_1 -> /Users/user/.pyenv/versions/2.7.18/envs/test_project_1
user@users-MacBook-Pro versions %

AHA! Symlinks! So that file is just a symlink pointing to /Users/user/.pyenv/versions/2.7.18/envs/test_project_1. Let’s look in there…

user@users-MacBook-Pro versions % cd 2.7.18 
user@users-MacBook-Pro 2.7.18 % ls
bin	envs	include	lib	share
user@users-MacBook-Pro 2.7.18 % cd envs
user@users-MacBook-Pro envs % ls
test_project_1
user@users-MacBook-Pro envs % cd test_project_1 
user@users-MacBook-Pro test_project_1 % ls
bin		include		lib		pyvenv.cfg
user@users-MacBook-Pro test_project_1 %

Interesting. So each venv looks like a discrete version of Python but is really created underneath the base version you specify. So in our case, since the venv we created uses 2.7.18 the files exist under /Users/user/.pyenv/versions/2.7.18/envs/. This is starting to look more like the setup of a “normal” virtualenv. Let’s dive into the bin directory and see what is there…

user@users-MacBook-Pro test_project_1 % cd bin
user@users-MacBook-Pro bin % ls -al
total 4248
drwxr-xr-x  23 user  staff      736 Sep 23 08:49 .
drwxr-xr-x   7 user  staff      224 Sep 23 08:49 ..
-rw-r--r--   1 user  staff     2243 Sep 23 08:49 activate
-rw-r--r--   1 user  staff     1462 Sep 23 08:49 activate.csh
-rw-r--r--   1 user  staff     3093 Sep 23 08:49 activate.fish
-rw-r--r--   1 user  staff     1751 Sep 23 08:49 activate.ps1
-rw-r--r--   1 user  staff     1199 Sep 23 08:49 activate_this.py
-rwxr-xr-x   1 user  staff      279 Sep 23 08:49 easy_install
-rwxr-xr-x   1 user  staff      279 Sep 23 08:49 easy_install-2.7
-rwxr-xr-x   1 user  staff      279 Sep 23 08:49 easy_install2
-rwxr-xr-x   1 user  staff      279 Sep 23 08:49 easy_install2.7
-rwxr-xr-x   1 user  staff      270 Sep 23 08:49 pip
-rwxr-xr-x   1 user  staff      270 Sep 23 08:49 pip-2.7
-rwxr-xr-x   1 user  staff      270 Sep 23 08:49 pip2
-rwxr-xr-x   1 user  staff      270 Sep 23 08:49 pip2.7
-rwxr-xr-x   1 user  staff      126 Sep 23 08:49 pydoc
-rwxr-xr-x   1 user  staff  2100208 Sep 23 08:49 python
lrwxr-xr-x   1 user  staff        6 Sep 23 08:49 python2 -> python
lrwxr-xr-x   1 user  staff        6 Sep 23 08:49 python2.7 -> python
-rwxr-xr-x   1 user  staff      257 Sep 23 08:49 wheel
-rwxr-xr-x   1 user  staff      257 Sep 23 08:49 wheel-2.7
-rwxr-xr-x   1 user  staff      257 Sep 23 08:49 wheel2
-rwxr-xr-x   1 user  staff      257 Sep 23 08:49 wheel2.7
user@users-MacBook-Pro bin %

Here we can see all of the python executables we’d expect to see for a given environment. So if we step back and think about the folder structure, and how PyEnv works, any guesses as to how we can leverage this venv? To do that, let’s go back to project directory and run this command…

user@users-MacBook-Pro test_project_1 % pyenv local test_project_1
user@users-MacBook-Pro test_project_1 % ls -al
total 8
drwxr-xr-x  3 user  staff   96 Sep 23 10:39 .
drwxr-xr-x  4 user  staff  128 Sep 23 08:48 ..
-rw-r--r--  1 user  staff   15 Sep 23 10:39 .python-version
user@users-MacBook-Pro test_project_1 % more .python-version 
test_project_1
user@users-MacBook-Pro test_project_1 %

The first thing we need to do is set the local Python version. To do that we just say that the “local” Python version is the one in our venv. So let’s see how this looks…

user@users-MacBook-Pro test_project_1 % pyenv which python
/Users/user/.pyenv/versions/test_project_1/bin/python
user@users-MacBook-Pro test_project_1 %

Nice! So if you followed the setup from our last post and have put the following lines in your .zshrc file…

export PATH="/Users/user/.pyenv/bin:$PATH"
eval "$(pyenv init - zsh)"

Note: We are specifically omitting the eval "$(pyenv virtualenv-init - zsh)" line. We do NOT need that.

You should now be automagically using the correct version of Python…

user@users-MacBook-Pro test_project_1 % pyenv which python
/Users/user/.pyenv/versions/test_project_1/bin/python
user@users-MacBook-Pro test_project_1 % python 
Python 2.7.18 (default, Sep 23 2020, 08:43:08) 
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

We should now be seeing the same automagic behavior we saw in the last post. Move out of this directory and we’ll be back to the global version…

user@users-MacBook-Pro test_project_1 % cd ..
user@users-MacBook-Pro venv_projects % pyenv which python
/Users/user/.pyenv/versions/3.8.5/bin/python
user@users-MacBook-Pro venv_projects %

Nice. But here is where I started questioning myself. By using the base functionality we established in the last post, we are already getting automatic switching to the version of Python that we want. If we think about virtual environments at a high level I think of them as providing two big things. The first is a specific version of Python and the second is a discrete set of Python packages managed typically by pip. It appears we’re already getting the first and we haven’t even had to “activate” our venv. Let’s test out the second…

user@users-MacBook-Pro test_project_1 % pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version
---------- -------
pip        20.2.2
setuptools 44.1.1
wheel      0.35.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_1/bin/python -m pip install --upgrade pip' command.
user@users-MacBook-Pro test_project_1 % 
user@users-MacBook-Pro test_project_1 % 
user@users-MacBook-Pro test_project_1 % pip install pyyaml
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Processing /Users/user/Library/Caches/pip/wheels/d1/d5/a0/3c27cdc8b0209c5fc1385afeee936cf8a71e13d885388b4be2/PyYAML-5.3.1-cp27-cp27m-macosx_10_15_x86_64.whl
Installing collected packages: pyyaml
Successfully installed pyyaml-5.3.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_1/bin/python -m pip install --upgrade pip' command.
user@users-MacBook-Pro test_project_1 % pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version
---------- -------
pip        20.2.2
PyYAML     5.3.1
setuptools 44.1.1
wheel      0.35.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_1/bin/python -m pip install --upgrade pip' command.
user@users-MacBook-Pro test_project_1 % cd ..
user@users-MacBook-Pro venv_projects % pip list
Package    Version
---------- ---------
certifi    2020.6.20
chardet    3.0.4
idna       2.10
pip        20.1.1
requests   2.24.0
setuptools 47.1.0
urllib3    1.25.10
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/3.8.5/bin/python3.8 -m pip install --upgrade pip' command.
user@users-MacBook-Pro venv_projects %

There’s a lot of output above – but the basic gist of what Im doing is installing pyyaml using pip from the test_project_1 directory. I then back up one directory and validate that I no longer see that package because I no longer have a local version of Python and am instead defaulting to the global Python (in other words I just switched versions of Python using PyEnv). To say this slightly differently, package management with PyEnv appears to be per “version”. Make sense? We saw some of this behavior previously. Recall at the end of the last post if we had two project folders both of which used the same local version of Python we inherited the packages between the two. Since my “global” version is different from my “local” version I no longer see the package I just installed. Now throw venvs on top of this with the understanding that each venv is also treated as a PyEnv “version”. If we create another virtual environment with PyEnv using the same version and set it as the local Python version we do NOT inherit the packages. For instance…

user@users-MacBook-Pro venv_projects % cd test_project_2
user@users-MacBook-Pro test_project_2 % pyenv virtualenv 2.7.18 test_project_2
created virtual environment CPython2.7.18.final.0-64 in 389ms
  creator CPython2Posix(dest=/Users/user/.pyenv/versions/2.7.18/envs/test_project_2, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/Users/user/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.2, setuptools==44.1.1, wheel==0.35.1
  activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
Looking in links: /var/folders/sb/1xk4f6tj1wq20ngdjw1cwdwm0000gn/T/tmpredLoM
Requirement already satisfied: setuptools in /Users/user/.pyenv/versions/2.7.18/envs/test_project_2/lib/python2.7/site-packages (44.1.1)
Requirement already satisfied: pip in /Users/user/.pyenv/versions/2.7.18/envs/test_project_2/lib/python2.7/site-packages (20.2.2)
user@users-MacBook-Pro test_project_2 % pyenv local test_project_2            
user@users-MacBook-Pro test_project_2 % pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version
---------- -------
pip        20.2.2
setuptools 44.1.1
wheel      0.35.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_2/bin/python -m pip install --upgrade pip' command.
user@users-MacBook-Pro test_project_2 %

So at this point – I started questioning things. When would I ever need to actually “activate” any of these venvs? Once you grasp how PyEnv does the version switching, how versions are handled, and how PyEnv treats virtual environments as versions, you can start to see why you might not need to “activate” a virtual environment to get the same kind of behavior. If you’re still struggling with what Im getting at – this helped me visualize things (note some folders are removed to make this not huge)…

user@users-MacBook-Pro versions % tree -L 6 -d
.
├── 2.7.18
│   ├── bin
│   ├── envs
│   │   ├── test_project_1
│   │   │   ├── bin
│   │   │   ├── include
│   │   │   │   └── python2.7 -> /Users/user/.pyenv/versions/2.7.18/include/python2.7
│   │   │   └── lib
│   │   │       └── python2.7
│   │   │           ├── config
│   │   │           ├── lib-dynload -> /Users/user/.pyenv/versions/2.7.18/lib/python2.7/lib-dynload
│   │   │           └── site-packages
│   │   └── test_project_2
│   │       ├── bin
│   │       ├── include
│   │       │   └── python2.7 -> /Users/user/.pyenv/versions/2.7.18/include/python2.7
│   │       └── lib
│   │           └── python2.7
│   │               ├── config
│   │               ├── lib-dynload -> /Users/user/.pyenv/versions/2.7.18/lib/python2.7/lib-dynload
│   │               └── site-packages
│   ├── include
│   │   └── python2.7
│   ├── lib
│   │   ├── pkgconfig
│   │   └── python2.7
│   │       ├── bsddb
│   │       ├── compiler
│   │       ├── config
│   │       ├── ctypes
│   │       ├── curses
│   │       ├── distutils
│   │       ├── email
│   │       ├── encodings
│   │       ├── ensurepip
│   │       ├── hotshot
│   │       ├── idlelib
│   │       ├── importlib
│   │       ├── json
│   │       ├── lib-dynload
│   │       ├── lib-tk
│   │       ├── lib2to3
│   │       ├── logging
│   │       ├── multiprocessing
│   │       ├── plat-darwin
│   │       ├── plat-mac
│   │       ├── pydoc_data
│   │       ├── site-packages
│   │       ├── sqlite3
│   │       ├── test
│   │       ├── unittest
│   │       ├── wsgiref
│   │       └── xml
│   └── share
│       └── man
│           └── man1
├── 3.5.1
│   ├── bin
│   ├── envs
│   │   └── test_project_3
│   │       ├── bin
│   │       ├── include
│   │       └── lib
│   │           └── python3.5
│   │               └── site-packages
│   ├── include
│   │   └── python3.5m
│   ├── lib
│   │   ├── pkgconfig
│   │   └── python3.5
│   │       ├── __pycache__
│   │       ├── asyncio
│   │       ├── collections
│   │       ├── concurrent
│   │       ├── config-3.5m
│   │       ├── ctypes
│   │       ├── curses
│   │       ├── dbm
│   │       ├── distutils
│   │       ├── email
│   │       ├── encodings
│   │       ├── ensurepip
│   │       ├── html
│   │       ├── http
│   │       ├── idlelib
│   │       ├── importlib
│   │       ├── json
│   │       ├── lib-dynload
│   │       ├── lib2to3
│   │       ├── logging
│   │       ├── multiprocessing
│   │       ├── plat-darwin
│   │       ├── pydoc_data
│   │       ├── site-packages
│   │       ├── sqlite3
│   │       ├── test
│   │       ├── tkinter
│   │       ├── turtledemo
│   │       ├── unittest
│   │       ├── urllib
│   │       ├── venv
│   │       ├── wsgiref
│   │       ├── xml
│   │       └── xmlrpc
│   └── share
│       └── man
│           └── man1
├── 3.8.5
│   ├── bin
│   ├── envs
│   ├── include
│   │   └── python3.8
│   │       ├── cpython
│   │       └── internal
│   ├── lib
│   │   ├── pkgconfig
│   │   └── python3.8
│   │       ├── __pycache__
│   │       ├── asyncio
│   │       ├── collections
│   │       ├── concurrent
│   │       ├── config-3.8-darwin
│   │       ├── ctypes
│   │       ├── curses
│   │       ├── dbm
│   │       ├── distutils
│   │       ├── email
│   │       ├── encodings
│   │       ├── ensurepip
│   │       ├── html
│   │       ├── http
│   │       ├── idlelib
│   │       ├── importlib
│   │       ├── json
│   │       ├── lib-dynload
│   │       ├── lib2to3
│   │       ├── logging
│   │       ├── multiprocessing
│   │       ├── pydoc_data
│   │       ├── site-packages
│   │       ├── sqlite3
│   │       ├── test
│   │       ├── tkinter
│   │       ├── turtledemo
│   │       ├── unittest
│   │       ├── urllib
│   │       ├── venv
│   │       ├── wsgiref
│   │       ├── xml
│   │       └── xmlrpc
│   └── share
│       └── man
│           └── man1
├── test_project_1 -> /Users/user/.pyenv/versions/2.7.18/envs/test_project_1
├── test_project_2 -> /Users/user/.pyenv/versions/2.7.18/envs/test_project_2
└── test_project_3 -> /Users/user/.pyenv/versions/3.5.1/envs/test_project_3

555 directories
user@users-MacBook-Pro versions %

As you can see – each “version” has it’s own site-packages directory. At this point, it seems clear to me that each “version” (I keep putting it in quotes since venvs are clearly treated as versions) has it’s own package directory. So it seems pretty clear to me that without doing an activation I am getting a virtual environment like experience.

I’ll remind folks at this point that I am NOT using the automatic venv activation scripts that PyEnv suggests you use ($(pyenv virtualenv-init - zsh)).

What’s left at this point is to figure out how each version of Python is being instructed to use it’s local site-packages directory. After much digging, I found that it really came down to the site Python module. Turns out you can take a look at what’s there pretty easily…

user@users-MacBook-Pro test_project_1 % python -m site 
sys.path = [
    '/Users/user/venv_projects/test_project_1',
    '/Users/user/.pyenv/versions/2.7.18/lib/python27.zip',
    '/Users/user/.pyenv/versions/2.7.18/lib/python2.7',
    '/Users/user/.pyenv/versions/2.7.18/lib/python2.7/plat-darwin',
    '/Users/user/.pyenv/versions/2.7.18/lib/python2.7/plat-mac',
    '/Users/user/.pyenv/versions/2.7.18/lib/python2.7/plat-mac/lib-scriptpackages',
    '/Users/user/.pyenv/versions/2.7.18/lib/python2.7/lib-tk',
    '/Users/user/.pyenv/versions/2.7.18/lib/python2.7/lib-old',
    '/Users/user/.pyenv/versions/2.7.18/lib/python2.7/lib-dynload',
    '/Users/user/.pyenv/versions/test_project_1/lib/python2.7/site-packages',
]
USER_BASE: '/Users/user/.local' (doesn't exist)
USER_SITE: '/Users/user/.local/lib/python2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: False
user@users-MacBook-Pro test_project_1 %

The really interesting output is the sys.path list. This list is used to search for Python packages and modules and is evaluated in order. Looking at the first entry we can see this makes sense. If you have other Python packages locally it will first search there. Then we have lines 4-11 which from what I can discern are created by the sites module based on platform and other information. But it’s the last line that really interested me. This line specifies the “global” site packages directory. Which you can find yourself by running this command…

python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

which I found here. If we run that in our project folder we’ll see that we get that last line there…

user@users-MacBook-Pro test_project_1 % python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
/Users/user/.pyenv/versions/test_project_1/lib/python2.7/site-packages
user@users-MacBook-Pro test_project_1 %

So this “global sites-packages” folder is where our Python packages are for a given project. Notice that this value changes as we switch versions of Python…

user@users-MacBook-Pro test_project_1 % cd ..
user@users-MacBook-Pro venv_projects % python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
/Users/user/.pyenv/versions/3.8.5/lib/python3.8/site-packages
user@users-MacBook-Pro venv_projects % mkdir test_project_3
user@users-MacBook-Pro venv_projects % cd test_project_3
user@users-MacBook-Pro test_project_3 % pyenv local 3.5.1
user@users-MacBook-Pro test_project_3 % python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
/Users/user/.pyenv/versions/3.5.1/lib/python3.5/site-packages
user@users-MacBook-Pro test_project_3 % pyenv virtualenv 3.5.1 test_project_3
Ignoring indexes: https://pypi.python.org/simple
Requirement already satisfied (use --upgrade to upgrade): setuptools in /Users/user/.pyenv/versions/3.5.1/envs/test_project_3/lib/python3.5/site-packages
Requirement already satisfied (use --upgrade to upgrade): pip in /Users/user/.pyenv/versions/3.5.1/envs/test_project_3/lib/python3.5/site-packages
user@users-MacBook-Pro test_project_3 % pyenv local test_project_3
user@users-MacBook-Pro test_project_3 % python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
/Users/user/.pyenv/versions/test_project_3/lib/python3.5/site-packages
user@users-MacBook-Pro test_project_3 %

On line 3 we see that since we’re out of the project directory we are once again using the global version we have set of 3.8.5. Now we make a new project directory called test_project_3 and inside of it we set the local version to 3.5.1. When we check again on line 8 – we see that we’re using the site packages folder that’s part of the base 3.5.1 version directory. Then we create a venv with PyEnv and set that as the local Python “version”. In doing so we can see on line 15, we are now using that site-packages folder.

So at this point – I think we have close to a full picture of how this is working. The bulk of the magic is in how PyEnv manages versions and the actual package management piece is a function of Python and just works the way that it does because of the PyEnv folder structure. My hypothesis is that by using PyEnv version management (and the shim model) you’re getting “virtual environment like” behavior without having to actually “activate” a virtual environment.

Now that said – there are some limitations to this. While discussing this on the Network to Code slack Jacob McGill (thank you for the comments!) pointed out one limitation I hadn’t considered. My model of laying out projects fit wells into PyEnv since I am typically working on files for a given project in that given project folder. But without “activating” the virtual environment, you HAVE to be in the project directory in order for any of this to work. In other words, once I leave the project folder (for instance test_project_1) I lose all of this. Now in my current work flow, this isn’t a problem since I tightly couple the project folder to the work Im doing. But I can see this being a drawback for others. As an example….

user@users-MacBook-Pro test_project_1 % pyenv activate test_project_1
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
(test_project_1) user@users-MacBook-Pro test_project_1 % 
(test_project_1) user@users-MacBook-Pro test_project_1 % cd ..
(test_project_1) user@users-MacBook-Pro venv_projects % pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version
---------- -------
pip        20.2.2
PyYAML     5.3.1
setuptools 44.1.1
wheel      0.35.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_1/bin/python -m pip install --upgrade pip' command. 
(test_project_1) user@users-MacBook-Pro venv_projects % cd test_project_3
(test_project_1) user@users-MacBook-Pro test_project_3 % pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version
---------- -------
pip        20.2.2
PyYAML     5.3.1
setuptools 44.1.1
wheel      0.35.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/2.7.18/envs/test_project_1/bin/python -m pip install --upgrade pip' command.
(test_project_1) user@users-MacBook-Pro test_project_3 %

You can see that if I do “activate” the venv then I can move out of the project folder and still have my Python version and packages that I expect. I can even move into another project folder and I’m still using the activated virtual environment.

This all said – I would love to hear from Python/PyEnv experts out there on if my hypothesis is true. Please comment and let me know your thoughts! At this point, if I do end up using PyEnv for virtual environment management, I don’t think I’ll be doing any virtual environment activations (unless someone can point out a good reason to do so!).

Python Pieces: Using PyEnv

Jon Langemak — Sat, 12 Sep 2020 14:49:31 +0000

If you’re like me – one of the most frustrating things about Python is version management. You get a new Mac, the system default is 2.x something, you need 3.x something, and you’re wondering what the best (right) way to get the version you want installed. You install Python 3 but the default Python version stays the same until you do some symlink hack thing that you know is just creating a mess. So for awhile you just call python3 explicitly but then you realize that all of the packages you installed using pip are no longer available and you need to install them again using pip3.

Sound familiar? Maybe I’m the only one that struggles with this – but I tend to muddle my way through just making things work while in the back of my head I know that Im creating a complete disaster of the local Python installation. I shall muddle no longer thanks to PyEnv. I was recently introduced to the tool and it’s a total game changer. It allows you to seemlessly manage your local Python install, easily install different versions, easily switch versions, and even has the capability of automgically switching versions for you based on the project you’re currently working on. So – let’s dive right into how we make this all work.

First things first – we need to install PyEnv. To do that – hit up the PyEnv-Installer page. You can also clone the PyEnv repo and do things that way if you’re uncomfortable with piping randomness into bash. For the sake of simplicity though, we’ll use the installer version which has you run this command…

curl https://pyenv.run | bash

The install should be pretty straight forward but you’ll likely hit this warning message at the end of the install…

WARNING: seems you still have not added 'pyenv' to the load path.

# Load pyenv automatically by adding
# the following to ~/.bashrc:

export PATH="/Users/user/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
```

Note: I’ve run into issues with my terminal responding super slowly if I include the last line in my .zshrc. For now, I’ve omitted eval "$(pyenv virtualenv-init -)" from the .zshrc. We’ll talk more about that later in my next post where we talk about PyEnv and virtual environments. It’s not needed for this post.

As the message describes – we need to do add PyEnv to the load path. We’ll see why we need to do this shortly but for now let’s just focus on getting that done. Check and see what shell you’re using…

users-MacBook-Pro:~ user$ echo $SHELL
/bin/zsh
users-MacBook-Pro:~ user$

In my case Im using zsh so I need to add that stuff to .zshrc. If you’re using bash you’ll need to to add it to .bashrc. Once done, restart your terminal (or source the requisite file). Now we should have access to the pyenv CLI tool. Take a look at what versions PyEnv knows about…

user@users-MacBook-Pro ~ % pyenv versions
* system (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro ~ %

So right now – it only knows about the system default version. Let’s see what that is…

user@users-MacBook-Pro ~ % which python
/usr/bin/python
user@users-MacBook-Pro ~ % python --version
Python 2.7.16
user@users-MacBook-Pro ~ %

Nice – Python 2.7.16. Almost certainly not the version you want to use. So let’s take a look at what versions PyEnv knows about that are available for install. Specifically, let’s look for versions in 3.8…

user@users-MacBook-Pro ~ % pyenv install --list | grep "3.8"   
  3.8.0
  3.8-dev
  3.8.1
  3.8.2
  3.8.3
  3.8.4
  3.8.5
  miniconda-3.8.3
  miniconda3-3.8.3
user@users-MacBook-Pro ~ %

If you omit the grep you’ll see an outstanding list of Python versions you can install. For right now – let’s focus on installing 3.8.5. To do that we simply do this…

user@users-MacBook-Pro ~ % pyenv install 3.8.5
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.5.tar.xz...
-> https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tar.xz
Installing Python-3.8.5...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.8.5 to /Users/user/.pyenv/versions/3.8.5

user@users-MacBook-Pro ~ %

Note that this will take some time as it’s building Python from scratch. Once installed, we should see it in our versions…

user@users-MacBook-Pro ~ % pyenv versions
* system (set by /Users/user/.pyenv/version)
  3.8.5
user@users-MacBook-Pro ~ %

Note that the star is still next to the system row meaning that we’re still currently using the system default. If we wish to switch our new global version to 3.8.5 we simply say…

user@users-MacBook-Pro ~ % pyenv global 3.8.5
user@users-MacBook-Pro ~ % 
user@users-MacBook-Pro ~ % which python
/Users/user/.pyenv/shims/python
user@users-MacBook-Pro ~ % python --version
Python 3.8.5
user@users-MacBook-Pro ~ %

Nice! So now our global version of Python will be 3.8.5. Now that we switched Python versions for global use, let’s talk about local use. This is where PyEnv really starts to shine. Let’s install a couple more versions of Python so we can play around…

user@users-MacBook-Pro ~ % pyenv install 2.7.18
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-2.7.18.tar.xz...
-> https://www.python.org/ftp/python/2.7.18/Python-2.7.18.tar.xz
Installing Python-2.7.18...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-2.7.18 to /Users/user/.pyenv/versions/2.7.18

user@users-MacBook-Pro ~ % pyenv install 3.5.1 
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.5.1.tar.xz...
-> https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tar.xz
Installing Python-3.5.1...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.5.1 to /Users/user/.pyenv/versions/3.5.1

user@users-MacBook-Pro ~ %

Ok – so we should now see those as options…

user@users-MacBook-Pro ~ % pyenv versions
  system
  2.7.18
  3.5.1
* 3.8.5 (set by /Users/user/.pyenv/version)
user@users-MacBook-Pro ~ %

Awesome! Now let’s create a set of test projects and associated folders…

user@users-MacBook-Pro ~ % cd test_projects   
user@users-MacBook-Pro test_projects % mkdir project_1
user@users-MacBook-Pro test_projects % mkdir project_2
user@users-MacBook-Pro test_projects % mkdir project_3

Alright – so now let’s start in project_1 and setup PyEnv…

user@users-MacBook-Pro test_projects % cd project_1
user@users-MacBook-Pro project_1 % pyenv local 2.7.18
user@users-MacBook-Pro project_1 % python --version
Python 2.7.18
user@users-MacBook-Pro project_1 %

Notice how we use the command pyenv local rather than pyenv global. Using the local syntax sets the Python distro you wish to use in this local directory. So now let’s go to project_2…

user@users-MacBook-Pro test_projects % cd project_2
user@users-MacBook-Pro project_2 % pyenv local 3.5.1
user@users-MacBook-Pro project_2 % python --version
Python 3.5.1
user@users-MacBook-Pro project_2 %

Nothing too surprising here. So now let’s go to project_3 and just see what the default is…

user@users-MacBook-Pro project_2 % cd ..
user@users-MacBook-Pro test_projects % cd project_3
user@users-MacBook-Pro project_3 % python --version
Python 3.8.5
user@users-MacBook-Pro project_3 %

So we’re back to 3.8.5 which makes sense given that 3.8.5 is our global version and we haven’t set a local version here. Ready for the cool part? Go back to project_1 or project_2 and check again…

user@users-MacBook-Pro project_3 % cd ..
user@users-MacBook-Pro test_projects % cd project_1
user@users-MacBook-Pro project_1 % python --version
Python 2.7.18
user@users-MacBook-Pro project_1 % cd ..
user@users-MacBook-Pro test_projects % cd project_2    
user@users-MacBook-Pro project_2 % python --version
Python 3.5.1
user@users-MacBook-Pro project_2 %

It automagically switches to the correct version based on what directory we’re in! How cool is that?! How does it do that? Well when we set the local version it creates a local file .python-version that holds the version number…

user@users-MacBook-Pro project_2 % ls -al
total 8
drwxr-xr-x  3 user  staff   96 Sep 11 20:10 .
drwxr-xr-x  6 user  staff  192 Sep 11 20:07 ..
-rw-r--r--  1 user  staff    6 Sep 11 20:10 .python-version
user@users-MacBook-Pro project_2 % more .python-version 
3.5.1
user@users-MacBook-Pro project_2 %

Now you might be wondering how this all works though. The fundamental thing that makes PyEnv work are “shims”. They’re basically scripts that redirect commands to PyEnv. We enabled this by making those initial changes to our .zshrc file above. If we look at our $PATH variable now we’ll see it has changed…

user@users-MacBook-Pro ~ % echo $PATH
/Users/user/.pyenv/shims:/Users/user/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
user@users-MacBook-Pro ~ %

Notice how the PyEnv directories are now at the front of the path? And if we look in that directory we’ll see “shims” for all of the relevant Python commands…

user@users-MacBook-Pro shims % ls    
2to3			easy_install-3.8	pip2.7			pydoc3.8		python2.7-gdb.py	python3.5m-config
2to3-3.5		idle			pip3			python			python3			python3.8
2to3-3.8		idle3			pip3.5			python-config		python3-config		python3.8-config
chardetect		idle3.5			pip3.8			python2			python3.5		python3.8-gdb.py
easy_install		idle3.8			pydoc			python2-config		python3.5-config	pyvenv
easy_install-2.7	pip			pydoc3			python2.7		python3.5-gdb.py	pyvenv-3.5
easy_install-3.5	pip2			pydoc3.5		python2.7-config	python3.5m		smtpd.py
user@users-MacBook-Pro shims %

In fact – we can even see this if we just look at what Python version we’re using right now…

user@users-MacBook-Pro shims % which python
/Users/user/.pyenv/shims/python
user@users-MacBook-Pro shims %

So just by calling python a search in PATH catches the PyEnv shim first and returns that rather than the system default Python. And if we look at this file we’ll see it’s a script that just puts PyEnv in the way of our command…

user@users-MacBook-Pro shims % more /Users/user/.pyenv/shims/python
#!/usr/bin/env bash
set -e
[ -n "$PYENV_DEBUG" ] && set -x

program="${0##*/}"
if [[ "$program" = "python"* ]]; then
  for arg; do
    case "$arg" in
    -c* | -- ) break ;;
    */* )
      if [ -f "$arg" ]; then
        export PYENV_FILE_ARG="$arg"
        break
      fi
      ;;
    esac
  done
fi

export PYENV_ROOT="/Users/user/.pyenv"
exec "/Users/user/.pyenv/libexec/pyenv" exec "$program" "$@"
user@users-MacBook-Pro shims %

Pretty cool huh? If you want to see the actual Python you’re using you can call the pyenv which command to see the actual path…

user@users-MacBook-Pro project_1 % pyenv which python
/Users/user/.pyenv/versions/2.7.18/bin/python
user@users-MacBook-Pro project_1 % pyenv which pip   
/Users/user/.pyenv/versions/2.7.18/bin/pip
user@users-MacBook-Pro project_1 %

Same goes for PIP and any other Python tools that have shim files. Speaking of PIP, how does that work with PyEnv? Let’s take a look…

user@users-MacBook-Pro project_1 % pyenv which python
/Users/user/.pyenv/versions/2.7.18/bin/python
user@users-MacBook-Pro project_1 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version  
---------- ---------
certifi    2020.6.20
chardet    3.0.4    
idna       2.10     
pip        19.2.3   
setuptools 41.2.0   
urllib3    1.25.10  
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_1 %

Lets say we want to use pyyaml. Let’s install that quick…

user@users-MacBook-Pro project_1 % pip install pyyaml
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pyyaml
  Using cached https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz
Installing collected packages: pyyaml
  Running setup.py install for pyyaml ... done
Successfully installed pyyaml-5.3.1
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_1 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version  
---------- ---------
certifi    2020.6.20
chardet    3.0.4    
idna       2.10     
pip        19.2.3   
PyYAML     5.3.1    
setuptools 41.2.0   
urllib3    1.25.10  
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_1 %

Cool – now let’s go and take a look at one of our other Python installs….

user@users-MacBook-Pro project_1 % cd ..
user@users-MacBook-Pro test_projects % cd project_2
user@users-MacBook-Pro project_2 % pyenv which python
/Users/user/.pyenv/versions/3.5.1/bin/python
user@users-MacBook-Pro project_2 % pip list
pip (7.1.2)
setuptools (18.2)
You are using pip version 7.1.2, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_2 %

No pyyaml. So it seems to be that the PIP installs are per “environment” or version. Let’s test this out by setting a local Python version for project_3 that matches project_1…

user@users-MacBook-Pro project_3 % cd ..
user@users-MacBook-Pro test_projects % cd project_2
user@users-MacBook-Pro project_2 % cd ..
user@users-MacBook-Pro test_projects % cd project_3
user@users-MacBook-Pro project_3 % pyenv which python
/Users/user/.pyenv/versions/3.8.5/bin/python
user@users-MacBook-Pro project_3 % pip list
Package    Version
---------- ---------
certifi    2020.6.20
chardet    3.0.4
idna       2.10
pip        20.1.1
requests   2.24.0
setuptools 47.1.0
urllib3    1.25.10
WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
You should consider upgrading via the '/Users/user/.pyenv/versions/3.8.5/bin/python3.8 -m pip install --upgrade pip' command.
user@users-MacBook-Pro project_3 % pyenv local 2.7.18
user@users-MacBook-Pro project_3 % pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package    Version  
---------- ---------
certifi    2020.6.20
chardet    3.0.4    
idna       2.10     
pip        19.2.3   
PyYAML     5.3.1    
setuptools 41.2.0   
urllib3    1.25.10  
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
user@users-MacBook-Pro project_3 %

We can see that in the global version (3.8.5) we didn’t have the pyyaml library but the instant we switched to version 2.7.18 we inherited all of the packages that came with that version. Pretty cool right? So while we have package management by version we still don’t have package management by project which is what we really want. The solution for that? Virtual environments of course! Luckily PyEnv works well with those as well and we’ll start exploring those in our next post. Stay tuned!

Working with TC on Linux systems

Jon Langemak — Wed, 01 Jul 2020 16:55:38 +0000

Hi folks! Long time no talk : ) Life has been incredibly busy for me over the last few months so I’ll apologize in advance for the lack of posts. However – I’m aiming to get back on the horse so please stay tuned!

With that out of the way – I wanted to spend some time in this post talking about the command line tool found on Linux systems called tc. We’ve talked about tc before when we discussed creating some network/traffic simulated topologies and it worked awesome for that use case. If you recall from that earlier post tc is short for Traffic Control and allows users to configure qdiscs. A qdisc is short for Queuing Discipline. I like to think of it as manipulating the Linux kernels packet scheduler.

Note: tc is traditionally part of the iproute2 toolset which Im pretty sure (but not positive) is included in most base Linux distros these days.

When tc comes up – it’s easy to immediately start thinking about QOS, queuing, and packet(traffic) control. And while some of the actions available to you when using tc seem obvious, or at least fit within the mindset of queue disciplines (the drop action comes to mind here), you might be surprised to learn that tc can actually do much more. For instance, it can do things like perform encapsulation and modify packet/frame headers. But – rather than keep rambling, let’s jump into a basic example. Per usual – let’s start with what our lab will look like…

To start with, we’ll begin with a single host (it’s really just a VM), test_host_1. The host is supporting two different network namespaces called NS1 and NS2. They are connected back to the main or default network namespace through the use of VETH pairs. One end of the VETH pair resides in the network namespace while the other end resides in default or main namespace of the host. In addition, the host has a single NIC ens6 which we wont be using much in this post at all.

Note: I’m assuming any and all test hosts used in this lab are newly hosts (Im using Ubuntu 18). with no underlying network configuration besides that of ens6.

So let’s go ahead and do the base namespace configuration…

ip netns add ns1
ip link add ns1_veth_ns type veth peer name ns1_veth_float
ip link set ns1_veth_ns netns ns1
ip link set dev ns1_veth_float up
ip netns exec ns1 ip addr add 192.168.10.1/24 dev ns1_veth_ns
ip netns exec ns1 ip link set ns1_veth_ns address 14:ec:d4:01:f1:2b
ip netns exec ns1 ip link set ns1_veth_ns up
ip netns exec ns1 ip link set lo up

ip netns add ns2
ip link add ns2_veth_ns type veth peer name ns2_veth_float
ip link set ns2_veth_ns netns ns2
ip link set dev ns2_veth_float up
ip netns exec ns2 ip addr add 192.168.10.2/24 dev ns2_veth_ns
ip netns exec ns2 ip link set ns2_veth_ns address 84:12:5d:2f:d2:4c
ip netns exec ns2 ip link set ns2_veth_ns up
ip netns exec ns2 ip link set lo up

So at this point, each NS is up and alive and should have the correct VETH pair configuration. But if you’ve noticed, they aren’t able to talk to each other. Why is this?..

root@test_host_1:~# ip netns exec ns1 ping 192.168.10.2 -c 3
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.
From 192.168.10.1 icmp_seq=1 Destination Host Unreachable
From 192.168.10.1 icmp_seq=2 Destination Host Unreachable
From 192.168.10.1 icmp_seq=3 Destination Host Unreachable

--- 192.168.10.2 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2038ms
pipe 3
root@test_host_1:~#

The answer is pretty simple – we haven’t done anything with the other end of the VETH pairs that live in the default network namespace. They are as I like to say “floating” at this point. Typically, in the case of Docker or any other containerization kind of technology these ends that are currently floating would plug into a bridge. The bridge would allow all the things we need to occur (MAC learning, ARP, etc) to happen so that the two NS could talk. The problem right now is that the ARP request packets are showing up on the floating side of the VETH pair and just getting ditched because there’s no where else for them to go. L2 broadcast frames like this would typically be flooded within a given broadcast domain but the problem is that this broadcast domain begins and ends on each side of the VETH pair. If the floating side were to be plugged into a bridge that bridge would propagate the ARP broadcasts to other VETH pairs and everything would work out just fine.

So how do we fix this? Well – we can actually write some tc rules that make things appear to be in the same broadcast domain. To do that, we first need to enable a qdisc or queuing discipline on the interfaces in question. In our case – that would be the floating sides of the VETH pairs. Let’s do that now on both of the interfaces just to get it out of the way…

tc qdisc add dev ns1_veth_float ingress
tc qdisc add dev ns2_veth_float ingress

This syntax is pretty straightforward – we’re adding an ingress queuing discipline to both of the VETH interfaces floating sides that are still in the default name space. Now comes the interesting part – tc has a classifier referred to as Flower. From my understanding there was already something similiar named flow so they just called this one flower but it allows you to select traffic quite easily based on well know header fields such as IP, Mac address, VLAN, etc. So let’s take a look at writing some rules that do this. What’s the first thing that our two namespace need to do in order to talk? ARP! So let’s write some rules that let that happen…

tc filter add dev ns1_veth_float protocol arp parent ffff: prio 1 \
flower \
dst_mac ff:ff:ff:ff:ff:ff \
src_mac 14:ec:d4:01:f1:2b \
action mirred egress redirect dev ns2_veth_float

Once again, I think this is fairly easy to read but let’s talk through it for the sake of making sure we’re on the same page. Here’s how the above breaks down….

Line 1 – Add a filter to the floating side of the NS1 VETH pair interface that matches protocol ARP. Assign it to the ingress queue of the interface (ffff is the static reference to this) and make this filter rule the 1st priority.

Line 2 – For filter matching rules we’ll be using the flower classifier

Line 3 – Match on a broadcast destination MAC

Line 4 – Match on a source Mac address of the NS side of NS1 VETH pair (now you know why we set static MAC addresses when we built those).

Line 5 – If all of the above matches, on egress redirect the traffic to the NS2 VETH pair floating interface.

So let’s put this rule in and see what happens!

root@test_host_1:~# tc filter add dev ns1_veth_float protocol arp parent ffff: prio 1 \
> flower \
> dst_mac ff:ff:ff:ff:ff:ff \
> src_mac 14:ec:d4:01:f1:2b \
> action mirred egress redirect dev ns2_veth_float
root@test_host_1:~#

If the rule goes in with no feedback – you’re off to the races. If you get the dreaded…

RTNETLINK answers: Invalid argument
We have an error talking to the kernel

It means something went wrong. Make sure you added the qdisc to the interface you’re working with as that’s a requirement before writing filter rules. Then double check your syntax and try whittling the rule down until you find the problem statement. In our case though it worked! We can see the rule is applied by using this command…

root@test_host_1:~# tc -s filter show dev ns1_veth_float parent ffff:
filter protocol arp pref 1 flower chain 0 
filter protocol arp pref 1 flower chain 0 handle 0x1 
  dst_mac ff:ff:ff:ff:ff:ff
  src_mac 14:ec:d4:01:f1:2b
  eth_type arp
  not_in_hw
	action order 1: mirred (Egress Redirect to device ns2_veth_float) stolen
 	index 1 ref 1 bind 1 installed 6 sec used 6 sec
 	Action statistics:
	Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
	backlog 0b 0p requeues 0

root@test_host_1:~#

Awesome – looks good. So now try a ping from NS1 to NS2 and see what we get…

root@test_host_1:~# ip netns exec ns1 ping 192.168.10.2
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.

No response, but if we look at the rule we should see hits…

root@test_host_1:~# tc -s filter show dev ns1_veth_float parent ffff:
filter protocol arp pref 1 flower chain 0 
filter protocol arp pref 1 flower chain 0 handle 0x1 
  dst_mac ff:ff:ff:ff:ff:ff
  src_mac 14:ec:d4:01:f1:2b
  eth_type arp
  not_in_hw
	action order 1: mirred (Egress Redirect to device ns2_veth_float) stolen
 	index 1 ref 1 bind 1 installed 205 sec used 191 sec
 	Action statistics:
	Sent 84 bytes 3 pkt (dropped 0, overlimits 0 requeues 0) 
	backlog 0b 0p requeues 0

root@test_host_1:~#

And if you were doing a tcpdump on ns2_veth_float you would have seen not only the traffic arrive, but also a reply from NS2. NS2 doesn’t need any rules to generate a reply back over it’s VETH pair…

root@test_host_1:~# tcpdump -nne -i ns2_veth_float
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ns2_veth_float, link-type EN10MB (Ethernet), capture size 262144 bytes
01:45:58.442778 14:ec:d4:01:f1:2b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.2 tell 192.168.10.1, length 28
01:45:58.442798 84:12:5d:2f:d2:4c > 14:ec:d4:01:f1:2b, ethertype ARP (0x0806), length 42: Reply 192.168.10.2 is-at 84:12:5d:2f:d2:4c, length 28
01:45:59.473754 14:ec:d4:01:f1:2b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.2 tell 192.168.10.1, length 28
01:45:59.473763 84:12:5d:2f:d2:4c > 14:ec:d4:01:f1:2b, ethertype ARP (0x0806), length 42: Reply 192.168.10.2 is-at 84:12:5d:2f:d2:4c, length 28
01:46:00.497742 14:ec:d4:01:f1:2b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.2 tell 192.168.10.1, length 28
01:46:00.497750 84:12:5d:2f:d2:4c > 14:ec:d4:01:f1:2b, ethertype ARP (0x0806), length 42: Reply 192.168.10.2 is-at 84:12:5d:2f:d2:4c, length 28

Alright – we’re doing great! But we need some more rules for this to work end to end at layer 3. Let’s add the same rule for the ARP broadcast on the other interface just in case the initial request comes from NS2 (remember that for every interface you have to define the qdisc the first time!)….

tc qdisc add dev ns2_veth_float ingress
tc filter add dev ns2_veth_float protocol arp parent ffff: prio 1 \
flower \
dst_mac ff:ff:ff:ff:ff:ff \
src_mac 84:12:5d:2f:d2:4c \
action mirred egress redirect dev ns1_veth_float

Now – this hasn’t gotten us any further down the road to L3 connectivity, so what rule do we need to tackle next? Well – when NS1 pings NS2 we see that NS2 is generating the ARP reply, but if look at the interface ns1_veth_float we see that the reply is never getting there…

root@test_host_1:~# tcpdump -nne -i ns1_veth_float
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ns1_veth_float, link-type EN10MB (Ethernet), capture size 262144 bytes
02:13:49.329808 14:ec:d4:01:f1:2b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.2 tell 192.168.10.1, length 28
02:13:50.353738 14:ec:d4:01:f1:2b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.2 tell 192.168.10.1, length 28
02:13:51.377736 14:ec:d4:01:f1:2b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.2 tell 192.168.10.1, length 28
02:13:52.401797 14:ec:d4:01:f1:2b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.2 tell 192.168.10.1, length 28

That’s because we have no matching tc rule! And also because we’re doing our own L2 semantics here to make this work. So we need to add rules to make the ARP reply work as well. Those might look like this…

tc filter add dev ns2_veth_float protocol arp parent ffff: prio 2 \
flower \
dst_mac 14:ec:d4:01:f1:2b  \
src_mac 84:12:5d:2f:d2:4c \
action mirred egress redirect dev ns1_veth_float

tc filter add dev ns1_veth_float protocol arp parent ffff: prio 2 \
flower \
dst_mac 84:12:5d:2f:d2:4c \
src_mac 14:ec:d4:01:f1:2b \
action mirred egress redirect dev ns2_veth_float

The above rules look a lot like the first two rules we entered in, but they’re not looking for broadcasts. Rather – they’re looking to match an ARP reply which would be direct MAC to MAC communication. If we put those two rules in – we ought to see a full ARP conversation take place in our captures…

root@test_host_1:~# tcpdump -nne -i ns1_veth_float
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ns1_veth_float, link-type EN10MB (Ethernet), capture size 262144 bytes
02:16:56.062897 14:ec:d4:01:f1:2b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.2 tell 192.168.10.1, length 28
02:16:56.062913 84:12:5d:2f:d2:4c > 14:ec:d4:01:f1:2b, ethertype ARP (0x0806), length 42: Reply 192.168.10.2 is-at 84:12:5d:2f:d2:4c, length 28
02:16:56.062916 14:ec:d4:01:f1:2b > 84:12:5d:2f:d2:4c, ethertype IPv4 (0x0800), length 98: 192.168.10.1 > 192.168.10.2: ICMP echo request, id 2872, seq 1, length 64

Awesome! So on the first higlighted line we see the ARP request, followed by the ARP reply, and then the actual ICMP echo request since NS1 now has a valid ARP entry for NS2. But alas, ping is still not working. So we need to add one more rule to allow that…

tc filter add dev ns2_veth_float protocol ip parent ffff: prio 3 \
flower \
dst_mac 14:ec:d4:01:f1:2b \
src_mac 84:12:5d:2f:d2:4c \
action mirred egress redirect dev ns1_veth_float

tc filter add dev ns1_veth_float protocol ip parent ffff: prio 3 \
flower \
dst_mac 84:12:5d:2f:d2:4c \
src_mac 14:ec:d4:01:f1:2b \
action mirred egress redirect dev ns2_veth_float

The above rules should look awfully familiar as the only thing that’s changed is the protocol type from arp to ip. But that’s all we need for this to work. So let’s see a full capture…

root@test_host_1:~# tcpdump -nne -i ns1_veth_float
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ns1_veth_float, link-type EN10MB (Ethernet), capture size 262144 bytes
02:20:09.759492 14:ec:d4:01:f1:2b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.10.2 tell 192.168.10.1, length 28
02:20:09.759594 84:12:5d:2f:d2:4c > 14:ec:d4:01:f1:2b, ethertype ARP (0x0806), length 42: Reply 192.168.10.2 is-at 84:12:5d:2f:d2:4c, length 28
02:20:09.759599 14:ec:d4:01:f1:2b > 84:12:5d:2f:d2:4c, ethertype IPv4 (0x0800), length 98: 192.168.10.1 > 192.168.10.2: ICMP echo request, id 2894, seq 1, length 64
02:20:09.759631 84:12:5d:2f:d2:4c > 14:ec:d4:01:f1:2b, ethertype IPv4 (0x0800), length 98: 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 2894, seq 1, length 64
02:20:10.769757 14:ec:d4:01:f1:2b > 84:12:5d:2f:d2:4c, ethertype IPv4 (0x0800), length 98: 192.168.10.1 > 192.168.10.2: ICMP echo request, id 2894, seq 2, length 64
02:20:10.769773 84:12:5d:2f:d2:4c > 14:ec:d4:01:f1:2b, ethertype IPv4 (0x0800), length 98: 192.168.10.2 > 192.168.10.1: ICMP echo reply, id 2894, seq 2, length 64

And there you have it! A full L3 conversation done entirely with tc rules! While this isn’t a super useful example, I hope it delivers the point that tc is capable of a lot! In the next post, we’ll look at some more useful tc actions and extend our lab a little bit. Stay tuned!

Carrier supporting Carrier with BGP-LU

Jon Langemak — Sat, 16 Nov 2019 01:29:47 +0000

In our last post we talked about the less used method of deploying CsC where we ran OSPF and LDP inside the CSC-PE routing-instance.

Note: I can’t help myself apparently so be aware that Carrier of Carriers (CoC) is the same as Carrier supporting Carrier (CsC)

This required some changes to be made to our default LDP export policy as well as how we moved routes between the inet.3 and inet.0 tables. That being said, if you’re a single org it might make good sense to run things that way. I liked how you were able to see all of the remote LDP domain loopbacks in your local inet.3 table which in my mind made it easier to imagine the LSP paths.

That being said, it is clearly not the preferred deployment methodology. Most examples you’ll find leverage BGP (BGP-LU specifically) for the CSC-CE to CSC-PE connections as well as within the local label domains. So in this example, we’ll do just that. Larges chunks of the base configuration will be the same as they were in the previous post but for the sake of clarity I’ll post our starting post the starting configurations and diagrams here once again. Here is our base lab topology…

And once again we’ll split the lab into the orange carrier, who has a network on the left and the right, and the green carrier, who has the top network…

As before, our goal is to make the orange carrier whole by leveraging connectivity through the green carrier so that the orange carrier can support a VPN service for its customer CE routers. The interesting bit about this is that the connectivity we leverage in the green carrier is itself a VPN hence the terminology of Carrier supporting Carrier. So let’s take a look at our base configuration for each router once again…

Note: Im not going to talk through all the specifics of the base configuration in this post. If you’re curious about it – take a look at the walk through of the base configuration in the last post.

vMX1

set interfaces ge-0/0/0 unit 0 family inet address 10.10.10.1/24
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.0/31
set routing-options autonomous-system 65001
set protocols bgp group to_provider type external
set protocols bgp group to_provider export customer_routes
set protocols bgp group to_provider neighbor 169.254.10.1 peer-as 65002
set policy-options prefix-list customer_routes 10.10.10.0/24
set policy-options policy-statement customer_routes term direct from protocol direct
set policy-options policy-statement customer_routes term direct from prefix-list customer_routes
set policy-options policy-statement customer_routes term direct then accept
set policy-options policy-statement customer_routes then reject

vMX2

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.1/31
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.2/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 2.2.2.2/32 primary
set routing-options router-id 2.2.2.2
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/1.0
set routing-instances customer instance-type vrf
set routing-instances customer interface ge-0/0/0.0
set routing-instances customer route-distinguisher 100:100
set routing-instances customer vrf-target target:100:100
set routing-instances customer vrf-table-label
set routing-instances customer routing-options autonomous-system 65002
set routing-instances customer protocols bgp group customer1 type external
set routing-instances customer protocols bgp group customer1 neighbor 169.254.10.0 peer-as 65001

vMX3

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.3/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.4/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 3.3.3.3/32 primary
set routing-options router-id 3.3.3.3
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX4

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.5/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.6/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 4.4.4.4/32 primary
set routing-options router-id 4.4.4.4
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX5

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.7/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.8/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 5.5.5.5/32 primary
set routing-options router-id 5.5.5.5
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX6

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.9/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.10/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 6.6.1.1/32
set routing-options router-id 6.6.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols mpls label-switched-path to-vmx9 to 9.9.1.1
set protocols mpls interface ge-0/0/1.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

vMX7

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.11/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.12/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 7.7.1.1/32 primary
set routing-options router-id 7.7.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols rsvp interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

vMX8

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.13/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.14/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 8.8.1.1/32 primary
set routing-options router-id 8.8.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols rsvp interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

vMX9

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.15/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.16/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 9.9.1.1/32
set routing-options router-id 9.9.1.1
set routing-options autonomous-system 65000
set protocols rsvp interface ge-0/0/0.0
set protocols mpls label-switched-path to-vmx6 to 6.6.1.1
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p

vMX10

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.17/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.18/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 10.10.10.10/32 primary
set routing-options router-id 10.10.10.10
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX11

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.19/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.20/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 11.11.11.11/32 primary
set routing-options router-id 11.11.11.11
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX12

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.21/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.22/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces fxp0 unit 0 family inet address 192.168.127.12/24
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 12.12.12.12/32 primary
set routing-options static route 10.171.200.0/22 next-hop 192.168.127.100
set routing-options router-id 12.12.12.12
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX13

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.23/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.24/31
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 13.13.13.13/32 primary
set routing-options router-id 13.13.13.13
set protocols mpls interface ge-0/0/0.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set routing-instances customer instance-type vrf
set routing-instances customer interface ge-0/0/1.0
set routing-instances customer route-distinguisher 100:100
set routing-instances customer vrf-target target:100:100
set routing-instances customer vrf-table-label
set routing-instances customer routing-options autonomous-system 65004
set routing-instances customer protocols bgp group customer1 type external
set routing-instances customer protocols bgp group customer1 neighbor 169.254.10.25 peer-as 65003

vMX14

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.25/31
set interfaces ge-0/0/1 unit 0 family inet address 192.168.10.1/24
set routing-options autonomous-system 65003
set protocols bgp group to_provider type external
set protocols bgp group to_provider export customer_routes
set protocols bgp group to_provider neighbor 169.254.10.24 peer-as 65004
set policy-options prefix-list customer_routes 192.168.10.0/24
set policy-options policy-statement customer_routes term direct from protocol direct
set policy-options policy-statement customer_routes term direct from prefix-list customer_routes
set policy-options policy-statement customer_routes term direct then accept
set policy-options policy-statement customer_routes then reject

Right – so if you recall from the last post. This sort of got us what I call our “base label domain” configuration. That is – each “block” should have a means to for an MPLS LSP through it. This is an exact copy and past of the base configuration from the previous post. There should be nothing terribly new or exciting to you about this configuration.

As with the last post, the next step is to configure the CSC-PE routers routing-instance for the orange carrier. This configuration differs slightly from our last configuration…

vMX6

set routing-instances orange_carrier instance-type vrf
set routing-instances orange_carrier interface ge-0/0/0.0
set routing-instances orange_carrier route-distinguisher 1:1
set routing-instances orange_carrier vrf-target target:1:1
set routing-instances orange_carrier vrf-table-label
set routing-instances orange_carrier routing-options router-id 6.6.6.6
set routing-instances orange_carrier routing-options autonomous-system 65200
set routing-instances orange_carrier protocols bgp group to_orange_carrier type external
set routing-instances orange_carrier protocols bgp group to_orange_carrier family inet labeled-unicast
set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.8 peer-as 65100
set routing-instances orange_carrier protocols mpls interface ge-0/0/0.0

Notice the lack of OSPF and LDP configuration. This is now replaced by a BGP peering configuration that will be used to peer to the CSC-PE routers. vMX9 looks largely the same…

vMX9

set routing-instances orange_carrier instance-type vrf
set routing-instances orange_carrier interface ge-0/0/1.0
set routing-instances orange_carrier route-distinguisher 1:1
set routing-instances orange_carrier vrf-target target:1:1
set routing-instances orange_carrier vrf-table-label
set routing-instances orange_carrier routing-options router-id 9.9.9.9
set routing-instances orange_carrier routing-options autonomous-system 65200
set routing-instances orange_carrier protocols bgp group to_orange_carrier type external
set routing-instances orange_carrier protocols bgp group to_orange_carrier family inet labeled-unicast
set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.17 peer-as 65100
set routing-instances orange_carrier protocols mpls interface ge-0/0/1.0

Notice that we’re using the same AS number for the peerings to the CSC-CE routers. As you might already be guessing, we’ll have to make some adjustments later on for that to work correctly but for now let’s table that.

The next thing to do is to configure BGP within the orange carrier blocks. To do this, we’ll peer the PE router to the CSC-CE node iBGP. Then we’ll also configure a peering between the CSC-CE and the CSC-PE nodes with eBGP. Let’s start on the left side with vMX2…

vMX2

set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 2.2.2.2
set protocols bgp group internal family inet labeled-unicast resolve-vpn
set protocols bgp group internal neighbor 5.5.5.5 peer-as 65100

Nothing too fancy here but do note that we added the flag resolve-vpn to the BGP configuration. If you recall from our previous posts, this allows the router to add prefixes it receives through BGP-LU to the inet.3 table which is required for VPN route resolution. With that in place, we can configure vMX5 for that same peering as well as another BGP group so it can peer to the CSC-PE node (vMX6 in this case)…

vMX5

set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 5.5.5.5
set protocols bgp group internal family inet labeled-unicast
set protocols bgp group internal neighbor 2.2.2.2 peer-as 65100
set protocols bgp group to_green_carrier type external
set protocols bgp group to_green_carrier family inet labeled-unicast
set protocols bgp group to_green_carrier export to_green_carrier
set protocols bgp group to_green_carrier neighbor 169.254.10.9 peer-as 65200
set policy-options policy-statement to_green_carrier term local_loops from prefix-list local_loops
set policy-options policy-statement to_green_carrier term local_loops then accept
set policy-options policy-statement to_green_carrier then reject
set policy-options prefix-list local_loops 2.2.2.2/32
set policy-options prefix-list local_loops 3.3.3.3/32
set policy-options prefix-list local_loops 4.4.4.4/32
set policy-options prefix-list local_loops 5.5.5.5/32

Notice that we’re doing the loopback export on the CSC-CE node. The reason for doing that here and not on vMX2 is that we also know about that route from OSPF. That is – vMX5 prefers the route from OSPF so it won’t advertise a BGP route it learned about through BGP to another BGP peer. Note that we’re advertising all of the loopbacks from the left orange block through the BGP-LU session to vMX6. Now let’s do the same configuration on the right side orange block…

vMX13

set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 13.13.13.13
set protocols bgp group internal family inet labeled-unicast resolve-vpn
set protocols bgp group internal neighbor 10.10.10.10 peer-as 65100

vMX10

set routing-options autonomous-system 65100
set protocols bgp group internal type internal
set protocols bgp group internal local-address 10.10.10.10
set protocols bgp group internal family inet labeled-unicast
set protocols bgp group internal neighbor 13.13.13.13 peer-as 65100
set protocols bgp group to_green_carrier type external
set protocols bgp group to_green_carrier family inet labeled-unicast
set protocols bgp group to_green_carrier export to_green_carrier
set protocols bgp group to_green_carrier neighbor 169.254.10.16 peer-as 65200
set policy-options policy-statement to_green_carrier term local_loops from prefix-list local_loops
set policy-options policy-statement to_green_carrier term local_loops then accept
set policy-options policy-statement to_green_carrier then reject
set policy-options prefix-list local_loops 10.10.10.10/32
set policy-options prefix-list local_loops 11.11.11.11/32
set policy-options prefix-list local_loops 12.12.12.12/32
set policy-options prefix-list local_loops 13.13.13.13/32

And last but not least, we need to configure the VPN peering within the green carrier…

vMX6

set routing-options autonomous-system 65200
set protocols bgp group green_carrier_vpn type internal
set protocols bgp group green_carrier_vpn family inet-vpn unicast
set protocols bgp group green_carrier_vpn neighbor 9.9.1.1 local-address 6.6.1.1
set protocols bgp group green_carrier_vpn neighbor 9.9.1.1 peer-as 65200

vMX9

set routing-options autonomous-system 65200
set protocols bgp group green_carrier_vpn type internal
set protocols bgp group green_carrier_vpn family inet-vpn unicast
set protocols bgp group green_carrier_vpn neighbor 6.6.1.1 local-address 9.9.1.1
set protocols bgp group green_carrier_vpn neighbor 6.6.1.1 peer-as 65200

Now what we’ve done here is slightly changed out BGP configuration to make things easier to read. The green carrier in this case is wholly inside of AS 65200 while the orange provider (both blocks) are in AS 65100. Now, we still have the sort of one off eBGP peering between the CE and PE routers in each orange block where we use AS 65001 to 65002 and 65003 to 65004. But to be honest, that really doesn’t matter too much for us at this point. As in the last post, that’s really just a means to get the customer prefixes to the orange PE routers. It could very easily have been anything else including static routes or an IGP etc. So our BGP config at this point looks like this…

The one BGP peering we’re clearly missing here is the iBGP VPNv4 peering between vMX2 and vMX13 but before we configure that let’s make sure we have good reachability…

admin@vmx3.lab> show route table inet.3 

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2.2.2.2/32         *[LDP/9] 00:14:14, metric 1
                    > to 169.254.10.2 via ge-0/0/0.0
4.4.4.4/32         *[LDP/9] 00:14:08, metric 1
                    > to 169.254.10.5 via ge-0/0/1.0
5.5.5.5/32         *[LDP/9] 00:13:59, metric 1
                    > to 169.254.10.5 via ge-0/0/1.0, Push 299920

admin@vmx3.lab>

Well that’s not what I had expected. I had hoped that I’d be getting some BGP-LU routes for the routers in the right orange block that be populated into the inet.3 table. So what’s going on? Well – I sort of mentioned this before, but since each CSC-CE is using the same AS number, we’ve got an AS loop. And Juniper is smart enough to know not even to try and advertise routes back to an AS that’s already in a routes AS path list. We can validate that here with just one of the prefixes from the right orange block…

admin@vmx6.lab> show route table orange_carrier.inet.0 11.11.11.11/32              

orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

11.11.11.11/32     *[BGP/170] 00:14:22, MED 1, localpref 100, from 9.9.1.1
                      AS path: 65100 I, validation-state: unverified
                    > to 169.254.10.11 via ge-0/0/1.0, label-switched-path to-vmx9

admin@vmx6.lab> show route advertising-protocol bgp 169.254.10.8 11.11.11.11/32    

admin@vmx6.lab>

Notice that the AS path includes 65100 which was added by vMX10 as it advertised the loopbacks to vMX9 over the eBGP peering. Also notice that despite the routes we want being in the table we need them to be in, they aren’t being sent to vMX5. To fix this, we can tell the green provider to do an AS override…

vMX6

set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.8 as-override

vMX9

set routing-instances orange_carrier protocols bgp group to_orange_carrier neighbor 169.254.10.17 as-override

AS-override is a commonly misunderstood tool. It lets a router replace all instances of a peer’s AS number in the AS-path list with it’s own. It’s important to know that it works on egress. That is, the AS path will be modified as the prefix leaves the green carrier and is advertised toward the orange carrier. We can see that now that this configuration is in place…

admin@vmx6.lab> show route receive-protocol bgp 9.9.1.1 11.11.11.11/32   

orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 11.11.11.11/32          9.9.1.1              1       100        65100 I

bgp.l3vpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
  1:1:11.11.11.11/32                    
*                         9.9.1.1              1       100        65100 I

admin@vmx6.lab> show route advertising-protocol bgp 169.254.10.8 11.11.11.11/32    

orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 11.11.11.11/32          Self                                    65200 I

admin@vmx6.lab>

We can see that when we get the route from vMX9 the AS path is unchanged. But when we advertise it to vMX5 we’ve replaced the 65100 AS with 65200. Now you might be looking at this and be wondering why that looks wrong. Indeed, it looks like we’re missing an AS path entry here. It’s each BGP routers job to add it’s local AS number to the route as it is advertised to an eBGP peer. In this case, it appears that all we’ve done is done the replacement. Where is the AS entry for the route coming from 65200 in the first place?

admin@vmx5.lab> show route receive-protocol bgp 169.254.10.9 11.11.11.11/32 

inet.0: 20 destinations, 20 routes (20 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 11.11.11.11/32          169.254.10.9                            65200 65200 I

admin@vmx5.lab>

As it turns out it’s there. We just don’t get to see it with the advertising-protocol command for some reason. A minor annoyance, but one that can be particularly irritating as you attempt to do troubleshooting.

Now that we have our AS loop sorted, let’s look back on vMX2 to see what we have in the inet.3 table…

admin@vmx2.lab> show route table inet.3                        

inet.3: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

3.3.3.3/32         *[LDP/9] 00:25:07, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0
4.4.4.4/32         *[LDP/9] 00:24:59, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299904
5.5.5.5/32         *[LDP/9] 00:24:49, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299920
10.10.10.10/32     *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
                      AS path: 65200 65200 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 300608, Push 299920(top)
11.11.11.11/32     *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
                      AS path: 65200 65200 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 300592, Push 299920(top)
12.12.12.12/32     *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
                      AS path: 65200 65200 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 300592, Push 299920(top)
13.13.13.13/32     *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
                      AS path: 65200 65200 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 300592, Push 299920(top)
169.254.10.16/31   *[BGP/170] 00:05:40, localpref 100, from 5.5.5.5
                      AS path: 65200 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 300576, Push 299920(top)

admin@vmx2.lab>

Success! So at this point, we should be able to configure out iBGP VPNv4 peering for the orange carrier…

vMX2

set routing-options autonomous-system 65100
set protocols bgp group vpn type internal
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn neighbor 13.13.13.13 local-address 2.2.2.2
set protocols bgp group vpn neighbor 13.13.13.13 peer-as 65100

vMX13

set routing-options autonomous-system 65100
set protocols bgp group vpn type internal
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn neighbor 2.2.2.2 local-address 13.13.13.13
set protocols bgp group vpn neighbor 2.2.2.2 peer-as 65100

And with that complete, our BGP peerings are now complete and should look like this…

But alas – our pings still aren’t working…

left_client:~# ping 192.168.10.100 -c 5
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
  
--- 192.168.10.100 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms
  
left_client:~#

So what could be going on now? Well if I were a betting man, I’d guess that as long as the VPNv4 routes are being sent and accepted between vMX2 and vMX13 that we have a problem with LSP stitching…

admin@vmx2.lab> show route table customer.inet.0 192.168.10.0/24   

customer.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.0/24    *[BGP/170] 00:03:55, localpref 100, from 13.13.13.13
                      AS path: 65004 65003 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 16, Push 300592, Push 299920(top)

admin@vmx2.lab>

And that looks good. So let’s look at our stitching points…

admin@vmx10.lab> show route table inet.0 13.13.13.13/32    

inet.0: 20 destinations, 20 routes (20 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

13.13.13.13/32     *[OSPF/10] 00:33:21, metric 3
                    > to 169.254.10.19 via ge-0/0/1.0

admin@vmx10.lab>

You see – the CSC-CE routers think that their best route to get to 13.13.13.13/32 is through OSPF. Which is totally fair, but also means that we won’t be hopping into an LSP on vMX10 to reach vMX13. And since vMX13 is the endpoint that vMX2 wants to reach…

admin@vmx2.lab> show route table customer.inet.0 192.168.10.0/24    

customer.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.0/24    *[BGP/170] 00:07:36, localpref 100, from 13.13.13.13
                      AS path: 65004 65003 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 16, Push 300592, Push 299920(top)

admin@vmx2.lab>

We now have a problem. We’ve covered the fix(es) for this before and in this case, we should be able to get away with the safest option which is to use mpls-forwarding…

vMX5 and vMX10

set protocols mpls traffic-engineering mpls-forwarding

With that in place, our preferred route on vMX10 should now become the LSP route from LDP…

admin@vmx10.lab> show route table inet.0 13.13.13.13/32    

inet.0: 20 destinations, 23 routes (20 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

13.13.13.13/32     @[OSPF/10] 00:00:48, metric 3
                    > to 169.254.10.19 via ge-0/0/1.0
                   #[LDP/9] 00:00:48, metric 1
                    > to 169.254.10.19 via ge-0/0/1.0, Push 299904

admin@vmx10.lab>

And our pings should now be working as expected…

root@left_client:~# ping 192.168.10.100 -c 5
64 bytes from 192.168.10.100: icmp_seq=1 ttl=57 time=3.86 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=57 time=4.63 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=57 time=4.12 ms
64 bytes from 192.168.10.100: icmp_seq=4 ttl=57 time=6.33 ms
64 bytes from 192.168.10.100: icmp_seq=5 ttl=57 time=4.68 ms
root@left_client:~#

Awesome. So let’s walk through one direction of this flow with labels just so you can see what’s going on. We’ll start with vMX2…

admin@vmx2.lab> show route table customer.inet.0 192.168.10.0/24 

customer.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.0/24    *[BGP/170] 00:02:19, localpref 100, from 13.13.13.13
                      AS path: 65004 65003 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 16, Push 300624, Push 299920(top)

admin@vmx2.lab>

vMX2 is pushing a whopping 3 labels onto the stack. Let’s see what vMX3 does with that top label…

admin@vmx3.lab> show route table mpls.0 label 299920 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299920             *[LDP/9] 00:39:30, metric 1
                    > to 169.254.10.5 via ge-0/0/1.0, Swap 299920

admin@vmx3.lab>

It will swap it for the same label. Now onto vMX4…

admin@vmx4.lab> show route table mpls.0 label 299920 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299920             *[LDP/9] 00:40:00, metric 1
                    > to 169.254.10.7 via ge-0/0/1.0, Pop      
299920(S=0)        *[LDP/9] 00:40:00, metric 1
                    > to 169.254.10.7 via ge-0/0/1.0, Pop      

admin@vmx4.lab>

vMX4 is the PHP router for the LDP LSP ending at vMX5 so it will pop the label. This means that vMX5 will get a top label of 300624 which was imposed at vMX2…

admin@vmx5.lab> show route table mpls.0 label 300624 

mpls.0: 19 destinations, 19 routes (19 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

300624             *[VPN/170] 00:04:22
                    > to 169.254.10.9 via ge-0/0/1.0, Swap 300832

admin@vmx5.lab>

vMX5 will swap that label for 300832…

admin@vmx6.lab> show route table orange_carrier.mpls.0 label 300832 

orange_carrier.mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

300832             *[VPN/170] 00:05:07, metric2 3, from 9.9.1.1
                    > to 169.254.10.11 via ge-0/0/1.0, label-switched-path to-vmx9

admin@vmx6.lab>

vMX6 will use that label to get the traffic into the RSVP LSP to vMX9. Let’s look at the actual label operation…

admin@vmx6.lab> show route forwarding-table vpn orange_carrier label 300832 
Routing table: orange_carrier.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
300832             user     0                    indr  1048577     2
                              169.254.10.11     Swap 300848, Push 299904(top)      610     2 ge-0/0/1.0

admin@vmx6.lab>

Wow – so vMX6 is going to swap the label of 300832 for 300848 and then push 299904 onto the label. So our label stack at this point looks like 299904 300848 16 (assuming the topmost label is on the left). It’s worthwhile here to know where these labels come from. We can assume that 299904 is an RSVP label since it’s the topmost label and we are no in an RSVP domain…

admin@vmx6.lab> show rsvp session    
Ingress RSVP: 1 sessions
To              From            State   Rt Style Labelin Labelout LSPname 
9.9.1.1         6.6.1.1         Up       0  1 FF       -   299904 to-vmx9
Total 1 displayed, Up 1, Down 0

Egress RSVP: 1 sessions
To              From            State   Rt Style Labelin Labelout LSPname 
6.6.1.1         9.9.1.1         Up       0  1 FF       3        - to-vmx6
Total 1 displayed, Up 1, Down 0

Transit RSVP: 0 sessions
Total 0 displayed, Up 0, Down 0

admin@vmx6.lab>

Jackpot – now what about 300848?

admin@vmx6.lab> show route receive-protocol bgp 9.9.1.1 13.13.13.13/32 extensive 

orange_carrier.inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
* 13.13.13.13/32 (1 entry, 1 announced)
     Import Accepted
     Route Distinguisher: 1:1
     VPN Label: 300848
     Nexthop: 9.9.1.1
     MED: 3
     Localpref: 100
     AS path: 65100 I 
     Communities: target:1:1

bgp.l3vpn.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)

* 1:1:13.13.13.13/32 (1 entry, 0 announced)
     Import Accepted
     Route Distinguisher: 1:1
     VPN Label: 300848
     Nexthop: 9.9.1.1
     MED: 3
     Localpref: 100
     AS path: 65100 I 
     Communities: target:1:1

admin@vmx6.lab>

It’s our BGP LU label used to reach the final LSP endpoint! Ok, let’s keep moving along…

admin@vmx7.lab> show route forwarding-table label 299904 
Routing table: default.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
299904             user     0 169.254.10.13     Swap 299904      584     2 ge-0/0/1.0

Routing table: __mpls-oam__.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      540     1

admin@vmx7.lab>

vMX7 will swap the top label with 299904 and send it on it’s way to vMX8…

admin@vmx8.lab> show route forwarding-table label 299904 
Routing table: default.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
299904             user     0 169.254.10.15     Pop        584     2 ge-0/0/1.0
299904(S=0)        user     0 169.254.10.15     Pop        585     2 ge-0/0/1.0

Routing table: __mpls-oam__.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      553     1

admin@vmx8.lab>

vMX8 is the PHP router in the RSVP domain so it will pop the label and send the remaining label stack to vMX9 (which is 300848 16 (assuming the topmost label is on the left))…

admin@vmx9.lab> show route table mpls.0 label 300848 

mpls.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

300848             *[VPN/170] 00:11:32
                    > to 169.254.10.17 via ge-0/0/1.0, Swap 300800

admin@vmx9.lab>

vMX9 will swap the label for 300800. Now this is crucial here. This is the BGP LU route label that vMX10 allocated and sent to vMX9 to tell it how to reach vMX13…

admin@vmx10.lab> show route advertising-protocol bgp 169.254.10.16 13.13.13.13/32 extensive   

inet.0: 20 destinations, 23 routes (20 active, 0 holddown, 0 hidden)
@ 13.13.13.13/32 (2 entries, 2 announced)
 BGP group to_green_carrier type External
     Route Label: 300800
     Nexthop: Self
     Flags: Nexthop Change
     MED: 3
     AS path: [65100] I 
     Entropy label capable

admin@vmx10.lab>

And if we look – vMX10 will use that label to stitch the traffic into the LDP LSP toward vMX13…

admin@vmx10.lab> show route table mpls.0 label 300800 

mpls.0: 19 destinations, 19 routes (19 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

300800             *[VPN/170] 00:13:19
                    > to 169.254.10.19 via ge-0/0/1.0, Swap 299904

admin@vmx10.lab> show ldp database                
Input label database, 10.10.10.10:0--11.11.11.11:0
Labels received: 4
  Label     Prefix
 299872      10.10.10.10/32
      3      11.11.11.11/32
 299888      12.12.12.12/32
 299904      13.13.13.13/32

Output label database, 10.10.10.10:0--11.11.11.11:0
Labels advertised: 4
  Label     Prefix
      3      10.10.10.10/32
 300720      11.11.11.11/32
 300736      12.12.12.12/32
 300752      13.13.13.13/32

admin@vmx10.lab>

vMX11 will then get a label of 299904…

admin@vmx11.lab> show route table mpls.0 label 299904 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299904             *[LDP/9] 00:14:35, metric 1
                    > to 169.254.10.21 via ge-0/0/1.0, Swap 299904

admin@vmx11.lab>

Which it will then swap for 299904 before sending it to vMX12…

admin@vmx12.lab> show route table mpls.0 label 299904 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299904             *[LDP/9] 00:50:57, metric 1
                    > to 169.254.10.23 via ge-0/0/1.0, Pop      
299904(S=0)        *[LDP/9] 00:50:57, metric 1
                    > to 169.254.10.23 via ge-0/0/1.0, Pop      

admin@vmx12.lab>

vMX12 as the PHP router in this LDP domain will pop the label leaving only the initial VPN label imposed by vMX2 on the frame which it will send to vMX13…

admin@vmx13.lab> show route table mpls.0 label 16 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

16                 *[VPN/0] 00:51:44
                    > via lsi.2 (customer), Pop      

admin@vmx13.lab>

Which vMX13 will pop off and dump into the customer routing-instance. If we put that all together in a diagram it would look like this…

Alright. So there you have it. As you can see, the configuration was just about as involved as using LDP and OSPF but strongly mimicked what we had seen earlier when using BGP-LU for LSP stitching. Clearly design requirements would win in terms of which way you do CsC but I can see the argument for having more routing policy flexibility when using BGP-LU.

Carrier Supporting Carrier with OSPF and LDP

Jon Langemak — Tue, 05 Nov 2019 16:36:59 +0000

I know we haven’t gotten to the point of actually discussing inter-as option B or option C but we did cover all of the mechanics required for them to work (at least briefly). Before we go there though – I did want to cover a different technology that can also help solve our end to end LSP problems. Carrier Supporting Carrier – or more commonly just referred to as CsC (or in Cisco parlance Carrier of Carriers or CoC (or maybe I have that backwards?)) is a means to nest MPLS VPN sessions. Think of it this way. If you’re a huge backbone provider (we’ll call you the “uber provider”) selling transport to customers (we’ll call them customer carriers) – there’s a good chance that those customer carriers will need a way to isolate their customer on their backbone. How do we do that? Well MPLS VPNs of course! However you, the uber provider, also need a means to keep the customer carriers isolated as they traverse your backbone. So what do we do? Sounds a lot like a carrier supporting a carrier huh? CsC to the rescue!

At this point you might be wondering how we got here. To be clear, I didn’t just pull CsC out of my “write a random blog on this topic” hat. We’ve spent the last few posts really trying to describe what it means to have end to end LSPs and why they are so important. We’ve talked briefly about how we can do this with label stitching and even touched briefly (admittedly by accident) on the inter-AS architectures B and C that allow us to stitch together discrete label domains. But those solutions are not the only way to crack this problem. CsC can solve this problem too! Although, for some reason, it always seems to be discounted in this space. I’m not sure why that is – perhaps it’s assumed to be complex or geared toward a specific uses case. However – for those of you that read the great book “MPLS in the SDN era” you may recall that CsC is covered at the end of chapter 9 which itself is titled “Inter-Domain MPLS Services”. And while the authors don’t discuss it in detail or specifically call out that it can be used to solve this problem – I believe it’s in that chapter for a reason. Alright – enough chit chat – let’s get to the lab.

For the sake of making this look more real life(ish) I’ve expanded the lab greatly as you can see below. The goal of doing this was to make the LSP segments longer so that we saw some real label operations happening (not just routers popping off labels since they were PHP routers).

As per usual, the blue numbering indicates interface numbering (eg. 0 = ge-0/0/0) and the black numbering indicates the IP addressing. In our case, all of the interface addressing is link local and comes out of 169.254.10.0/24. So for example the link between vMX1 and vMX2 is 169.254.10.0/31. I’ve also removed the loopbacks from the diagram this time around since those will be a little more dynamic this time.

So let’s talk about what we want this to look like…

The easiest way to think of this is that we have a carrier that is split in two which is represented by the routers in the orange blocks. We’ll call this the “orange carrier”. In the left orange block we have a carrier network which connects to a customer CE router. That customer wishes to talk to it’s other CE router on the right side but our orange carrier doesn’t have any direct connectivity to it’s network on the right. To make the carrier “whole”, we need to connect the left side of the orange carrier network to the right side. To do this, we leverage a connection from another transport provider which is represented by the routers in the green box. We’ll call this the “green carrier”. Going forward, we’ll assume that the orange carrier is a customer to the green carrier. You’ll notice that we have updated the name of the router to include its role. You’ll notice some familiar terminology here. There are PE, CE, and P routers – but we also now have a new class of those routers prefaced with “CSC”. The CSC model works much like the normal PE/CE model. In this case, the orange carrier is a customer of the green carrier. So the orange carrier node that connects to the green carrier (vMX5) is considered the CSC-CE router while the green carrier router that connects to it’s customer is considered the CSC-PE router (vMX6).

So at this point – why don’t I stop talking your ear off and we can just get into the config. Since this configuration is more in depth, Im going to start each router with a different baseline configuration. Don’t worry, I’ll talk through anything that looks crazy…

vMX1

set interfaces ge-0/0/0 unit 0 family inet address 10.10.10.1/24
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.0/31
set routing-options autonomous-system 65001
set protocols bgp group to_provider type external
set protocols bgp group to_provider export customer_routes
set protocols bgp group to_provider neighbor 169.254.10.1 peer-as 65002
set policy-options prefix-list customer_routes 10.10.10.0/24
set policy-options policy-statement customer_routes term direct from protocol direct
set policy-options policy-statement customer_routes term direct from prefix-list customer_routes
set policy-options policy-statement customer_routes term direct then accept
set policy-options policy-statement customer_routes then reject

The base configuration of vMX1 is fairly straight forward. In this lab, it’s acting as a pure CE router and our PE/CE protocol of choice is going to be BGP. vMX1 will come from AS 65001 and connect to the PE in AS 65002. For the sake of clarity, I haven’t configured a loopback address since we wont be using it in this lab and I don’t want to confuse things.

Note: I’ll just mention here that the choice of CE/PE protocol being BGP here has very little bearing on the larger architecture. AKA – Don’t read too much into the BGP config between the CE and PE routes in the orange carrier. It’s just there to get the customer prefixes into the VRF table on the PE. Don’t let it distract you from the other BGP peerings that are going on that support CsC.

vMX2

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.1/31
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.2/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 2.2.2.2/32 primary
set routing-options router-id 2.2.2.2
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/1.0
set routing-instances customer instance-type vrf
set routing-instances customer interface ge-0/0/0.0
set routing-instances customer route-distinguisher 100:100
set routing-instances customer vrf-target target:100:100
set routing-instances customer vrf-table-label
set routing-instances customer routing-options autonomous-system 65002
set routing-instances customer protocols bgp group customer1 type external
set routing-instances customer protocols bgp group customer1 neighbor 169.254.10.0 peer-as 65001

The base configuration of vMX2 get’s us a few things. First – an OSPF configuration so that we can talk to the rest of the orange provider router loopbacks. We also enable MPLS and LDP on the upstream interface. Secondly – we define a customer routing instance for our customer. Inside the routing-instance we define our BGP parameters so that we can peer with the customer.

vMX3

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.3/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.4/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 3.3.3.3/32 primary
set routing-options router-id 3.3.3.3
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX3 is our first P router so the configuration here is drop dead easy. OSPF, MPLS, and LDP on all of the interfaces.

vMX4

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.5/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.6/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 4.4.4.4/32 primary
set routing-options router-id 4.4.4.4
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

Same deal here on vMX4. Another P router configuration.

vMX5

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.7/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.8/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 5.5.5.5/32 primary
set routing-options router-id 5.5.5.5
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX5 is our CSC-CE router between the orange and green provider. Now this is where things “could” get interesting but surprisingly don’t. What other configuration wasn’t that interesting again? Oh yeah – the CE config on vMX1 (and vMX14). That’s because a CSC-CE should look a lot like a CE router! Since this is a CE/PE configuration, you might expect BGP here – but in this case, we’re not going to do it. I’ll talk about that more below.

vMX6

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.9/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.10/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 6.6.1.1/32
set routing-options router-id 6.6.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols mpls label-switched-path to-vmx9 to 9.9.1.1
set protocols mpls interface ge-0/0/1.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

The base configuration of the green carrier routers aims to get things rolling for communication within the green carrier. Notice that the loopback address is different in this case (6.6.1.1) and is a theme we will see with all the routers inside of the green carrier. Since we’re going to run RSVP within the green carrier – we also define a LSP to the far end of the green carrier (vMX9). Beyond that it’s just a standard OSPF configuration.

vMX7

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.11/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.12/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 7.7.1.1/32 primary
set routing-options router-id 7.7.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols rsvp interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

Again – a boring P router configuration. Nothing to see here – move along.

vMX8

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.13/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.14/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 8.8.1.1/32 primary
set routing-options router-id 8.8.1.1
set protocols rsvp interface ge-0/0/1.0
set protocols rsvp interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

Same deal – a boring P router configuration. Boring is nice right? 🙂

vMX9

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.15/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.16/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 9.9.1.1/32
set routing-options router-id 9.9.1.1
set routing-options autonomous-system 65000
set protocols rsvp interface ge-0/0/0.0
set protocols mpls label-switched-path to-vmx6 to 6.6.1.1
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p

Just like we saw on vMX6. This should get us connectivity within the green carrier as well as define the RSVP LSP back to vMX6.

vMX10

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.17/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.18/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 10.10.10.10/32 primary
set routing-options router-id 10.10.10.10
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

This configuration is pretty much the same that we saw on vMX5. Don’t worry about how simple this looks for now. We’ll talk more about why this is later on.

vMX11

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.19/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.20/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 11.11.11.11/32 primary
set routing-options router-id 11.11.11.11
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

Another boring P router config. OSPF, LDP, and MPLS.

vMX12

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.21/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.22/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces fxp0 unit 0 family inet address 192.168.127.12/24
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 12.12.12.12/32 primary
set routing-options static route 10.171.200.0/22 next-hop 192.168.127.100
set routing-options router-id 12.12.12.12
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

More boring P router config.

vMX13

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.23/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.24/31
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 13.13.13.13/32 primary
set routing-options router-id 13.13.13.13
set protocols mpls interface ge-0/0/0.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set routing-instances customer instance-type vrf
set routing-instances customer interface ge-0/0/1.0
set routing-instances customer route-distinguisher 100:100
set routing-instances customer vrf-target target:100:100
set routing-instances customer vrf-table-label
set routing-instances customer routing-options autonomous-system 65004
set routing-instances customer protocols bgp group customer1 type external
set routing-instances customer protocols bgp group customer1 neighbor 169.254.10.25 peer-as 65003

We’re doing the same thing here that we did on vMX2. We configure our customer routing instance and map the interface facing the CE into it. We also configure BGP inside the routing-instance to peer with the CE in AS 65003 and we configure out local AS to 65004.

vMX14

set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.25/31
set interfaces ge-0/0/1 unit 0 family inet address 192.168.10.1/24
set routing-options autonomous-system 65003
set protocols bgp group to_provider type external
set protocols bgp group to_provider export customer_routes
set protocols bgp group to_provider neighbor 169.254.10.24 peer-as 65004
set policy-options prefix-list customer_routes 192.168.10.0/24
set policy-options policy-statement customer_routes term direct from protocol direct
set policy-options policy-statement customer_routes term direct from prefix-list customer_routes
set policy-options policy-statement customer_routes term direct then accept
set policy-options policy-statement customer_routes then reject

Alright – so that was a lot of configuration – but hopefully none of it was too new for you. The goal of the baseline configuration was to get the base protocols running in each of our orange and green “boxes”. At this point, our CE’s should be configured and peered into their VRFs in the orange carrier. Let’s make sure that’s the case…

admin@vmx2.lab> show bgp summary instance customer                                    
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
customer.inet.0      
                       1          1          0          0          0          0
customer.mdt.0       
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
169.254.10.0          65001       2257       2256       0       0    16:53:02 Establ
  customer.inet.0: 1/1/1/0

admin@vmx2.lab> show route receive-protocol bgp 169.254.10.0 table customer.inet.0    

customer.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 10.10.10.0/24           169.254.10.0                            65001 I

admin@vmx2.lab>

Notice the syntax we use to look at things related to the customer routing instance. In the case of seeing the BGP summary we can tack on the instance customer command. However, when looking at the routes received we can just call out the table we want to see. So it looks like the peering is up on the left side and we’re receiving the advertisement, let’s check the right side as well…

admin@vmx13.lab> show bgp summary instance customer                                    
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
customer.inet.0      
                       1          1          0          0          0          0
customer.mdt.0       
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
169.254.10.25         65003       2256       2254       0       0    16:52:20 Establ
  customer.inet.0: 1/1/1/0

admin@vmx13.lab> show route receive-protocol bgp 169.254.10.25 table customer.inet.0   

customer.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 192.168.10.0/24         169.254.10.25                           65003 I

admin@vmx13.lab>

Again – all looks good there. Now let’s take a look at what’s going on within the orange carrier left and right boxes. Based on the baseline configuration I’d expect that each router within each box can see each other router in the same box both with OSPF as well as LDP. Let’s check…

admin@vmx2.lab> show route table inet.3 

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

3.3.3.3/32         *[LDP/9] 16:58:21, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0
4.4.4.4/32         *[LDP/9] 16:58:19, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299792
5.5.5.5/32         *[LDP/9] 16:58:08, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299808

admin@vmx2.lab>

Ok – so that looks just about like I’d expect it to look. vMX2 has inet.3 entries for all the other routers within it’s currently reachable network (orange box). Confirming this is enough to prove to me that OSPF is also working since LDP relies on it. Let’s validate the right side orange box as well…

admin@vmx13.lab> show route table inet.3 

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.10.10.10/32     *[LDP/9] 16:59:04, metric 1
                    > to 169.254.10.22 via ge-0/0/0.0, Push 299792
11.11.11.11/32     *[LDP/9] 16:59:04, metric 1
                    > to 169.254.10.22 via ge-0/0/0.0, Push 299776
12.12.12.12/32     *[LDP/9] 16:59:04, metric 1
                    > to 169.254.10.22 via ge-0/0/0.0

admin@vmx13.lab>

Perfect. And last but not least, let’s confirm that the green carrier also has a baseline configuration. I’m hoping that we see an RSVP LSP between vMX6 and vMX9 fully formed…

admin@vmx6.lab> show rsvp session 
Ingress RSVP: 1 sessions
To              From            State   Rt Style Labelin Labelout LSPname 
9.9.1.1         6.6.1.1         Up       0  1 FF       -   299792 to-vmx9
Total 1 displayed, Up 1, Down 0

Egress RSVP: 1 sessions
To              From            State   Rt Style Labelin Labelout LSPname 
6.6.1.1         9.9.1.1         Up       0  1 FF       3        - to-vmx6
Total 1 displayed, Up 1, Down 0

Transit RSVP: 0 sessions
Total 0 displayed, Up 0, Down 0

admin@vmx6.lab>

Perfect! So where do we go from here? At this point, we once again appear to have 3 distinct label domains. Each orange carrier is running LDP within it’s respective box (left and right side) and the backbone or green carrier is running RSVP. So let’s remember what our goal is here…

If it was all the same to the orange carrier, they’d prefer that this looked like one big happy provider network. In fact, they require that. They don’t care about what the green provider is doing to make that happen – their only requirement is connectivity between their two discrete networks that supports them running MPLS VPNs end to end (which dictates the use of end to end LSPs).

Now – when we talk about CsC – there are two primary ways to tackle this solution. One way is to run LDP and an IGP (like OSPF) inside the CSC-PE (vMX6 and vMX9) orange carrier VRF. This solution is often frowned upon because of the complexities that can come along with using OSPF as a PE/CE protocol as well as the fact that BGP really is the defacto standard for PE/CE peering. You’ll hear no disagreement from me on BGP being the clear choice for CE/PE protocols. It’s is the defacto standard for good reason. BGP allows a much greater level of policy control than OSPF does and quite simply is better designed for this kind of situation. HOWEVER – one does not always need this level of policy control. Remember that these types of deployments are not always company A and company B situations where A is trying to protect itself from B and vice-versa. That is – there’s no reason a single entity can’t use CsC to scale its MPLS network and help stitch together end to end LSPs.

In my research to write this post, I found out that configuring CsC using LDP and OSPF as I’ve described above wasn’t talked about at all. Cisco had some documentation on it and it’s mentioned periodically but I couldn’t find any examples of doing it on JunOS. That being said, I think it’s a reasonable solution and so that will be the way we configure CsC in this blog. We will also try to cover the alternative solution of doing CsC with BGP (BGP-LU specifically) but that will be in another blog.

So – now that we have our baseline configuration – where do we start? Let’s start by trying to get the connectivity between the two orange carrier segments working through the green carrier. As I’ve hinted at above, to do that we’ll be configuring LDP and OSPF inside of the VRF on each CSC-PE. One of the things I’d like to try to reiterate about CsC is that it is not unlike a standard MPLS VPN setup. The PE/CE configuration looks pretty darn similar as the CsC layer as it does at the lower customer layer. So let’s start by configuring a routing instance for the orange carrier on each CSC-PE….

vMX6

set interfaces lo0 unit 1 family inet address 6.6.6.6/32
set routing-instances orange_carrier instance-type vrf
set routing-instances orange_carrier interface ge-0/0/0.0
set routing-instances orange_carrier interface lo0.1
set routing-instances orange_carrier route-distinguisher 1:1
set routing-instances orange_carrier vrf-target target:1:1
set routing-instances orange_carrier vrf-table-label
set routing-instances orange_carrier routing-options router-id 6.6.6.6
set routing-instances orange_carrier protocols mpls interface ge-0/0/0.0
set routing-instances orange_carrier protocols mpls interface lo0.1
set routing-instances orange_carrier protocols ospf area 0.0.0.0 interface lo0.1
set routing-instances orange_carrier protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set routing-instances orange_carrier protocols ldp interface ge-0/0/0.0

The above configuration accomplishes a number of things. First – note that we added another loopback interface (lo0.1) which we’ll use inside the VRF. Second – we defined a typical routing-instance configuration that included configuring MPLS, OSPF, and LDP inside of the VRF. Beyond them being there in the first place, their configuration should look pretty straight forward. Alright – now let’s do the same on vMX9 – the other CSC-PE…

vMX9

set interfaces lo0 unit 1 family inet address 9.9.9.9/32
set routing-instances orange_carrier instance-type vrf
set routing-instances orange_carrier interface ge-0/0/1.0
set routing-instances orange_carrier interface lo0.1
set routing-instances orange_carrier route-distinguisher 1:1
set routing-instances orange_carrier vrf-target target:1:1
set routing-instances orange_carrier vrf-table-label
set routing-instances orange_carrier routing-options router-id 9.9.9.9
set routing-instances orange_carrier protocols mpls interface ge-0/0/1.0
set routing-instances orange_carrier protocols mpls interface lo0.1
set routing-instances orange_carrier protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set routing-instances orange_carrier protocols ospf area 0.0.0.0 interface lo0.1
set routing-instances orange_carrier protocols ldp interface ge-0/0/1.0

Once that’s in place. We should now see that the orange carrier routers view the CSC-PE routers as part of their LDP domain…

admin@vmx13.lab> show route table inet.3    

inet.3: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

9.9.9.9/32         *[LDP/9] 00:17:55, metric 1
                    > to 169.254.10.22 via ge-0/0/0.0, Push 299824
10.10.10.10/32     *[LDP/9] 18:54:21, metric 1
                    > to 169.254.10.22 via ge-0/0/0.0, Push 299792
11.11.11.11/32     *[LDP/9] 18:54:21, metric 1
                    > to 169.254.10.22 via ge-0/0/0.0, Push 299776
12.12.12.12/32     *[LDP/9] 18:54:21, metric 1
                    > to 169.254.10.22 via ge-0/0/0.0

admin@vmx13.lab>

Notice that we now see a path to vMX9 in inet.3. We’ll see that on the left side we now also see the CSC-PE loopback as well…

admin@vmx2.lab> show route table inet.3    

inet.3: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

3.3.3.3/32         *[LDP/9] 18:57:15, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0
4.4.4.4/32         *[LDP/9] 18:57:13, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299792
5.5.5.5/32         *[LDP/9] 18:57:02, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299808
6.6.6.6/32         *[LDP/9] 00:00:04, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299824

admin@vmx2.lab>

Perfect. So now we’ve begun the integration of the orange carrier into the green carrier network, However, beyond providing connectivity for LDP and OSPF – we haven’t really done anything. If the green carrier is going to provide VPN connectivity for the orange carrier – then we need to provide a way for both of the green carrier CSC-PEs to exchange route information. This will come in the form a MP-BGP peering in the inet-vpn address family. So let’s get that configured next…

vMX6

set routing-options autonomous-system 65000
set protocols bgp group green_carrier_vpn type internal
set protocols bgp group green_carrier_vpn family inet-vpn unicast
set protocols bgp group green_carrier_vpn neighbor 9.9.1.1 local-address 6.6.1.1
set protocols bgp group green_carrier_vpn neighbor 9.9.1.1 peer-as 65000

vMX9

set routing-options autonomous-system 65000
set protocols bgp group green_carrier_vpn type internal
set protocols bgp group green_carrier_vpn family inet-vpn unicast
set protocols bgp group green_carrier_vpn neighbor 6.6.1.1 local-address 9.9.1.1
set protocols bgp group green_carrier_vpn neighbor 6.6.1.1 peer-as 65000

Once that iBGP peering is up – we should see the session come up and become established. As we’ve seen before – a BGP peering in the inet-vpn family will automatically export all of the VRF or routing-instance routes as VPNv4 routes to the peer…

admin@vmx6.lab> show bgp summary                               
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
bgp.l3vpn.0          
                       9          9          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
9.9.1.1               65000         11         10       0       0        1:45 Establ
  bgp.l3vpn.0: 9/9/9/0
  orange_carrier.inet.0: 9/9/9/0

admin@vmx6.lab> show route advertising-protocol bgp 9.9.1.1    

orange_carrier.inet.0: 21 destinations, 21 routes (21 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 2.2.2.2/32              Self                 4       100        I
* 3.3.3.3/32              Self                 3       100        I
* 4.4.4.4/32              Self                 2       100        I
* 5.5.5.5/32              Self                 1       100        I
* 6.6.6.6/32              Self                         100        I
* 169.254.10.2/31         Self                 4       100        I
* 169.254.10.4/31         Self                 3       100        I
* 169.254.10.6/31         Self                 2       100        I
* 169.254.10.8/31         Self                         100        I

admin@vmx6.lab>

As you can see the prefixes we’re interested in are the left side orange carrier loopbacks. If we look on vMX9 we should see them in the bgp.l3vpn.0 table…

admin@vmx9.lab> show route table bgp.l3vpn.0    

bgp.l3vpn.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1:1:2.2.2.2/32                
                   *[BGP/170] 00:02:45, MED 4, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
1:1:3.3.3.3/32                
                   *[BGP/170] 00:02:45, MED 3, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
1:1:4.4.4.4/32                
                   *[BGP/170] 00:02:45, MED 2, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
1:1:5.5.5.5/32                
                   *[BGP/170] 00:02:45, MED 1, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
1:1:6.6.6.6/32                
                   *[BGP/170] 00:02:45, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
1:1:169.254.10.2/31                
                   *[BGP/170] 00:02:45, MED 4, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
1:1:169.254.10.4/31                
                   *[BGP/170] 00:02:45, MED 3, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
1:1:169.254.10.6/31                
                   *[BGP/170] 00:02:45, MED 2, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
1:1:169.254.10.8/31                
                   *[BGP/170] 00:02:45, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6

admin@vmx9.lab>

Perfect. But now what do we do with these? Our job as the green carrier is to help exchange these routes between the left and the right side orange carrier networks. And if we look at the routing instance table, we’ll see that this has happened…

admin@vmx9.lab> show route table customer_carrier.inet.0 protocol bgp   

customer_carrier.inet.0: 21 destinations, 21 routes (21 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2.2.2.2/32         *[BGP/170] 00:09:09, MED 4, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
3.3.3.3/32         *[BGP/170] 00:09:09, MED 3, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
4.4.4.4/32         *[BGP/170] 00:09:09, MED 2, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
5.5.5.5/32         *[BGP/170] 00:09:09, MED 1, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
6.6.6.6/32         *[BGP/170] 00:09:09, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
169.254.10.2/31    *[BGP/170] 00:09:09, MED 4, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
169.254.10.4/31    *[BGP/170] 00:09:09, MED 3, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
169.254.10.6/31    *[BGP/170] 00:09:09, MED 2, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
169.254.10.8/31    *[BGP/170] 00:09:09, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6

admin@vmx9.lab>

So while that’s awesome that this worked as expected – the orange carrier routers in the left box won’t get these routes? Why not? Because we’re only running OSPF to those routers. In order for them to get the routes, we have to redistribute them into OSPF. Since I don’t really care about the link local addresses, let’s write a policy to only import the remote loopbacks on both vMX6 and vMX9…

vMX6

set policy-options prefix-list BGP-INTO-OSPF 9.9.9.9/32 
set policy-options prefix-list BGP-INTO-OSPF 10.10.10.10/32 
set policy-options prefix-list BGP-INTO-OSPF 11.11.11.11/32    
set policy-options prefix-list BGP-INTO-OSPF 12.12.12.12/32    
set policy-options prefix-list BGP-INTO-OSPF 13.13.13.13/32  
set policy-options policy-statement BGP-INTO-OSPF term BGP from protocol bgp
set policy-options policy-statement BGP-INTO-OSPF term BGP from prefix-list BGP-INTO-OSPF
set policy-options policy-statement BGP-INTO-OSPF term BGP then accept
set policy-options policy-statement BGP-INTO-OSPF then reject
set routing-instances orange_carrier protocols ospf export BGP-INTO-OSPF

vMX9

set policy-options prefix-list BGP-INTO-OSPF 2.2.2.2/32
set policy-options prefix-list BGP-INTO-OSPF 3.3.3.3/32
set policy-options prefix-list BGP-INTO-OSPF 4.4.4.4/32
set policy-options prefix-list BGP-INTO-OSPF 5.5.5.5/32
set policy-options prefix-list BGP-INTO-OSPF 6.6.6.6/32
set policy-options policy-statement BGP-INTO-OSPF term BGP from protocol bgp
set policy-options policy-statement BGP-INTO-OSPF term BGP from prefix-list BGP-INTO-OSPF
set policy-options policy-statement BGP-INTO-OSPF term BGP then accept
set policy-options policy-statement BGP-INTO-OSPF then reject
set routing-instances orange_carrier protocols ospf export BGP-INTO-OSPF

So with this config our remote PEs should see all of the loopbacks of the orange carrier routers from the remote side…

admin@vmx2.lab> show route table inet.0    

inet.0: 20 destinations, 20 routes (20 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2.2.2.2/32         *[Direct/0] 19:55:39
                    > via lo0.0
3.3.3.3/32         *[OSPF/10] 19:20:18, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0
4.4.4.4/32         *[OSPF/10] 19:20:18, metric 2
                    > to 169.254.10.3 via ge-0/0/1.0
5.5.5.5/32         *[OSPF/10] 19:20:04, metric 3
                    > to 169.254.10.3 via ge-0/0/1.0
6.6.6.6/32         *[OSPF/10] 00:23:07, metric 4
                    > to 169.254.10.3 via ge-0/0/1.0
9.9.9.9/32         *[OSPF/150] 00:01:07, metric 0, tag 3489725928
                    > to 169.254.10.3 via ge-0/0/1.0
10.10.10.10/32     *[OSPF/10] 00:00:16, metric 5
                    > to 169.254.10.3 via ge-0/0/1.0
10.171.200.0/22    *[Static/5] 19:55:39
                    > to 192.168.127.100 via fxp0.0
11.11.11.11/32     *[OSPF/10] 00:00:16, metric 6
                    > to 169.254.10.3 via ge-0/0/1.0
12.12.12.12/32     *[OSPF/10] 00:00:16, metric 7
                    > to 169.254.10.3 via ge-0/0/1.0
13.13.13.13/32     *[OSPF/10] 00:00:16, metric 8
                    > to 169.254.10.3 via ge-0/0/1.0
169.254.10.2/31    *[Direct/0] 19:53:21
                    > via ge-0/0/1.0
169.254.10.2/32    *[Local/0] 19:53:21
                      Local via ge-0/0/1.0
169.254.10.4/31    *[OSPF/10] 19:20:18, metric 2
                    > to 169.254.10.3 via ge-0/0/1.0
169.254.10.6/31    *[OSPF/10] 19:20:18, metric 3
                    > to 169.254.10.3 via ge-0/0/1.0
169.254.10.8/31    *[OSPF/10] 19:20:04, metric 4
                    > to 169.254.10.3 via ge-0/0/1.0
192.168.127.0/24   *[Direct/0] 19:55:39
                    > via fxp0.0
192.168.127.2/32   *[Local/0] 19:55:39
                      Local via fxp0.0
224.0.0.2/32       *[LDP/9] 19:22:11, metric 1
                      MultiRecv
224.0.0.5/32       *[OSPF/10] 19:20:38, metric 1
                      MultiRecv

admin@vmx2.lab>

Perfect! But now we have another problem. Just because we have a route to them – doesn’t mean we can build an LSP to them…

admin@vmx2.lab> show route table inet.3 

inet.3: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

3.3.3.3/32         *[LDP/9] 19:26:30, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0
4.4.4.4/32         *[LDP/9] 19:26:28, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299792
5.5.5.5/32         *[LDP/9] 19:26:17, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299808
6.6.6.6/32         *[LDP/9] 00:29:19, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Push 299824

admin@vmx2.lab>

They’re still missing from our inet.3 table. If our goal is for this to look rather seamless to the orange carrier, then they are going to need inet.3 entries in order to build their LSPs. The fix for this is to modify the default LDP policy. In Juniper – by default – only loopback addresses are exported into LDP (which I think makes things awfully clean). And when I say exported – I mean put into the inet.3 table. We need to change this now to import other prefixes, in this case the ones we learned from BGP, into LDP as well. To do this, we modify the LDP egress policy inside of the VRF as follows…

vMX6 and vMX9

set policy-options policy-statement ldp-export term from_bgp from protocol bgp
set policy-options policy-statement ldp-export term from_bgp from prefix-list BGP-INTO-OSPF
set policy-options policy-statement ldp-export term from_bgp then accept
set policy-options policy-statement ldp-export term from_l0 from interface lo0.1
set policy-options policy-statement ldp-export term from_l0 then accept
set policy-options policy-statement ldp-export then reject
set routing-instances orange_carrier protocols ldp egress-policy ldp-export

In addition to validating that they came from BGP we can leverage the existing prefix-list we created BGP-INTO-OSPF to only capture the remote loopbacks we’re interested in. We also want to make sure that we include the local loopback interface as part of the export. Once this is applied on both CSC-PE nodes (vMX6 and vMX9) we should see all the entries we’d expect in the inet.3 table of all of the orange carrier routers. Let’s pick one at random to validate that assumption…

admin@vmx11.lab> show route table inet.3    

inet.3: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2.2.2.2/32         *[LDP/9] 00:04:23, metric 1
                    > to 169.254.10.18 via ge-0/0/0.0, Push 299856
3.3.3.3/32         *[LDP/9] 00:04:23, metric 1
                    > to 169.254.10.18 via ge-0/0/0.0, Push 299872
4.4.4.4/32         *[LDP/9] 00:04:23, metric 1
                    > to 169.254.10.18 via ge-0/0/0.0, Push 299888
5.5.5.5/32         *[LDP/9] 00:04:23, metric 1
                    > to 169.254.10.18 via ge-0/0/0.0, Push 299904
6.6.6.6/32         *[LDP/9] 00:04:23, metric 1
                    > to 169.254.10.18 via ge-0/0/0.0, Push 299920
9.9.9.9/32         *[LDP/9] 00:00:14, metric 1
                    > to 169.254.10.18 via ge-0/0/0.0, Push 299936
10.10.10.10/32     *[LDP/9] 21:54:56, metric 1
                    > to 169.254.10.18 via ge-0/0/0.0
12.12.12.12/32     *[LDP/9] 21:54:51, metric 1
                    > to 169.254.10.21 via ge-0/0/1.0
13.13.13.13/32     *[LDP/9] 21:54:46, metric 1
                    > to 169.254.10.21 via ge-0/0/1.0, Push 299808

admin@vmx11.lab>

Great! So at this point, we have inet.3 resolution for LDP for all of the orange carrier routers as well as their LDP endpoints int he CSC-PE routing-instances!

So at this point – there’s really only one thing left to do and that’s configure a inet-vpn peering for our orange carrier. To do this, we’ll peer vMX2 directly to vMX13 since we now have reachability between the two loopbacks through the green carrier.

vMX2

set routing-options autonomous-system 65100
set protocols bgp group vpn type internal
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn neighbor 13.13.13.13 local-address 2.2.2.2
set protocols bgp group vpn neighbor 13.13.13.13 peer-as 65100

vMX13

set routing-options autonomous-system 65100
set protocols bgp group vpn type internal
set protocols bgp group vpn family inet-vpn unicast
set protocols bgp group vpn neighbor 2.2.2.2 local-address 13.13.13.13
set protocols bgp group vpn neighbor 2.2.2.2 peer-as 65100

Once that’s configured – we should see the peering come up and the peers start automatically advertising any VPNv4 prefixes to each other…

admin@vmx2.lab> show bgp summary 
Groups: 2 Peers: 2 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
bgp.l3vpn.0          
                       2          2          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
13.13.13.13           65100          6          7       0       0          31 Establ
  customer.inet.0: 2/2/2/0
  bgp.l3vpn.0: 2/2/2/0
169.254.10.0          65001       2946       2947       0       0    22:02:20 Establ
  customer.inet.0: 1/1/1/0

admin@vmx2.lab> show route advertising-protocol bgp 13.13.13.13 

customer.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 10.10.10.0/24           Self                         100        65002 65001 I
* 169.254.10.0/31         Self                         100        I

admin@vmx2.lab>

Perfect! To help level set where we are with BGP configuration the image below describes the various peerings we have configured at this point. Note with the exception of the PE/CE eBGP peerings (which are largely of no consequence to the larger design) the two main BGP peerings we have are for a VPNv4 BGP peering between both the CSC-PE nodes of the green carrier and the PE nodes of the orange carrier…

So will our ping work at this point? Let’s go look and see…

left_client:~# ping 192.168.10.100 -c 5
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
 
--- 192.168.10.100 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms
 
left_client:~#

Darn. Ok – so what’s going on now? Let’s trace the label path and see if we can sort this out…

admin@vmx2.lab> show route table customer.inet.0 192.168.10.100 

customer.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.0/24    *[BGP/170] 00:03:42, localpref 100, from 13.13.13.13
                      AS path: 65004 65003 I, validation-state: unverified
                    > to 169.254.10.3 via ge-0/0/1.0, Push 16, Push 299904(top)

admin@vmx2.lab>

Ok – so vMX2 thinks it should be pushing a VPN label and then what Im guessing is an LDP label…

admin@vmx2.lab> show ldp database | grep 299904 
 299904      13.13.13.13/32
 299904      13.13.13.13/32

admin@vmx2.lab>

Yes, a LDP label onto the frame. So vMX3 will get a frame with a top label of 299904….

admin@vmx3.lab> show route table mpls.0 label 299904 

mpls.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299904             *[LDP/9] 00:17:28, metric 1
                    > to 169.254.10.5 via ge-0/0/1.0, Swap 299904

admin@vmx3.lab>

Which it then swaps for the same label. So vMX4 will get 299904…

admin@vmx4.lab> show route table mpls.0 label 299904 

mpls.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299904             *[LDP/9] 00:18:05, metric 1
                    > to 169.254.10.7 via ge-0/0/1.0, Swap 299904

admin@vmx4.lab>

Im sensing a theme here…

admin@vmx5.lab> show route table mpls.0 label 299904 

mpls.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299904             *[LDP/9] 00:18:35, metric 1
                    > to 169.254.10.9 via ge-0/0/1.0, Swap 299904

admin@vmx5.lab>

Now what will vMX6 do with it?

admin@vmx6.lab> show route table orange_carrier.mpls.0 label 299904 

orange_carrier.mpls.0: 16 destinations, 16 routes (16 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

299904             *[LDP/9] 00:19:03, metric 1, metric2 3
                    > to 169.254.10.11 via ge-0/0/1.0, label-switched-path to-vmx9

admin@vmx6.lab>

Aha! It’s putting it into the RSVP tunnel to vMX9. Let’s see what label operation it’s doing though…

Note: Im doing fancy greps here to see the label operations just since it’s the easiest way to see the RSVP label operations.

admin@vmx6.lab> show route table orange_carrier.mpls.0 label 299904 extensive | grep "Label operation" 
                Label operation: Swap 16, Push 299792(top)
                Label operation: Swap 16
                        Label operation: Swap 16

admin@vmx6.lab>

Ok – so vMX7 will get a label of 299792…

admin@vmx7.lab> show route table mpls.0 label 299792 extensive | grep "Label operation"   
                Label operation: Swap 299792

admin@vmx7.lab>

Which is then swaps for 299792 before it sends it to vMX8…

admin@vmx8.lab> show route table mpls.0 label 299792 extensive | grep "Label operation" 
                Label operation: Pop      
                Label operation: Pop      

admin@vmx8.lab>

vMX8 would be the PHP router for the RSVP label domain so popping the label might make some sense here. So that would mean that vMX9 would get a label of 16…

admin@vmx9.lab> show route table mpls.0 label 16 

mpls.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

16                 *[VPN/0] 02:49:55
                    > via lsi.1 (orange_carrier), Pop      

admin@vmx9.lab>

Huh. So it’s popping the label off and dumping it into the VRF. So what we have looks a lot like this…

The labels in orange are imposed by the orange carrier while the ones in green are by the green carrier. If this were a normal MPLS VPN scenario I’d expect the PHP router for the green carrier, which in our case is vMX8, to pop off the transport label and send a frame with a VPN label to vMX9. Now remember – this VPN we’re talking about here is in the green carrier – so it’s the VPN label that was imposed by vMX6 (label 16 in the above example). vMX9 would use that VPN label to sort out what VRF the traffic should land in. But the problem is that when it does that, there’s still yet another VPN label on the frame which was imposed by our orange carrier. We can see this in a packet capture between vMX8 and vMX9…

We’d expect (actually need) vMX9 to leave the bottom layer VPN label alone since that’s only relevant to the orange carrier. In this case, it’s popping off the green carriers VPN label and dumping the labelled frame into the orange_carrier VRF which is doing us no good at all. So why isn’t it stitching our LSPs together?

The problem as it turns out is related to the vrf-table-label command. The router in this case is largely doing what we asked it to do. We’ve talked about the vrf-table-label command in previous posts – and you might recall that it has two rather interesting impacts on how the router handles the VRF. First, when enabled, it assigns the same label for every prefix from that given VRF. We can see that if we look at the routes being advertised over to vMX6….

admin@vmx9.lab> show route advertising-protocol bgp 6.6.1.1 extensive 

orange_carrier.inet.0: 21 destinations, 21 routes (21 active, 0 holddown, 0 hidden)
* 9.9.9.9/32 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 16
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1

* 10.10.10.10/32 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 16
     Nexthop: Self
     Flags: Nexthop Change
     MED: 1
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 11.11.11.11/32 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 16
     Nexthop: Self
     Flags: Nexthop Change
     MED: 2
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 12.12.12.12/32 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 16
     Nexthop: Self
     Flags: Nexthop Change
     MED: 3
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 13.13.13.13/32 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 16
     Nexthop: Self
     Flags: Nexthop Change
     MED: 4
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 169.254.10.16/31 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1           
     VPN Label: 16
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1

* 169.254.10.18/31 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 16
     Nexthop: Self
     Flags: Nexthop Change
     MED: 2
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 169.254.10.20/31 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 16
     Nexthop: Self
     Flags: Nexthop Change
     MED: 3
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 169.254.10.22/31 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 16
     Nexthop: Self
     Flags: Nexthop Change
     MED: 4
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

admin@vmx9.lab>

Second – it forces a local VRF lookup through an automatically created LSI interface. This allows us to overcome some of the problems that come along with having to do things like ARP lookups inside the VRF. However, in this case, it’s actually causing us some problems. If we force a lookup in the VRF – we can’t really precompute an end to end LSP path. So vMX9 in this case is going dump the traffic in the VRF through the LSI interface and try to resolve things there. That’s what we saw above. Unfortunately for us, in this case, that just won’t work since we have another VPN label on the frame for the orange carrier.

To fix this, lets start by removing vrf-table-label on both CSC-PE routers…

vMX6 and vMX9

delete routing-instances orange_carrier vrf-table-label

After making that change, let’s look at the routes being advertised to vMX6 again…

admin@vmx9.lab> show route advertising-protocol bgp 6.6.1.1 extensive    

orange_carrier.inet.0: 21 destinations, 21 routes (21 active, 0 holddown, 0 hidden)
* 9.9.9.9/32 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301216
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1

* 10.10.10.10/32 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301232
     Nexthop: Self
     Flags: Nexthop Change
     MED: 1
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 11.11.11.11/32 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301232
     Nexthop: Self
     Flags: Nexthop Change
     MED: 2
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 12.12.12.12/32 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301232
     Nexthop: Self
     Flags: Nexthop Change
     MED: 3
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 13.13.13.13/32 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301232
     Nexthop: Self
     Flags: Nexthop Change
     MED: 4
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 169.254.10.16/31 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1           
     VPN Label: 301232
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1

* 169.254.10.18/31 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301232
     Nexthop: Self
     Flags: Nexthop Change
     MED: 2
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 169.254.10.20/31 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301232
     Nexthop: Self
     Flags: Nexthop Change
     MED: 3
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

* 169.254.10.22/31 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301232
     Nexthop: Self
     Flags: Nexthop Change
     MED: 4
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

admin@vmx9.lab>

Alright – do we have a unique label for each possible prefix? No – we dont. The thing is – locally allocated labels are a means for a router to say “if I get a label of X I know I need to do Y”. So in some cases, the router regardless of destination prefix will do the same thing. If we look at the destinations for the majority of the prefixes we’ll see that they are all using the label of 301232. So what does that label do?..

admin@vmx9.lab> show route table mpls.0 label 301232                     

mpls.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301232             *[VPN/170] 00:04:15
                    > to 169.254.10.17 via ge-0/0/1.0, Pop      
301232(S=0)        *[VPN/170] 00:04:15
                    > to 169.254.10.17 via ge-0/0/1.0, Pop      

admin@vmx9.lab>

Pop again… Darn. So as you guessed – our ping still isn’t working. So let’s look at the orange carriers VRF table again…

admin@vmx9.lab> show route table orange_carrier.inet.0 

orange_carrier.inet.0: 21 destinations, 21 routes (21 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

2.2.2.2/32         *[BGP/170] 00:14:25, MED 4, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
3.3.3.3/32         *[BGP/170] 00:14:25, MED 3, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
4.4.4.4/32         *[BGP/170] 00:14:25, MED 2, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
5.5.5.5/32         *[BGP/170] 00:14:25, MED 1, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
6.6.6.6/32         *[BGP/170] 00:14:26, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
9.9.9.9/32         *[Direct/0] 3d 02:39:56
                    > via lo0.1
10.10.10.10/32     *[OSPF/10] 00:22:44, metric 1
                    > to 169.254.10.17 via ge-0/0/1.0
11.11.11.11/32     *[OSPF/10] 00:22:44, metric 2
                    > to 169.254.10.17 via ge-0/0/1.0
12.12.12.12/32     *[OSPF/10] 00:22:44, metric 3
                    > to 169.254.10.17 via ge-0/0/1.0
13.13.13.13/32     *[OSPF/10] 00:22:44, metric 4
                    > to 169.254.10.17 via ge-0/0/1.0
169.254.10.2/31    *[BGP/170] 00:14:25, MED 4, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
169.254.10.4/31    *[BGP/170] 00:14:25, MED 3, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
169.254.10.6/31    *[BGP/170] 00:14:25, MED 2, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
169.254.10.8/31    *[BGP/170] 00:14:26, localpref 100, from 6.6.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.14 via ge-0/0/0.0, label-switched-path to-vmx6
169.254.10.16/31   *[Direct/0] 3d 02:39:56
                    > via ge-0/0/1.0
169.254.10.16/32   *[Local/0] 3d 02:39:56
                      Local via ge-0/0/1.0
169.254.10.18/31   *[OSPF/10] 00:22:44, metric 2
                    > to 169.254.10.17 via ge-0/0/1.0
169.254.10.20/31   *[OSPF/10] 00:22:44, metric 3
                    > to 169.254.10.17 via ge-0/0/1.0
169.254.10.22/31   *[OSPF/10] 00:22:44, metric 4
                    > to 169.254.10.17 via ge-0/0/1.0
224.0.0.2/32       *[LDP/9] 3d 02:39:56, metric 1
                      MultiRecv
224.0.0.5/32       *[OSPF/10] 3d 02:39:56, metric 1
                      MultiRecv

admin@vmx9.lab>

Can anyone spot the issue? The routes that vMX9 knows about to reach the destination we need to get to (in this case we really want to get to the loopback of vMX13 (13.13.13.13)) are all know locally through OSPF. That wont work if we want to ride an LSP all the way there! We’ve run into this in a previous post – but the fix for this is to copy the routes from inet.3 into inet.0 since the router itself can’t resolve the destinations in inet.3 (only BGP can). To do this, we’ll use our mpls-forwarding command for MPLS traffic engineering…

vMX6 and vMX9

set routing-instances orange_carrier protocols mpls traffic-engineering mpls-forwarding

Once that’s in – our ping should start working…

root@left_client:~# ping 192.168.10.100 -c 5
64 bytes from 192.168.10.100: icmp_seq=1 ttl=57 time=3.86 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=57 time=4.63 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=57 time=4.12 ms
64 bytes from 192.168.10.100: icmp_seq=4 ttl=57 time=6.33 ms
64 bytes from 192.168.10.100: icmp_seq=5 ttl=57 time=4.68 ms
root@left_client:~#

And if we look at our routing table again – we’ll see that the destinations we’re interested in now have LDP routes for forwarding…

admin@vmx9.lab> show route table orange_carrier.inet.0 protocol ldp 

orange_carrier.inet.0: 21 destinations, 25 routes (21 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

10.10.10.10/32     #[LDP/9] 00:01:31, metric 1
                    > to 169.254.10.17 via ge-0/0/1.0
11.11.11.11/32     #[LDP/9] 00:01:31, metric 1
                    > to 169.254.10.17 via ge-0/0/1.0, Push 299776
12.12.12.12/32     #[LDP/9] 00:01:31, metric 1
                    > to 169.254.10.17 via ge-0/0/1.0, Push 299792
13.13.13.13/32     #[LDP/9] 00:01:31, metric 1
                    > to 169.254.10.17 via ge-0/0/1.0, Push 299808
224.0.0.2/32       *[LDP/9] 3d 02:44:45, metric 1
                      MultiRecv

admin@vmx9.lab>

This is one case where not having vrf-table-label on could be a good idea. Not having it on allows us to remove the constraint of one label per VRF which means that the router can allocate as many labels as it needs based on how many different actions it needs to perform. However, this also means that we’re chewing through labels. Look at this excerpt of the routes being advertised to vMX6 now…

* 9.9.9.9/32 (1 entry, 1 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301216
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1

@ 10.10.10.10/32 (2 entries, 2 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301392
     Nexthop: Self
     Flags: Nexthop Change
     MED: 1
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

@ 11.11.11.11/32 (2 entries, 2 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301408
     Nexthop: Self
     Flags: Nexthop Change
     MED: 2
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

@ 12.12.12.12/32 (2 entries, 2 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301424
     Nexthop: Self
     Flags: Nexthop Change
     MED: 3
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

@ 13.13.13.13/32 (2 entries, 2 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301440
     Nexthop: Self
     Flags: Nexthop Change
     MED: 4
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

We’re now allocating a lot more unique labels than we were before. So is there a happy middle ground here? As it turns out – we can also get this to work with vrf-table-label enabled as well. So let’s turn that back on…

vMX6 and vMX9

set routing-instances orange_carrier vrf-table-label

Our ping should break once more – but let’s talk about why this isn’t working in greater detail. Let’s make sure we’re seeing the same behavior again…

admin@vmx9.lab> show route advertising-protocol bgp 6.6.1.1 13.13.13.13/32 extensive 

orange_carrier.inet.0: 21 destinations, 25 routes (21 active, 0 holddown, 0 hidden)
@ 13.13.13.13/32 (2 entries, 2 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 16
     Nexthop: Self
     Flags: Nexthop Change
     MED: 4
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1 rte-type:0.0.0.0:1:0

admin@vmx9.lab> show route table mpls.0 label 16  

mpls.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

16                 *[VPN/0] 00:00:49
                    > via lsi.768 (orange_carrier), Pop      

admin@vmx9.lab>

Yep – ok. Same problem. But let’s think about this for a moment. We talked earlier that routes from the orange carrier would be turned into VPNv4 routes on the CSC-PEs and then exported back into OSPF. Since we turned on our traffic engineering flag for mpls-forwarding we copied the routes we care about (the router loopbacks) from inet.3 into inet.0 right?..

admin@vmx9.lab> show route table orange_carrier.inet.0 13.13.13.13/32  

orange_carrier.inet.0: 21 destinations, 25 routes (21 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

13.13.13.13/32     @[OSPF/10] 00:09:29, metric 4
                    > to 169.254.10.17 via ge-0/0/1.0
                   #[LDP/9] 00:09:29, metric 1
                    > to 169.254.10.17 via ge-0/0/1.0, Push 299808

admin@vmx9.lab>

Remember that the mpls-forwarding option is the safest option to use since it only copies the routes for forwarding use. But in this case, that’s actually biting us. See – when the router converts those routes to VPNv4 routes it’s doing so with the OSPF route since that’s the one thats winning. We can tell this is the case by looking at the communities attached to the route. Notice above how the community list included rte-type:0.0.0.0:1:0. That’s actually a community that BGP uses so it knows what kind of OSPF route it’s dealing with. This is actually the “route type” field as described in RFC 4577. The layout looks like this…

AREA Number (4 bytes) : Route Type (1 byte) : Options (1 byte)

So we’re saying this came from area 0 (0.0.0.0) and is a route type of intra-area. In either case, this is an OSPF route that’s being translated and sent to vMX6 as a VPNv4 route. Since vMX9 believes that the OSPF route is the winning route – it still think that it should pop off what it believes to be the last label (16 (the green carrier VPN label)) and send it into the VRF toward the destination which it knows about through OSPF. To fix this, we need to tell the router to prefer the LDP route. We can do this by changing our traffic engineering flag to bgp-igp-both-ribs which will whole sale copy the routes from inet.3 into inet.0. Let’s try that out…

vMX6 and vMX9

set routing-instances orange_carrier protocols mpls traffic-engineering bgp-igp-both-ribs

Now if we look at the routing table, we should see the LDP routes in there and notice how they are no longer just for forwarding, they’re winning for both forwarding and routing use…

admin@vmx9.lab> show route table orange_carrier.inet.0 13.13.13.13/32 

orange_carrier.inet.0: 21 destinations, 25 routes (21 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

13.13.13.13/32     *[LDP/9] 00:00:28, metric 1
                    > to 169.254.10.17 via ge-0/0/1.0, Push 299808
                    [OSPF/10] 00:47:39, metric 4
                    > to 169.254.10.17 via ge-0/0/1.0

admin@vmx9.lab>

This means that the route being sent to vMX6 is no longer the OSPF route…

admin@vmx9.lab> show route advertising-protocol bgp 6.6.1.1 13.13.13.13/32 extensive    

orange_carrier.inet.0: 21 destinations, 25 routes (21 active, 0 holddown, 0 hidden)
* 13.13.13.13/32 (2 entries, 2 announced)
 BGP group green_carrier_vpn type Internal
     Route Distinguisher: 1:1
     VPN Label: 301568
     Nexthop: Self
     Flags: Nexthop Change
     MED: 1
     Localpref: 100
     AS path: [65000] I 
     Communities: target:1:1

admin@vmx9.lab>

Notice the lack of the rte-type community above. And if we look at what the router will do with that label…

admin@vmx9.lab> show route table mpls.0 label 301568 

mpls.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301568             *[VPN/170] 00:02:08
                    > to 169.254.10.17 via ge-0/0/1.0, Swap 299808

admin@vmx9.lab> show ldp database instance orange_carrier 
Input label database, 9.9.9.9:0--10.10.10.10:0
Labels received: 10
  Label     Prefix
 300240      2.2.2.2/32
 300256      3.3.3.3/32
 300272      4.4.4.4/32
 300288      5.5.5.5/32
 299920      6.6.6.6/32
 299936      9.9.9.9/32
      3      10.10.10.10/32
 299776      11.11.11.11/32
 299792      12.12.12.12/32
 299808      13.13.13.13/32

Output label database, 9.9.9.9:0--10.10.10.10:0
Labels advertised: 10
  Label     Prefix
 301504      2.2.2.2/32
 301472      3.3.3.3/32
 301456      4.4.4.4/32
 301488      5.5.5.5/32
 299968      6.6.6.6/32
      3      9.9.9.9/32
 301328      10.10.10.10/32
 301344      11.11.11.11/32
 301360      12.12.12.12/32
 301376      13.13.13.13/32

admin@vmx9.lab>

It will swap that label with one the one it learned through LDP to get the traffic to vMX13. And our test ping should once again be working…

root@left_client:~# ping 192.168.10.100 -c 5
64 bytes from 192.168.10.100: icmp_seq=1 ttl=57 time=3.86 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=57 time=4.63 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=57 time=4.12 ms
64 bytes from 192.168.10.100: icmp_seq=4 ttl=57 time=6.33 ms
64 bytes from 192.168.10.100: icmp_seq=5 ttl=57 time=4.68 ms
root@left_client:~#

If we were to visualize what an end to end flow would look like it might look like this…

Note: Im only posting the graphic – we already posted an example of the commands used to get this information earlier.

You can see that the green labels imposed by the green carrier are simply for isolating the orange carrier traffic while it traverses the green carrier.

So there you have it. We’ve used CsC to connect discrete label domains. In the next post we’ll talk about how we can do CsC using BGP instead of OSPF and LDP in the CSC-PE VRF. Stay tuned!

Fundamentals of MPLS LSPs

Jon Langemak — Fri, 18 Oct 2019 19:01:46 +0000

One of the items that often trips folks up with MPLS is the concept of label switched paths or LSPs. We’ve talked about them extensively before in many of the blog posts here and I’ve described them a couple of different ways. Many people look at an LSP as a sort of unidirectional tunnel. In fact, most network diagrams aiming to describe an LSP often show it just as that – a tunnel. It’s an easy thing to visualize especially when you start talking about nested tunnels or LSPs inside of LSPs, but I also think it can be rather confusing. This becomes even more confusing when people start talking about end to end LSPs or how a service label is the same end to end as traffic traverses an LSP. What does that mean? Where does an LSP start or stop? Is it really a tunnel? How far can an LSP reach? What if we run different label distribution protocols? In this post, and perhaps the next, I hope to address these questions as well as talk about how we can solve some of the common problems that are often encountered with LSPs.

So let’s dive right in and talk about the LSP to tunnel analogy. As we’ve already seen in previous posts, LSPs are not a tunnel in the sense that most networking engineering folks are accustom to. When I think of tunneling, the first thing that comes to my mind is GRE tunnels. In that model, we’re encapsulating IP inside of IP. Router A encapsulates a packet inside another packet and sends it toward the tunnel destination router B. Router B then strips off the outer IP header and now has the original encapsulated packet to work with. In MPLS – it’s the same model except we’re dealing with a label instead of an outer IP packet header. The difference between something like GRE and MPLS is how the transit, or “routers in the middle”, deal with the traffic. In the MPLS world – transit routers only care about labels and label operations. They never inspect the packet to make a decision as to how to forward the tunneled traffic. On the other hand, a GRE transit router still needs to do a route lookup on the outer IP packet to sort out where to send the tunneled traffic. Another interesting difference to think about is that MPLS routers change the label (in most cases) as a means to facilitate moving traffic through the LSP. A GRE transit router doesn’t change the outer, or tunnel, IP packet. It simply performs a lookup and passes the traffic to the next transit router who will also do a lookup on that same IP header. When thinking about it at that level – one could compare an MPLS label to the ethernet headers destination MAC address. A GRE transit router needs to know the MAC address for the next hop it resolves during it’s IP lookup and that MAC address needs to be written into the L2 header so that the next hop router knows to accept the frame. In that case you might compare ARP with a label distribution protocol. In either case, the routers need to exchange information that tells them how to talk to one another and that information is used to facilitate forwarding at the tunnel level. The significant difference between the two methodologies when viewed from this perspective is that an MPLS LSP (tunnel) is precomputed on all routers that take part in the LSP. A GRE router is still performing lookups at each hop along the path to the tunnel destination. This brings us back to one of the initial selling points of label or tag based forwarding which was that the router did not need to do IP lookups at each hop.

So at a high level – I think the comparison between an LSP being a networking “tunnel” is rather apt since it behaves much like any other tunnel we typically see in networking. Put more simply – the mechanism used to move the tunneled traffic from point A to point B never needs to examine the traffic that’s being tunneled. This is one of the huge benefits of MPLS since it allows us to have architectures like BGP free core designs where “routers in the middle” don’t need to know how to forward tunneled traffic. More importantly, MPLS allows us to tack on additional “services” like MPLS VPNs that really make the architecture significantly more useful and scalable than a tunneling technology such as GRE.

My point with this rambling is that tunneling or encapsulation in networking is an analogy that can be looked at lots of different ways. I mean one look at the OSI model and you should see that. Despite this – when I initially started working with MPLS years ago – I had a a hard time picturing an LSP as a tunnel. It just didn’t align with my thinking of what a tunnel was. But once I started hands on working with it, the LSP to tunnel analogy made a lot more sense to me. You’re encapsulating some traffic inside of MPLS labels which allow routers along the way to not have to inspect the encapsulated datagrams. And as we’ll see in this post (or maybe the next) the LSP tunnel analogy is helpful as we try to visualize things like hierarchical LSPs (don’t panic – we haven’t talked about what those are yet).

To help cement the concept of LSPs as tunnels, let’s look at an example lab topology where multiple LSPs come into play…

Note: The lab is the same one we used in the previous BGP-LU posts so if you’ve read those then this should look familiar to you.

Above you can see we have a simple topology. I’m including the above diagram to show you the link local IP addressing and the interface names so you can follow along. In future diagrams, I need to take that stuff out so it isn’t so cluttered. The blue numbers represent the interface numbering. AKA – 0 is the same as ge-0/0/0. Here is what our label distribution protocol use will look like…

The above lab demonstrates an important aspect of MPLS LSPs. Here we have two sites (left and right (I know I’m super creative)) separated by a core. The core in this case is running RSVP as a label distribution protocol whereas the left and right sites are running LDP. In this initial iteration, we’re not running any MPLS services across the core (think MPLS VPNs etc), rather we’re just pursuing the basic MPLS use case of a BGP free core. The intent here is that the routers between the two tail routers have no knowledge of the prefixes being advertised through BGP between the the left and the right sites. Routers 1 and 7 provide connectivity for two clients aptly named left_client and right_client. Before we layer in BGP – let’s look at the base configuration of each of the routers so you can follow along if you like…

vMX1

set system host-name vmx1.lab
set interfaces ge-0/0/0 unit 0 family inet address 10.10.10.1/24
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.0/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 1.1.1.1/32 primary
set routing-options router-id 1.1.1.1
set protocols ldp interface ge-0/0/1.0
set protocols mpls interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

vMX2

set system host-name vmx2.lab
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.1/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.2/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 2.2.2.2/32 primary
set routing-options router-id 2.2.2.2
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

vMX3

set system host-name vmx3.lab
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.3/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.4/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 3.3.3.3/32 primary
set protocols rsvp interface ge-0/0/1.0
set protocols ldp interface ge-0/0/0.0 
set protocols mpls label-switched-path to-vmx5 to 5.5.5.5
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

vMX4

set system host-name vmx4.lab
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.5/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.6/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 4.4.4.4/32 primary
set routing-options router-id 4.4.4.4
set protocols rsvp interface ge-0/0/1.0
set protocols rsvp interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

vMX5

set system host-name vmx5.lab
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.7/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.8/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 5.5.5.5/32 primary
set routing-options router-id 5.5.5.5
set protocols rsvp interface ge-0/0/0.0
set protocols mpls label-switched-path to-vmx3 to 3.3.3.3
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ldp interface ge-0/0/1.0

vMX6

set system host-name vmx6.lab
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.9/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 169.254.10.10/31
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 6.6.6.6/32 primary
set routing-options router-id 6.6.6.6
set protocols mpls interface ge-0/0/1.0
set protocols mpls interface ge-0/0/0.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface ge-0/0/1.0

vMX7

set system host-name vmx7.lab
set interfaces ge-0/0/0 unit 0 family inet address 169.254.10.11/31
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 192.168.10.1/24
set interfaces lo0 unit 0 family inet no-redirects
set interfaces lo0 unit 0 family inet address 7.7.7.7/32 primary
set routing-options static route 10.171.200.0/22 next-hop 192.168.127.100
set routing-options router-id 7.7.7.7
set protocols mpls interface ge-0/0/0.0
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ldp interface ge-0/0/0.0

As you can see – this is all pretty standard configuration. I’ve enabled MPLS on all of the interfaces where we expect to see labels and RSVP or LDP on the interfaces that require it. On vMX3 and vMX4 I created RSVP LSPs that simply reference the other IP addresses loopback. You’ll also notice that all the routers are part of one big OSPF area. That’s me cheating, typically you wouldn’t have the same IGP area in both the core and the remote sites but for the sake of this discussion, it doesn’t matter.

Now that we have our base configuration done, let’s look at configuring BGP. If our goal was simply to advertise the client prefixes from vMX1 to vMX7 – you’d think that a direct peering between the two would be sufficient. So let’s imagine that vMX1 is in AS 65001 and vMX2 is in AS 65002 and try that out…

vMX1

set routing-options autonomous-system 65001 
set protocols bgp group external peer-as 65002 
set protocols bgp group external neighbor 7.7.7.7 family inet unicast               
set protocols bgp group external multihop ttl 255    
set protocols bgp group external type external      
set protocols bgp group external local-address 1.1.1.1 
set policy-options policy-statement bgp-export term direct from protocol direct 
set policy-options policy-statement bgp-export term direct from prefix-list bgp-export           
set policy-options policy-statement bgp-export term direct then accept                    
set protocols bgp group external export bgp-export                                        
set policy-options prefix-list bgp-export 10.10.10.0/24

vMX7

set routing-options autonomous-system 65002
set protocols bgp group external peer-as 65001 
set protocols bgp group external neighbor 1.1.1.1 family inet unicast               
set protocols bgp group external multihop ttl 255    
set protocols bgp group external type external  
set protocols bgp group external local-address 7.7.7.7      
set policy-options policy-statement bgp-export term direct from protocol direct 
set policy-options policy-statement bgp-export term direct from prefix-list bgp-export           
set policy-options policy-statement bgp-export term direct then accept                    
set protocols bgp group external export bgp-export                                        
set policy-options prefix-list bgp-export 192.168.10.0/24

So nothing too fancy here. The only weird part is that I’m doing an eBGP peering to a loopback so I need to increase the TTL for multihop but the rest seems pretty straight forward. Once that config is committed we should see our peerings come up and we should get the routes from the other side…

admin@vmx1.lab> show route table inet.0 192.168.10.0/24 

inet.0: 22 destinations, 22 routes (22 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.0/24    *[BGP/170] 00:03:39, localpref 100, from 7.7.7.7
                      AS path: 65002 I, validation-state: unverified
                    > to 169.254.10.1 via ge-0/0/1.0

admin@vmx1.lab>

admin@vmx7.lab> show route table inet.0 10.10.10.0/24 

inet.0: 22 destinations, 22 routes (22 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.10.10.0/24      *[BGP/170] 00:04:00, localpref 100, from 1.1.1.1
                      AS path: 65001 I, validation-state: unverified
                    > to 169.254.10.10 via ge-0/0/0.0

admin@vmx7.lab>

All looks good. So can the clients ping each other?

left_client:~# ping 192.168.10.100 -c 5
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.

--- 192.168.10.100 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms

left_client:~#

Negative. Any guesses as to why? Let’s try and track this down…

admin@vmx1.lab> show route table inet.0 192.168.10.100 extensive 

inet.0: 22 destinations, 22 routes (22 active, 0 holddown, 0 hidden)
192.168.10.0/24 (1 entry, 1 announced)
TSI:
KRT in-kernel 192.168.10.0/24 -> {indirect(1048574)}
        *BGP    Preference: 170/-101
                Next hop type: Indirect, Next hop index: 0
                Address: 0xd0078d0
                Next-hop reference count: 2
                Source: 7.7.7.7
                Next hop type: Router, Next hop index: 583
                Next hop: 169.254.10.1 via ge-0/0/1.0, selected
                Session Id: 0x140
                Protocol next hop: 7.7.7.7
                Indirect next hop: 0xb68cec0 1048574 INH Session ID: 0x142
                State: 
                Local AS: 65001 Peer AS: 65002
                Age: 9 	Metric2: 6 
                Validation State: unverified 
                Task: BGP_65002.7.7.7.7+179
                Announcement bits (2): 0-KRT 6-Resolve tree 2 
                AS path: 65002 I 
                Accepted
                Localpref: 100
                Router ID: 7.7.7.7
                Indirect next hops: 1
                        Protocol next hop: 7.7.7.7 Metric: 6
                        Indirect next hop: 0xb68cec0 1048574 INH Session ID: 0x142
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 169.254.10.1 via ge-0/0/1.0
                                Session Id: 0x140
			7.7.7.7/32 Originating RIB: inet.0
			  Metric: 6			  Node path count: 1
			  Forwarding nexthops: 1
				Nexthop: 169.254.10.1 via ge-0/0/1.0
				Session Id: 140

admin@vmx1.lab>

If we look at the route we’re getting for vMX7 – we’ll see that it’s being advertised as having a protocol next hop of 7.7.7.7. This is to be expected since that’s the router-ID of vMX7. So let’s look and see what the router wants to do with this route when we try to talk to right_client….

admin@vmx1.lab> show route forwarding-table destination 192.168.10.100 
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
192.168.10.0/24    user     0                    indr  1048574     2
                              169.254.10.1       ucst      583    19 ge-0/0/1.0

Routing table: __pfe_private__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      514     2

Routing table: __juniper_services__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      527     2

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct      542     1

admin@vmx1.lab>

Notice the above highlighted lines. vMX1 thinks it should forward the traffic over to 169.254.10.1 which is the ge-0/0/0 interface of vMX2. This doesn’t seem like a problem since that’s the direction we need to go. So now let’s go checkout out what vMX2 will do with that traffic…

admin@vmx2.lab> show route forwarding-table destination 192.168.10.100 
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct       36     1

Routing table: __juniper_services__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      514     2

Routing table: __pfe_private__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      527     2

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct      542     1

admin@vmx2.lab>

Huh. So vMX2 wants nothing to do with that traffic is is going to drop it. So why is that? Well the reason is that vMX2 doesn’t know about the route to 192.168.10.0/24 since that’s only being shared between the 2 eBGP peers vMX1 and vMX7…

admin@vmx2.lab> show route table inet.0 192.168.10.0/24 

admin@vmx2.lab>

This is one of the reason we wanted to run MPLS – so that every router doesn’t need to know about all the BGP prefixes. So this is actually the behavior we expect to see. In order for this to work, traffic toward 192.168.10.100 needs to enter an MPLS LSP on vMX1 and for some reason that isn’t happening.

The reason is simple – MPLS LSPs need to be end to end. As this design stands, we have 3 distinct label domains. The left site is only aware of the routers in it’s LDP domain, the core is only aware of routers speaking RSVP, and the right site is only aware of routers in it’s LDP domain. We know that for LDP and RSVP to get into an LSP we need to have an entry in the routers inet.3 table. Let’s see what vMX1 knows about…

admin@vmx1.lab> show route table inet.3 

inet.3: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2.2.2.2/32         *[LDP/9] 00:53:22, metric 1
                    > to 169.254.10.1 via ge-0/0/1.0
3.3.3.3/32         *[LDP/9] 00:53:20, metric 1
                    > to 169.254.10.1 via ge-0/0/1.0, Push 299792

admin@vmx1.lab>

As you can see vMX1 is only aware of vMX2 and vMX3. If we look at vMX3 we’ll see that it’s aware of it’s LDP peers (vMX1 and vMX2) as well as it’s RSVP peer (vMX5) in the inet.3 table. In other words – these are the only routers that vMX3 knows how to get to through an LSP…

admin@vmx3.lab> show route table inet.3 

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[LDP/9] 00:11:27, metric 1
                    > to 169.254.10.2 via ge-0/0/0.0, Push 299776
2.2.2.2/32         *[LDP/9] 00:11:27, metric 1
                    > to 169.254.10.2 via ge-0/0/0.0
5.5.5.5/32         *[RSVP/7/1] 00:53:08, metric 2
                    > to 169.254.10.5 via ge-0/0/1.0, label-switched-path to-vmx5

admin@vmx3.lab>

So the problem is that vMX1 doesn’t know how to get traffic to 192.168.10.100 into an LSP since it’s not aware of an LSP towards 7.7.7.7. If we were to quickly enable LDP on the interfaces where we’re running RSVP currently we’d see that our ping would start working. In fact, let’s do that so you can see…

vMX3

set protocols ldp interface ge-0/0/1

vMX4

set protocols ldp interface ge-0/0/0
set protocols ldp interface ge-0/0/1

vMX5

set protocols ldp interface ge-0/0/0

Now our ping should start working…

root@left_client:~# ping 192.168.10.100 -c 5
64 bytes from 192.168.10.100: icmp_seq=1 ttl=57 time=3.86 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=57 time=4.63 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=57 time=4.12 ms
64 bytes from 192.168.10.100: icmp_seq=4 ttl=57 time=6.33 ms
64 bytes from 192.168.10.100: icmp_seq=5 ttl=57 time=4.68 ms
root@left_client:~#

And if we look at the inet.3 table on vMX1 again…

admin@vmx1.lab> show route table inet.3    

inet.3: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2.2.2.2/32         *[LDP/9] 00:57:17, metric 1
                    > to 169.254.10.1 via ge-0/0/1.0
3.3.3.3/32         *[LDP/9] 00:57:15, metric 1
                    > to 169.254.10.1 via ge-0/0/1.0, Push 299792
4.4.4.4/32         *[LDP/9] 00:01:19, metric 1
                    > to 169.254.10.1 via ge-0/0/1.0, Push 299872
5.5.5.5/32         *[LDP/9] 00:00:50, metric 1
                    > to 169.254.10.1 via ge-0/0/1.0, Push 299888
6.6.6.6/32         *[LDP/9] 00:00:50, metric 1
                    > to 169.254.10.1 via ge-0/0/1.0, Push 299904
7.7.7.7/32         *[LDP/9] 00:00:50, metric 1
                    > to 169.254.10.1 via ge-0/0/1.0, Push 299920

admin@vmx1.lab>

Also notice that our forwarding table now shows a push action…

admin@vmx1.lab> show route forwarding-table destination 192.168.10.100    
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
192.168.10.0/24    user     0                    indr  1048574     2
                              169.254.10.1      Push 299920      593     2 ge-0/0/1.0

Routing table: __pfe_private__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      514     2

Routing table: __juniper_services__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      527     2

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct      542     1

admin@vmx1.lab>

So let’s rollback that config on vMX3, vMX4, and vMX5 (rollback 1) and talk about how to solve this. It’s obvious to me at this point that we have something that looks like this….

Within each label domain, we have a path from edge to edge. The problem is – we don’t have an end to end label path from vMX1 to vMX7. Without that – we’d be asking each border router (in this case vMX3 and vMX5) to have the traffic leave one LSP and re-enter another. That is client_left traffic would need to enter an LSP on vMX1 and head as far as it can to vMX3 which is the LDP LSP endpoint. At vMX3 some type of lookup would need to take place that would tell the traffic to enter into the RSVP LSP to vMX5 with a similar lookup having to occur at vMX5 to reach vMX7. I like to think of MPLS forwarding paths as having to be completely pre-determined. Having to leave an LSP, do a lookup for the next hop, and then enter another LSP, is not pre-determined. This is one of the bigger misconceptions I find when talking with folks about MPLS. There is a general assumption that because there exists a path of LSPs that this is the same as having an end to end LSP.

So how do we fix this? Well – there are a couple of solutions here. The one I want to focus on in this post requires the use of BGP-LU or BGP Labelled Unicast. If you’re unfamiliar with the labelled unicast address family have a quick read here. If you don’t want to do that – I can boil it down to saying that it causes BGP to allocate a label for each prefix it advertises. So why is this helpful? Well to create an end to end LSP we need some mechanism on our border routers (vMX3 and vMX5) to keep the LSP alive. That is – we need to have a label operation it can use to stitch together our discrete LSPs. Let’s configure this and I’ll talk more about it.

To configure BGP labelled unicast, we’re going to redo our BGP peering to look more like this…

You might be asking if we need all these peering sessions for this to work and the short answer is yes – we’ll see why shortly. So let’s drop these configurations on each router…

vMX1

delete protocols bgp
set protocols bgp group internal peer-as 65001
set protocols bgp group internal neighbor 3.3.3.3 family inet labeled-unicast
set protocols bgp group internal local-address 1.1.1.1
set protocols bgp group internal type internal
set protocols bgp group internal export bgp-export 
set policy-options policy-statement bgp-export term direct from protocol direct 
set policy-options policy-statement bgp-export term direct from prefix-list bgp-export           
set policy-options policy-statement bgp-export term direct then accept                                                      
set policy-options prefix-list bgp-export 10.10.10.0/24

vMX3

set routing-options autonomous-system 65001
set protocols bgp group internal peer-as 65001
set protocols bgp group internal neighbor 1.1.1.1 family inet labeled-unicast
set protocols bgp group internal local-address 3.3.3.3
set protocols bgp group internal type internal
set protocols bgp group external type external
set protocols bgp group external multihop ttl 255
set protocols bgp group external local-address 3.3.3.3
set protocols bgp group external peer-as 65002
set protocols bgp group external neighbor 5.5.5.5 family inet labeled-unicast

vMX5

set routing-options autonomous-system 65002
set protocols bgp group internal peer-as 65002
set protocols bgp group internal neighbor 7.7.7.7 family inet labeled-unicast
set protocols bgp group internal local-address 5.5.5.5
set protocols bgp group internal type internal
set protocols bgp group external type external
set protocols bgp group external multihop ttl 255
set protocols bgp group external local-address 5.5.5.5
set protocols bgp group external peer-as 65001
set protocols bgp group external neighbor 3.3.3.3 family inet labeled-unicast

vMX7

delete protocols bgp
set protocols bgp group internal peer-as 65002
set protocols bgp group internal neighbor 5.5.5.5 family inet labeled-unicast
set protocols bgp group internal local-address 7.7.7.7
set protocols bgp group internal type internal
set protocols bgp group internal export bgp-export 
set policy-options prefix-list customer1 192.168.10.0/24 
set policy-options policy-statement bgp-export term direct from protocol direct 
set policy-options policy-statement bgp-export term direct from prefix-list bgp-export           
set policy-options policy-statement bgp-export term direct then accept                                                           
set policy-options prefix-list bgp-export 192.168.10.0/24

Once this configuration is in place, you should be able to validate that all of the BGP peerings are up on each of the 4 routers. If they aren’t – double check your configuration. Once up – you should notice that your ping from client_left to client_right is now once again working!

left_client:~# ping 192.168.10.100 -c 5
64 bytes from 192.168.10.100: icmp_seq=1 ttl=57 time=3.86 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=57 time=4.63 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=57 time=4.12 ms
64 bytes from 192.168.10.100: icmp_seq=4 ttl=57 time=6.33 ms
64 bytes from 192.168.10.100: icmp_seq=5 ttl=57 time=4.68 ms
left_client:~#

So what magic is happening to allow this to work? They key to understanding what’s happening is what I mentioned in my post on BGP-LU…

Anytime a router changes the next-hop of a prefix it’s advertising – it must allocate a new label.

And when does BGP router change the next hop of the route it’s advertising? Well – by default that happens over a eBGP peering. But in the case of BGP-LU it happens more frequently…

When the labeled-unicast statement is used, the local router automatically performs a next hop to self on all routes advertised into EBGP from IBGP and into IBGP from EBGP

So in our case – each of the BGP enabled routers will allocate a new label. Let’s dig in and make sure that’s what’s is happening. Let’s start at vMX1 and look to see what it thinks it should be doing to reach 192.168.10.100…

admin@vmx1.lab> show route table inet.0 192.168.10.100 

inet.0: 22 destinations, 22 routes (22 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.0/24    *[BGP/170] 00:09:48, localpref 100, from 3.3.3.3
                      AS path: 65002 I, validation-state: unverified
                    > to 169.254.10.1 via ge-0/0/1.0, Push 301296, Push 301152(top)

admin@vmx1.lab>

Interesting. So not only do we now have an MPLS operation, we have two of them! So where are these labels coming from? Let’s look at little deeper to see what we can find…

admin@vmx1.lab> show ldp database 
Input label database, 1.1.1.1:0--2.2.2.2:0
Labels received: 3
  Label     Prefix
 301136      1.1.1.1/32
      3      2.2.2.2/32
 301152      3.3.3.3/32

Output label database, 1.1.1.1:0--2.2.2.2:0
Labels advertised: 3
  Label     Prefix
      3      1.1.1.1/32
 299856      2.2.2.2/32
 299872      3.3.3.3/32

admin@vmx1.lab>

Ok – so our top label of 301152 is something we learned through LDP. vMX2 is telling us that if we want to get to 3.3.3.3 we should send it a label of 299792. Ok – so now where does label 301296 come from?..

admin@vmx1.lab> show route receive-protocol bgp 3.3.3.3 table inet.0 extensive 

inet.0: 22 destinations, 22 routes (22 active, 0 holddown, 0 hidden)
* 192.168.10.0/24 (1 entry, 1 announced)
     Accepted
     Route Label: 301296
     Nexthop: 3.3.3.3
     Localpref: 100
     AS path: 65002 I 
     Entropy label capable, next hop field matches route next hop

admin@vmx1.lab>

It’s coming from BGP! So vMX1 thinks to get to 192.168.10.100 it should push a label of 301296 followed by 301152. If you’re confused as to what’s going on just stay with us for the moment. Let’s track this down and figure out what’s going on. Let’s see what vMX2 does with label 299792…

admin@vmx2.lab> show route table mpls.0 label 301152 

mpls.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301152             *[LDP/9] 23:10:09, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Pop      
301152(S=0)        *[LDP/9] 23:10:09, metric 1
                    > to 169.254.10.3 via ge-0/0/1.0, Pop      

admin@vmx2.lab>

Alright – so it’s going to pop the label and send it out of ge-0/0/1 toward vMX3. This makes sense since we know that vMX1 was trying to send traffic to the end of the local LSP which is vMX3. So that means vMX3 will get a frame with a single MPLS label of 301296. What will it do with it?..

admin@vmx3.lab> show route table mpls.0 label 301296 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301296             *[VPN/170] 00:12:01, metric2 2, from 5.5.5.5
                    > to 169.254.10.5 via ge-0/0/1.0, label-switched-path to-vmx5

admin@vmx3.lab>

It’s going to push it right into the RSVP LSP toward vMX5. Brilliant! So it appears that the use of BGP LU is successfully stitching our LSPs together. But let’s back up a second, how is it actually doing this? Let’s look at vMX1 again…

admin@vmx1.lab> show route table inet.3 

inet.3: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2.2.2.2/32         *[LDP/9] 23:11:06, metric 1
                    > to 169.254.10.1 via ge-0/0/1.0
3.3.3.3/32         *[LDP/9] 23:10:59, metric 1
                    > to 169.254.10.1 via ge-0/0/1.0, Push 301152

admin@vmx1.lab>

Notice that we still don’t know about the 7.7.7.7 endpoint so there’s no way we can create an LSP directly to it. Let’s look at the route we’re getting through BGP in more detail…

admin@vmx1.lab> show route table inet.0 192.168.10.0/24 extensive 

inet.0: 22 destinations, 22 routes (22 active, 0 holddown, 0 hidden)
192.168.10.0/24 (1 entry, 1 announced)
TSI:
KRT in-kernel 192.168.10.0/24 -> {indirect(1048574)}
        *BGP    Preference: 170/-101
                Next hop type: Indirect, Next hop index: 0
                Address: 0xd007f30
                Next-hop reference count: 2
                Source: 3.3.3.3
                Next hop type: Router, Next hop index: 589
                Next hop: 169.254.10.1 via ge-0/0/1.0, selected
                Label operation: Push 301296, Push 301152(top)
                Label TTL action: prop-ttl, prop-ttl(top)
                Load balance label: Label 301296: None; Label 301152: None; 
                Label element ptr: 0xd007800
                Label parent element ptr: 0xd006de0
                Label element references: 1
                Label element child references: 0
                Label element lsp id: 0
                Session Id: 0x165
                Protocol next hop: 3.3.3.3
                Label operation: Push 301296
                Label TTL action: prop-ttl
                Load balance label: Label 301296: None; 
                Indirect next hop: 0xb68d630 1048574 INH Session ID: 0x167
                State: 
                Local AS: 65001 Peer AS: 65001
                Age: 12:40 	Metric2: 1 
                Validation State: unverified 
                Task: BGP_65001.3.3.3.3+50896
                Announcement bits (2): 0-KRT 6-Resolve tree 2 
                AS path: 65002 I 
                Accepted
                Route Label: 301296
                Localpref: 100
                Router ID: 3.3.3.3
                Indirect next hops: 1
                        Protocol next hop: 3.3.3.3 Metric: 1
                        Label operation: Push 301296
                        Label TTL action: prop-ttl
                        Load balance label: Label 301296: None; 
                        Indirect next hop: 0xb68d630 1048574 INH Session ID: 0x167
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 169.254.10.1 via ge-0/0/1.0
                                Session Id: 0x165
			3.3.3.3/32 Originating RIB: inet.3
			  Metric: 1			  Node path count: 1
			  Forwarding nexthops: 1
				Nexthop: 169.254.10.1 via ge-0/0/1.0
				Session Id: 0
                                        
admin@vmx1.lab>

Ah ha. So vMX thinks that route is coming from vMX3. And fortunately for us, vMX1 does know how to get to vMX3 – through the LDP LSP. And since we’re getting a route label through the BGP LU address family, vMX1 also knows that if it wants to reach that destination, it should first push the route label onto the frame before putting it in the LDP LSP. Let’s finish walking through what’s happening from vMX3 to vMX7 and then put it all together in a neat little diagram.

So we saw that vMX3 was going to stitch our LDP LSP into our RSVP LSP above. Let’s look at that again with a little more output…

admin@vmx3.lab> show route table mpls.0 label 301296                

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301296             *[VPN/170] 00:22:59, metric2 2, from 5.5.5.5
                    > to 169.254.10.5 via ge-0/0/1.0, label-switched-path to-vmx5

admin@vmx3.lab> show rsvp session lsp name to-vmx5                  
Ingress RSVP: 1 sessions
To              From            State   Rt Style Labelin Labelout LSPname 
5.5.5.5         3.3.3.3         Up       0  1 FF       -   301120 to-vmx5
Total 1 displayed, Up 1, Down 0

Egress RSVP: 1 sessions
Total 0 displayed, Up 0, Down 0

Transit RSVP: 0 sessions
Total 0 displayed, Up 0, Down 0

admin@vmx3.lab> show route forwarding-table label 301296            
Routing table: default.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
301296             user     0                    indr  1048577     2
                              169.254.10.5      Swap 301248, Push 301120(top)      590     2 ge-0/0/1.0

Routing table: __mpls-oam__.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      553     1

admin@vmx3.lab>

If we look at like 7 above – you see that when vMX3 receives label 301296 from vMX2, it’s going to push it into the RSVP LSP that we created to vMX5. If we look at that RSVP LSP we’ll see that on line 12 it will push label 301120 to get into that LSP. But when we look at what’s going on in the actual forwarding plane, we’ll see on line 26 that it want’s to swap the current top label to 301248 and then push 301120 to the top of the label stack making it the new top label. So we know where 301120 came from, but where did the 301248 come from? As you might have guessed – it comes from BGP….

admin@vmx3.lab> show route receive-protocol bgp 5.5.5.5 extensive 

inet.0: 22 destinations, 22 routes (22 active, 0 holddown, 0 hidden)
* 192.168.10.0/24 (1 entry, 1 announced)
     Accepted
     Route Label: 301248
     Nexthop: 5.5.5.5
     AS path: 65002 I 
     Entropy label capable, next hop field matches route next hop

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

admin@vmx3.lab>

It comes from vMX5! Alright – so now let’s look at what vMX4 does with that top label…

admin@vmx4.lab> show route table mpls.0 label 301120 

mpls.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301120             *[RSVP/7/1] 23:32:14, metric 1
                    > to 169.254.10.7 via ge-0/0/1.0, label-switched-path to-vmx5
301120(S=0)        *[RSVP/7/1] 23:32:14, metric 1
                    > to 169.254.10.7 via ge-0/0/1.0, label-switched-path to-vmx5

admin@vmx4.lab> show route forwarding-table label 301120 
Routing table: default.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
301120             user     0 169.254.10.7      Pop        583     2 ge-0/0/1.0
301120(S=0)        user     0 169.254.10.7      Pop        584     2 ge-0/0/1.0

Routing table: __mpls-oam__.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      540     1

admin@vmx4.lab>

So we’re still in the RSVP LSP – but since vMX4 is the PHP router we’ll be popping off the label which means that vMX5 is going to receive a frame with the label 301248. vMX5 will then….

admin@vmx5.lab> show route table mpls.0 label 301248 

mpls.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301248             *[VPN/170] 00:35:35, metric2 1, from 7.7.7.7
                    > to 169.254.10.9 via ge-0/0/1.0, Swap 301216

admin@vmx5.lab>

Notice how vMX5 is only now performing a swap operation to label 301216. AKA – it will be sending vMX6 a labelled frame with a single label. Since vMX5 will be pushing us into our tail LSP there’s no need for a second label for further stitching. Let’s finish this up by seeing what vMX6 will do….

admin@vmx6.lab> show route table mpls.0 label 301216 

mpls.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301216             *[LDP/9] 23:35:20, metric 1
                    > to 169.254.10.11 via ge-0/0/1.0, Pop      
301216(S=0)        *[LDP/9] 23:35:20, metric 1
                    > to 169.254.10.11 via ge-0/0/1.0, Pop      

admin@vmx6.lab>

vMX6 is the PHP router so it will pop the label and deliver the the naked frame to vMX7 which has the destination network 192.168.10.0/24 directly attached. Looking at this from a visual perspective, it would look like this…

Now – keep in mind that what Im depicting above is the actual datapath. The arrows above indicate where the label being sent is acted upon or used. For instance, the LDP labels are used at each hop within the LDP domain whereas the BGP-LU is used and advertised across a label domain and is only used in the stitching from label domain to label domain.

Now – let’s finish up the post by making this a hair more complicated and throwing VPNv4 into the mix. So as we did in our last post, let’s convert the interfaces facing our clients to VRF type routing-instances…

vMX1

set routing-instances customer1 instance-type vrf
set routing-instances customer1 interface ge-0/0/0.0
set routing-instances customer1 route-distinguisher 1:1
set routing-instances customer1 vrf-target target:1:1
set routing-instances customer1 vrf-table-label

vMX7

set routing-instances customer1 instance-type vrf
set routing-instances customer1 interface ge-0/0/1.0
set routing-instances customer1 route-distinguisher 1:1
set routing-instances customer1 vrf-target target:1:1
set routing-instances customer1 vrf-table-label

Now we need to setup a VPNv4 peering to allow the advertisements for each VRF to be exchanged. There are a couple of ways we could do this, but for the sake of reusing what we already have let’s just add the inet-vpn address family to the existing BGP sessions…

vMX1

set protocols bgp group internal neighbor 3.3.3.3 family inet-vpn unicast

vMX3

set protocols bgp group internal neighbor 1.1.1.1 family inet-vpn unicast    
set protocols bgp group external neighbor 5.5.5.5 family inet-vpn unicast

vMX5

set protocols bgp group internal neighbor 7.7.7.7 family inet-vpn unicast                                       
set protocols bgp group external neighbor 3.3.3.3 family inet-vpn unicast

vMX7

set protocols bgp group internal neighbor 5.5.5.5 family inet-vpn unicast

Alright – so once that’s in – let’s make sure that the peering is up and that we’re exchanging VPNv4 routes…

admin@vmx7.lab> show bgp summary 
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0               
                       0          0          0          0          0          0
bgp.l3vpn.0          
                       1          1          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
5.5.5.5               65002          5          5       0       0          55 Establ
  inet.0: 0/0/0/0
  bgp.l3vpn.0: 1/1/1/0
  customer1.inet.0: 1/1/1/0

admin@vmx7.lab> show route table customer1.inet.0       

customer1.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.10.10.0/24      *[BGP/170] 00:00:45, localpref 100, from 5.5.5.5
                      AS path: 65001 I, validation-state: unverified
                    > to 169.254.10.10 via ge-0/0/0.0, Push 302336, Push 301376(top)
192.168.10.0/24    *[Direct/0] 00:03:55
                    > via ge-0/0/1.0
192.168.10.1/32    *[Local/0] 00:03:55
                      Local via ge-0/0/1.0

admin@vmx7.lab>

Nice! The route for the left_client has arrived and it appears to have been imported correctly. If we do a quick ping test, we should see that the clients have connectivity as well…

root@left_client:~# ping 192.168.10.100 -c 5
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
64 bytes from 192.168.10.100: icmp_seq=1 ttl=57 time=4.55 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=57 time=5.11 ms
64 bytes from 192.168.10.100: icmp_seq=3 ttl=57 time=4.88 ms
64 bytes from 192.168.10.100: icmp_seq=4 ttl=57 time=5.55 ms
64 bytes from 192.168.10.100: icmp_seq=5 ttl=57 time=3.95 ms
 
--- 192.168.10.100 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 3.955/4.812/5.553/0.539 ms
root@left_client:~#

But hold on a minute. That seemed too easy – and if we look carefully at the route output – we should notice something a little off. Recall that before we layered in the VPNv4 configuration, the tail routers had to push 2 labels to get traffic from end to end. One label was a local LDP label and the other was the BGP-LU label that was used to stitch the traffic into the next label domain. From the looks of the output above, we’re still only pushing 2 labels. So where’s our VPN label?

If you picked up on that kudos. And if you know why we’re still only sending two labels – I’ll give you even more kudos. But before I talk through that, let’s try an experiment…

vMX1

delete protocols bgp group internal neighbor 3.3.3.3 family inet labeled-unicast

vMX3

delete protocols bgp group internal neighbor 1.1.1.1 family inet labeled-unicast 
delete protocols bgp group external neighbor 5.5.5.5 family inet labeled-unicast

vMX5

delete protocols bgp group internal neighbor 7.7.7.7 family inet labeled-unicast 
delete protocols bgp group external neighbor 3.3.3.3 family inet labeled-unicast

vMX7

delete protocols bgp group internal neighbor 5.5.5.5 family inet labeled-unicast

You may be surprised to see that after the BGP sessions re-peer that our traffic is once again working. If you’re entirely confused as to what’s going on – that’s totally fair at this point. What’s happened here is that we’ve stumbled upon yet another MPLS VPN architecture that falls into the “MPLS Inter-AS VPN” category of architectures. You can read more about that here but they are typically referred to as either option A, B, or C. Suffice to say – we just haven’t gotten there yet – we’re still in the process of building the blocks required for those to make sense. So at this point, I don’t want to pursue talking through what’s going on (which happens to be option B) but I do want to finish making this topology work using our existing LSP stitching. This will look a lot like a Inter-AS option C setup but for now don’t worry about what the setup looks like. We’re just going to focus on making this work and in the coming posts we’ll talk through the inter-as options in much greater detail.

So we know that we want to use LSP stitching (BGP-LU) for our VPNv4 setup so let’s go ahead and put BGP-LU back in (just do a rollback 1 on vMX1, 3, 5, 7). Once that’s done let’s figure out why this isn’t working the way we thought it would.

Let’s think back to the MPLS VPN scenarios we’ve discussed previously. Do you recall what the requirement was for those to work? Well – typically we received a VPNv4 advertisement and the next-hop for that prefix was the remote PE router. This was important because that meant that the VPN label we received as part of that advertisement was used end to end. That is – it was at the bottom of the label stack and delivered as the top label to the PE that advertised it. Let’s see what we have now…

admin@vmx7.lab> show route receive-protocol bgp 5.5.5.5 extensive table bgp.l3vpn.0 

bgp.l3vpn.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
* 1:1:10.10.10.0/24 (1 entry, 0 announced)
     Import Accepted
     Route Distinguisher: 1:1
     VPN Label: 302400
     Nexthop: 5.5.5.5
     Localpref: 100
     AS path: 65001 I 
     Communities: target:1:1

admin@vmx7.lab>

Well – I can tell you right now that doesn’t look right. The next hop is 5.5.5.5 instead of 1.1.1.1. Let’s see if the VPN label lines up with what vMX1 is sending….

admin@vmx1.lab> show route advertising-protocol bgp 3.3.3.3 extensive 

customer1.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
* 10.10.10.0/24 (1 entry, 1 announced)
 BGP group internal type Internal
     Route Distinguisher: 1:1
     VPN Label: 16
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [65001] I 
     Communities: target:1:1

admin@vmx1.lab>

Nope. vMX1 is sending a VPN label of 16. So clearly – this isn’t working like we think it should. So it looks to me like vMX7 believes the route has a next-hop of 5.5.5.5 which is vMX5. This is default behavior in this case based on how we have the BGP peering configured. But we really want that next-hop to show up as vMX1. To do this – we can tell the border routers (vMX3 and vMX5) to not change the next hop of the routes they are advertising. Let’s try doing that…

vMX3

set protocols bgp group external neighbor 5.5.5.5 multihop no-nexthop-change

vMX5

set protocols bgp group external neighbor 3.3.3.3 multihop no-nexthop-change

You’ll notice that it’s a peer level change so it’s safe to assume that this impacts all of the route advertisements from that peer. If we look now – we’ll see that we’ve lost our remote routes in each customer VRF…

admin@vmx7.lab> show route table customer1.inet.0  

customer1.inet.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.0/24    *[Direct/0] 00:27:06
                    > via ge-0/0/1.0
192.168.10.1/32    *[Local/0] 00:27:06
                      Local via ge-0/0/1.0

admin@vmx7.lab>

So let’s think about this for a second. In the first example we did above before we layered on VPNv4 – what prefix were we advertising from the tail routers? Since it was non-VPN we were simply advertising the prefix we wished to reach. But in a VPN scenario, what prefix do we need to reach and furthermore recursively resolve through the inet.3 table? The loopback of the remote PE! So we need to update our bgp-export policy to remove the client subnet and add the PE loopback address. Let’s try that out on vMX1 and see what happens…

vMX1

delete policy-options prefix-list bgp-export 10.10.10.0/24 
set policy-options prefix-list bgp-export 1.1.1.1/32

So that prefix should be coming in the form of a BGP-LU advertisement into vMX3. Let’s check…

admin@vmx3.lab> show route receive-protocol bgp 1.1.1.1 table inet.0 extensive 

inet.0: 20 destinations, 21 routes (20 active, 0 holddown, 0 hidden)
  1.1.1.1/32 (2 entries, 1 announced)
     Accepted
     Route Label: 3
     Nexthop: 1.1.1.1
     Localpref: 100
     AS path: I 
     Entropy label capable, next hop field matches route next hop

admin@vmx3.lab>

Perfect – now let’s make sure it’s being sent to vMX5….

admin@vmx3.lab> show route advertising-protocol bgp 5.5.5.5 table inet.0 

admin@vmx3.lab>

Hmmmm. Any guesses? The problem is that vMX3 already has a better router for 1.1.1.1//32…

admin@vmx3.lab> show route table inet.0 1.1.1.1/32 

inet.0: 20 destinations, 21 routes (20 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[OSPF/10] 00:34:32, metric 2
                    > to 169.254.10.2 via ge-0/0/0.0
                    [BGP/170] 00:05:59, localpref 100, from 1.1.1.1
                      AS path: I, validation-state: unverified
                    > to 169.254.10.2 via ge-0/0/0.0, Push 301328

admin@vmx3.lab>

It knows about it through OSPF. So huh. We can’t take it out of OSPF otherwise we could peer within our AS. So what’s the fix? Typically, in this type of scenario, the loopback is readvertised into BGP as the AS border router. That is, we let vMX3 and vMX5 advertise the loopback into BGP since they already have it in OSPF. So let’s delete the entry we made on vMX1…

vMX1

delete policy-options prefix-list bgp-export 1.1.1.1/32

And instead put this policy on vMX3 and vMX5…

vMX3

set policy-options prefix-list bgp-export 1.1.1.1/32 
set policy-options policy-statement bgp-export term direct from prefix-list bgp-export
set policy-options policy-statement bgp-export term direct then accept
set protocols bgp group external export bgp-export

vMX5

set policy-options prefix-list bgp-export 7.7.7.7/32 
set policy-options policy-statement bgp-export term direct from prefix-list bgp-export
set policy-options policy-statement bgp-export term direct then accept
set protocols bgp group external export bgp-export

Once that’s in – we should see that vMX3 is sending vMX5 the loopback for vMX1…

admin@vmx3.lab> show route advertising-protocol bgp 5.5.5.5 table inet.0    

inet.0: 20 destinations, 21 routes (20 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 1.1.1.1/32              Self                 2                  I

admin@vmx3.lab>

Now let’s see what vMX5 is doing with it…

admin@vmx5.lab> show route table inet.0 1.1.1.1/32 

inet.0: 20 destinations, 21 routes (20 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[OSPF/10] 00:39:45, metric 4
                    > to 169.254.10.6 via ge-0/0/0.0
                    [BGP/170] 00:02:46, MED 2, localpref 100, from 3.3.3.3
                      AS path: 65001 I, validation-state: unverified
                    > to 169.254.10.6 via ge-0/0/0.0, label-switched-path to-vmx3

admin@vmx5.lab>

AHHHH! We have the same problem. Our sin of taking a shortcut and using one flat IGP domain is coming back to bite us. Let’s fix this by breaking the OSPF domains into 3 areas…

We’ll make the middle area the core or area 0, the left area area 1, and the right area area 2. Areas 1 and 2 will be stub areas and we’ll filter out most of the prefixes and loopbacks from entering area 0 except for the loopbacks of the border routers since that’s what’s used for peering etc.

vMX1

delete protocols ospf
set protocols ospf area 0.0.0.1 stub
set protocols ospf area 0.0.0.1 interface lo0.0
set protocols ospf area 0.0.0.1 interface ge-0/0/1.0 interface-type p2p

vMX2

delete protocols ospf
set protocols ospf area 0.0.0.1 stub
set protocols ospf area 0.0.0.1 interface lo0.0
set protocols ospf area 0.0.0.1 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.1 interface ge-0/0/1.0 interface-type p2p

vMX3

delete protocols ospf
set protocols ospf area 0.0.0.1 stub
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.1 interface lo0.0
set protocols ospf area 0.0.0.1 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p
set protocols ospf area 0.0.0.0 network-summary-export ospf-export 
set policy-options prefix-list ospf-export 3.3.3.3/32
set policy-options policy-statement ospf-export term accept from prefix-list ospf-export 
set policy-options policy-statement ospf-export term accept then accept   
set policy-options policy-statement ospf-export then reject

vMX4

delete protocols ospf
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface lo0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0 interface-type p2p

vMX5

delete protocols ospf
set protocols ospf area 0.0.0.2 stub
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.2 interface lo0.0
set protocols ospf area 0.0.0.2 interface ge-0/0/1.0 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.0 network-summary-export ospf-export 
set policy-options prefix-list ospf-export 5.5.5.5/32
set policy-options policy-statement ospf-export term accept from prefix-list ospf-export 
set policy-options policy-statement ospf-export term accept then accept   
set policy-options policy-statement ospf-export then reject

vMX6

delete protocols ospf
set protocols ospf area 0.0.0.2 stub
set protocols ospf area 0.0.0.2 interface lo0.0
set protocols ospf area 0.0.0.2 interface ge-0/0/0.0 interface-type p2p
set protocols ospf area 0.0.0.2 interface ge-0/0/1.0 interface-type p2p

vMX7

delete protocols ospf
set protocols ospf area 0.0.0.2 stub
set protocols ospf area 0.0.0.2 interface lo0.0
set protocols ospf area 0.0.0.2 interface ge-0/0/0.0 interface-type p2p

Right – now that we have that in place – vMX5 should no longer be getting a 1.1.1.1/32 advertisement from OSPF so it should be passing the BGP advertisement onto vMX7…

admin@vmx5.lab> show route advertising-protocol bgp 7.7.7.7                         

inet.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 1.1.1.1/32              3.3.3.3              2       100        65001 I

admin@vmx5.lab>

Perfect! Now let’s see what vMX7 thinks about all of this…

admin@vmx7.lab> show route 1.1.1.1/32 

inet.0: 16 destinations, 16 routes (16 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[BGP/170] 04:04:16, MED 2, localpref 100, from 5.5.5.5
                      AS path: 65001 I, validation-state: unverified
                    > to 169.254.10.10 via ge-0/0/0.0, Push 302656

admin@vmx7.lab>

Well – we have the route now that we need – but the route is in inet.0 and for MPLS VPNs to work we have to recurse through the inet.3 table. So how do we get the prefix there? We could do a rib-group copy but since this was learned through BGP-LU we can simply use the resolve-vpn flag on the BGP session to move it over as we saw in the last post. Let’s try that out…

vMX1

set protocols bgp group internal neighbor 3.3.3.3 family inet labeled-unicast resolve-vpn

vMX7

set protocols bgp group internal neighbor 5.5.5.5 family inet labeled-unicast resolve-vpn

Now that that’s there on each tail router, we should see the entry in the inet.3 table…

admin@vmx7.lab> show route table inet.3 

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[BGP/170] 00:01:00, MED 2, localpref 100, from 5.5.5.5
                      AS path: 65001 I, validation-state: unverified
                    > to 169.254.10.10 via ge-0/0/0.0, Push 302656
5.5.5.5/32         *[LDP/9] 07:43:50, metric 1
                    > to 169.254.10.10 via ge-0/0/0.0, Push 301376
6.6.6.6/32         *[LDP/9] 07:43:50, metric 1
                    > to 169.254.10.10 via ge-0/0/0.0

admin@vmx7.lab>

Awesome. But if you look, you’ll see that the tail routers are no longer receiving the remote customer prefix. And if we look further, we’ll see that the boundary routers (vMX3 and vMX5) aren’t even passing it along anymore…

admin@vmx5.lab> show route advertising-protocol bgp 7.7.7.7    

inet.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 1.1.1.1/32              3.3.3.3              2       100        65001 I

admin@vmx5.lab>

If we look in the bgp.l3vpn.0 table we’ll see that it’s hidden….

admin@vmx5.lab> show route table bgp.l3vpn.0 hidden extensive 

bgp.l3vpn.0: 2 destinations, 2 routes (1 active, 0 holddown, 1 hidden)
1:1:10.10.10.0/24 (1 entry, 0 announced)
         BGP    Preference: 170/-101
                Route Distinguisher: 1:1
                Next hop type: Unusable, Next hop index: 0
                Address: 0xa0ef164
                Next-hop reference count: 1
                State: 
                Local AS: 65002 Peer AS: 65001
                Age: 4:08:09 
                Validation State: unverified 
                Task: BGP_65001.3.3.3.3+54361
                AS path: 65001 I 
                Communities: target:1:1
                Accepted
                VPN Label: 16
                Localpref: 100
                Router ID: 3.3.3.3
                Indirect next hops: 1
                        Protocol next hop: 1.1.1.1
                        Label operation: Push 16
                        Label TTL action: prop-ttl
                        Load balance label: Label 16: None; 
                        Indirect next hop: 0x0 - INH Session ID: 0x0

admin@vmx5.lab>

It’s saying that the next-hop is unusable. Now why would that be? We know about the 1.1.1.1/32 prefix in the inet.0 table…

admin@vmx5.lab> show route 1.1.1.1     

inet.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[BGP/170] 04:08:38, MED 2, localpref 100, from 3.3.3.3
                      AS path: 65001 I, validation-state: unverified
                    > to 169.254.10.6 via ge-0/0/0.0, label-switched-path to-vmx3

admin@vmx5.lab>

Our problem is that we need to know about it in the inet.3 table for it to be a valid VPNv4 route. The fix for this is the same that we just did on the tail routers. We need to tell it to move the prefixes it learns through BGP-LU from inet.0 to inet.3 when they are listed as next-hops for VPNv4 prefixes. We’ll configure that on each boundary router on the external BGP peer…

vMX3

set protocols bgp group external neighbor 5.5.5.5 family inet labeled-unicast resolve-vpn

vMX5

set protocols bgp group external neighbor 3.3.3.3 family inet labeled-unicast resolve-vpn

After that add on both border routers we should be in business…

admin@vmx5.lab> show route 1.1.1.1 

inet.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[BGP/170] 23:55:08, MED 2, localpref 100, from 3.3.3.3
                      AS path: 65001 I, validation-state: unverified
                    > to 169.254.10.6 via ge-0/0/0.0, label-switched-path to-vmx3

inet.3: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[BGP/170] 19:44:55, MED 2, localpref 100, from 3.3.3.3
                      AS path: 65001 I, validation-state: unverified
                    > to 169.254.10.6 via ge-0/0/0.0, label-switched-path to-vmx3

admin@vmx5.lab>

Alright so let’s look at our tail routers again and see what’s up with our customer VRF routing table…

admin@vmx7.lab> show route table customer1.inet.0                            

customer1.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.10.10.0/24      *[BGP/170] 19:46:07, localpref 100, from 5.5.5.5
                      AS path: 65001 I, validation-state: unverified
                    > to 169.254.10.10 via ge-0/0/0.0, Push 16, Push 302656(top)
192.168.10.0/24    *[Direct/0] 1d 03:30:41
                    > via ge-0/0/1.0
192.168.10.1/32    *[Local/0] 1d 03:30:41
                      Local via ge-0/0/1.0

admin@vmx7.lab>

Alright! We have the remote route now and if we dig a little deeper, we’ll see that it’s showing up with a protocol next-hop of 1.1.1.1 which is what we wanted…

customer1.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
10.10.10.0/24 (1 entry, 1 announced)
TSI:
KRT in-kernel 10.10.10.0/24 -> {indirect(1048575)}
        *BGP    Preference: 170/-101
                Route Distinguisher: 1:1
                Next hop type: Indirect, Next hop index: 0
                Address: 0xd007cf0
                Next-hop reference count: 3
                Source: 5.5.5.5
                Next hop type: Router, Next hop index: 581
                Next hop: 169.254.10.10 via ge-0/0/0.0, selected
                Label operation: Push 16, Push 302656(top)
                Label TTL action: prop-ttl, prop-ttl(top)
                Load balance label: Label 16: None; Label 302656: None; 
                Label element ptr: 0xd0072c0
                Label parent element ptr: 0xd006d80
                Label element references: 1
                Label element child references: 0
                Label element lsp id: 0
                Session Id: 0x192
                Protocol next hop: 1.1.1.1
                Label operation: Push 16
                Label TTL action: prop-ttl
                Load balance label: Label 16: None; 
                Indirect next hop: 0xb68bfe0 1048575 INH Session ID: 0x198
                State: 
                Local AS: 65002 Peer AS: 65002
                Age: 19:46:04 	Metric2: 4 
                Validation State: unverified 
                Task: BGP_65002.5.5.5.5+49544
                Announcement bits (1): 0-KRT 
                AS path: 65001 I 
                Communities: target:1:1
                Import Accepted
                VPN Label: 16
                Localpref: 100
                Router ID: 5.5.5.5
                Primary Routing Table bgp.l3vpn.0
                Indirect next hops: 1
                        Protocol next hop: 1.1.1.1 Metric: 4
                        Label operation: Push 16
                        Label TTL action: prop-ttl
                        Load balance label: Label 16: None; 
                        Indirect next hop: 0xb68bfe0 1048575 INH Session ID: 0x198
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 169.254.10.10 via ge-0/0/0.0
                                Session Id: 0x192
			1.1.1.1/32 Originating RIB: inet.3
			  Metric: 4			  Node path count: 1
                                        
admin@vmx7.lab>

Cool – but take a look at that label operation. We’re still only pushing two labels at this point which doesn’t align with our thinking of the LSPs as being transport and the VPN label as being end to end. So what’s going on now? If we step back a bit and think about what we’re trying to do – let’s remember what had to happen in our non-VPN example we did above. To get from label domain to label domain we had to think that our next hop for the resolved route was within our label domain. That is – our ingress router had to believe that the resolved next hop lived within it’s own AS. Now we know that our VPN route has had it’s next-hop preserved as 1.1.1.1 on vMX7 which is what we wanted, but what does the route for 1.1.1.1/32 actually look like…

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
1.1.1.1/32 (1 entry, 1 announced)
        *BGP    Preference: 170/-101
                Next hop type: Indirect, Next hop index: 0
                Address: 0xd006d90
                Next-hop reference count: 4
                Source: 5.5.5.5
                Next hop type: Router, Next hop index: 610
                Next hop: 169.254.10.10 via ge-0/0/0.0, selected
                Label operation: Push 302656
                Label TTL action: prop-ttl
                Load balance label: Label 302656: None; 
                Label element ptr: 0xd007500
                Label parent element ptr: 0x0
                Label element references: 2
                Label element child references: 1
                Label element lsp id: 0
                Session Id: 0x192
                Protocol next hop: 3.3.3.3
                Label operation: Push 302656
                Label TTL action: prop-ttl
                Load balance label: Label 302656: None; 
                Indirect next hop: 0xb68cb90 1048574 INH Session ID: 0x19b
                State: 
                Local AS: 65002 Peer AS: 65002
                Age: 51         Metric: 2       Metric2: 4 
                Validation State: unverified 
                Task: BGP_65002.5.5.5.5+49544
                Announcement bits (3): 2-Resolve tree 1 3-Resolve tree 2 4-Resolve_IGP_FRR task 
                AS path: 65001 I 
                Accepted
                Route Label: 302656
                Localpref: 100
                Router ID: 5.5.5.5
                Primary Routing Table inet.0
                Indirect next hops: 1
                        Protocol next hop: 3.3.3.3 Metric: 4
                        Label operation: Push 302656
                        Label TTL action: prop-ttl
                        Load balance label: Label 302656: None; 
                        Indirect next hop: 0xb68cb90 1048574 INH Session ID: 0x19b
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 169.254.10.10 via ge-0/0/0.0
                                Session Id: 0x192
                        3.3.3.3/32 Originating RIB: inet.0
                          Metric: 4                       Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: 169.254.10.10 via ge-0/0/0.0
                                Session Id: 192

admin@vmx7.lab>

Aha! The protocol next-hop we’re getting for that route is the loopback of vMX3 which is not within our label domain/AS. If we want BGP-LU to do our stitching as part of the transport LSP we need to fix this. To fix this, we need to tell each border router to do a next-hop self on the BGP-LU route that’s being advertised to the tail router in it’s own AS. Let’s try that out…

vMX3 and vMX5

set policy-options policy-statement nhs term nhs from family inet 
set policy-options policy-statement nhs term nhs then next-hop self
set protocols bgp group internal export nhs

Now let’s look at that route again…

admin@vmx7.lab> show route 1.1.1.1 table inet.3 extensive    

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
1.1.1.1/32 (1 entry, 1 announced)
        *BGP    Preference: 170/-101
                Next hop type: Indirect, Next hop index: 0
                Address: 0xd0078d0
                Next-hop reference count: 4
                Source: 5.5.5.5
                Next hop type: Router, Next hop index: 613
                Next hop: 169.254.10.10 via ge-0/0/0.0, selected
                Label operation: Push 302496, Push 301376(top)
                Label TTL action: prop-ttl, prop-ttl(top)
                Load balance label: Label 302496: None; Label 301376: None; 
                Label element ptr: 0xd007f20
                Label parent element ptr: 0xd006e40
                Label element references: 2
                Label element child references: 1
                Label element lsp id: 0
                Session Id: 0x192
                Protocol next hop: 5.5.5.5
                Label operation: Push 302496
                Label TTL action: prop-ttl
                Load balance label: Label 302496: None; 
                Indirect next hop: 0xb68c640 1048576 INH Session ID: 0x19c
                State: 
                Local AS: 65002 Peer AS: 65002
                Age: 9  Metric: 2       Metric2: 1 
                Validation State: unverified 
                Task: BGP_65002.5.5.5.5+49544
                Announcement bits (3): 2-Resolve tree 1 3-Resolve tree 2 4-Resolve_IGP_FRR task 
                AS path: 65001 I 
                Accepted
                Route Label: 302496
                Localpref: 100
                Router ID: 5.5.5.5
                Primary Routing Table inet.0
                Indirect next hops: 1
                        Protocol next hop: 5.5.5.5 Metric: 1
                        Label operation: Push 302496
                        Label TTL action: prop-ttl
                        Load balance label: Label 302496: None; 
                        Indirect next hop: 0xb68c640 1048576 INH Session ID: 0x19c
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 169.254.10.10 via ge-0/0/0.0
                                Session Id: 0x192
                        5.5.5.5/32 Originating RIB: inet.3
                          Metric: 1                       Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: 169.254.10.10 via ge-0/0/0.0
                                Session Id: 0

admin@vmx7.lab>

Bingo! Now let’s look at the route in the customer VRF for the remote prefix…

admin@vmx7.lab> show route table customer1.inet.0    

customer1.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.10.10.0/24      *[BGP/170] 00:02:36, localpref 100, from 5.5.5.5
                      AS path: 65001 I, validation-state: unverified
                    > to 169.254.10.10 via ge-0/0/0.0, Push 16, Push 302496, Push 301376(top)
192.168.10.0/24    *[Direct/0] 1d 04:38:57
                    > via ge-0/0/1.0
192.168.10.1/32    *[Local/0] 1d 04:38:57
                      Local via ge-0/0/1.0

admin@vmx7.lab>

And at long last – the routing tables look as they should. We now have the 3 labels we were looking for. Whew. But… Our ping still isn’t working….

left_client:~# ping 192.168.10.100 -c 5
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.

--- 192.168.10.100 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms

left_client:~#

Grumble. Ok. At this point, let’s do a label trace from tail router to tail router and see what’s going on now. From the output above, we know that vMX6 is going to get a top label of 301376. Presumably this is for the LDP LSP but let’s make sure…

admin@vmx7.lab> show ldp database 
Input label database, 7.7.7.7:0--6.6.6.6:0
Labels received: 3
  Label     Prefix
 301376      5.5.5.5/32
      3      6.6.6.6/32
 301408      7.7.7.7/32

Output label database, 7.7.7.7:0--6.6.6.6:0
Labels advertised: 3
  Label     Prefix
 299984      5.5.5.5/32
 299968      6.6.6.6/32
      3      7.7.7.7/32

admin@vmx7.lab>

Yep – ok, that checks out. Let’s see what vMX6 does with this label now…

admin@vmx6.lab> show route table mpls.0 label 301376 

mpls.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

301376             *[LDP/9] 1d 04:43:40, metric 1
                    > to 169.254.10.8 via ge-0/0/0.0, Pop      
301376(S=0)        *[LDP/9] 1d 04:43:40, metric 1
                    > to 169.254.10.8 via ge-0/0/0.0, Pop      

admin@vmx6.lab>

Alright – it is the PHP router for the LDP LSP so it will pop that label and pass along what’s left to vMX5. So vMX5 will get a top label of 302496 which if we go back and look at vMX7 again, we’ll see is the BGP-LU stitching label that it received…

admin@vmx7.lab> show route receive-protocol bgp 5.5.5.5 1.1.1.1/32 table inet.3 extensive 

inet.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
* 1.1.1.1/32 (1 entry, 1 announced)
     Accepted
     Route Label: 302496
     Nexthop: 5.5.5.5
     MED: 2
     Localpref: 100
     AS path: 65001 I 
     Entropy label capable, next hop field matches route next hop

admin@vmx7.lab>

Checks out. Let’s see what vMX5 will do with that label now…

admin@vmx5.lab> show route table mpls.0 label 302496 extensive 

mpls.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)
302496 (1 entry, 1 announced)
TSI:
KRT in-kernel 302496 /52 -> {indirect(1048575)}
        *VPN    Preference: 170
                Next hop type: Indirect, Next hop index: 0
                Address: 0xd007510
                Next-hop reference count: 2
                Source: 3.3.3.3
                Next hop type: Router, Next hop index: 589
                Next hop: 169.254.10.6 via ge-0/0/0.0, selected
                Label-switched-path to-vmx3
                Label operation: Swap 302656, Push 301552(top)
                Label TTL action: prop-ttl, prop-ttl(top)
                Load balance label: Label 302656: None; Label 301552: None; 
                Label element ptr: 0xd0076e0
                Label parent element ptr: 0xd007c20
                Label element references: 1
                Label element child references: 0
                Label element lsp id: 0
                Session Id: 0x19a
                Protocol next hop: 3.3.3.3
                Label operation: Swap 302656
                Load balance label: Label 302656: None; 
                Indirect next hop: 0xb68d300 1048575 INH Session ID: 0x1a5
                State: 
                Local AS: 65002 
                Age: 7:57 	Metric2: 2 
                Validation State: unverified 
                Task: BGP_RT_Background
                Announcement bits (1): 1-KRT 
                AS path: 65001 I 
		Ref Cnt: 1
                Indirect next hops: 1
                        Protocol next hop: 3.3.3.3 Metric: 2
                        Label operation: Swap 302656
                        Load balance label: Label 302656: None; 
                        Indirect next hop: 0xb68d300 1048575 INH Session ID: 0x1a5
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 169.254.10.6 via ge-0/0/0.0
                                Session Id: 0x19a
			3.3.3.3/32 Originating RIB: inet.3
			  Metric: 2			  Node path count: 1
			  Forwarding nexthops: 1
				Nexthop: 169.254.10.6 via ge-0/0/0.0
				Session Id: 0

admin@vmx5.lab>

Since this is going into an RSVP LSP we have to do the extensive output to see the label operation but as you can see it’s going to swap 302496 for 302656 and then push another label of 301552. So vMX4 will be getting a top label of 301552 followed by 302656 and then our VPN label of 16. Ok so what does vMX4 do with that top label?

admin@vmx4.lab> show route table mpls.0 label 301552 extensive 

mpls.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
301552 (1 entry, 1 announced)
TSI:
KRT in-kernel 301552 /52 -> {Pop       Flags acct EL-capable}
        *RSVP   Preference: 7/1
                Next hop type: Router, Next hop index: 585
                Address: 0xd007450
                Next-hop reference count: 2
                Next hop: 169.254.10.4 via ge-0/0/0.0, selected
                Label-switched-path to-vmx3
                Label operation: Pop      
                Load balance label: None; 
                Label element ptr: 0xd007560
                Label parent element ptr: 0x0
                Label element references: 2
                Label element child references: 0
                Label element lsp id: 0
                Session Id: 0x181
                State: 
                Age: 1d 4:48:51         Metric: 1 
                Validation State: unverified 
                Task: RSVP
                Announcement bits (1): 1-KRT 
                AS path: I 

301552(S=0) (1 entry, 1 announced)
TSI:
KRT in-kernel 301552 /56 -> {Pop       Flags acct EL-capable}
        *RSVP   Preference: 7/1
                Next hop type: Router, Next hop index: 586
                Address: 0xd006f70
                Next-hop reference count: 2
                Next hop: 169.254.10.4 via ge-0/0/0.0, selected
                Label-switched-path to-vmx3
                Label operation: Pop      
                Load balance label: None; 
                Label element ptr: 0xd006fc0
                Label parent element ptr: 0x0
                Label element references: 1
                Label element child references: 0
                Label element lsp id: 0
                Session Id: 0x181
                State: 
                Age: 1d 4:48:51         Metric: 1 
                Validation State: unverified 
                Task: RSVP
                Announcement bits (1): 1-KRT 
                AS path: I 

admin@vmx4.lab>

Alright – to it’s going to pop it. Which again – makes sense since we are now in the RSVP domain and vMX4 would be the PHP router for this LSP to vMX3. So vMX3 is going to get a top label of 302656. Let’s see what vMX3 does with that…

admin@vmx3.lab> show route table mpls.0 label 302656 

mpls.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

302656             *[VPN/170] 1d 01:13:08
                    > to 169.254.10.2 via ge-0/0/0.0, Pop      
302656(S=0)        *[VPN/170] 1d 01:13:08
                    > to 169.254.10.2 via ge-0/0/0.0, Pop      

admin@vmx3.lab>

Pop… Huh. That doesn’t seem right. That would mean that vMX2 would be getting just the VPN label. Let’s do a packet capture between vMX3 and vMX4 to see if that’s whats going on…

Note: Im assuming in this case that you’re trying to do a ping from right_client to left_client. I realized as I was writing this that I’ve been doing all my other tests from left to right so apologies for the change up as part of this trace.

Alright – so sure enough – that’s what it’s doing. As we know – the VPN label is only relevant to the PE that’s advertising this. In this case, vMX2 is going to be getting a frame with a single VPN label as the top frame. And if our assumptions are correct, vMX2 will not know what to do with it…

admin@vmx2.lab> show route table mpls.0 label 16 

admin@vmx2.lab> show route forwarding-table label 16 
Routing table: default.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd       50     1

Routing table: __mpls-oam__.mpls
MPLS:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      553     1

admin@vmx2.lab>

Yep – so it’s going to discard. So what’s going on now? The problem seems to be on vMX3. So let’s take a look and see what’s going on there. We know that vMX3 is responsible for stitching the traffic coming from the RSVP LSP into the LDP LSP to vMX1. So let’s take a look at vMX3 and see if we have a LSP to vMX1…

admin@vmx3.lab> show route table inet.3 

inet.3: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[LDP/9] 1d 01:21:19, metric 1
                    > to 169.254.10.2 via ge-0/0/0.0, Push 301328
2.2.2.2/32         *[LDP/9] 1d 01:21:19, metric 1
                    > to 169.254.10.2 via ge-0/0/0.0
5.5.5.5/32         *[RSVP/7/1] 1d 04:58:22, metric 2
                    > to 169.254.10.5 via ge-0/0/1.0, label-switched-path to-vmx5
7.7.7.7/32         *[BGP/170] 21:10:16, MED 2, localpref 100, from 5.5.5.5
                      AS path: 65002 I, validation-state: unverified
                    > to 169.254.10.5 via ge-0/0/1.0, label-switched-path to-vmx5

admin@vmx3.lab>

Alright so that looks good. So what’s going on here? Well – this sentence from the Juniper documentation describes the issue pretty well…

The only use for inet.3 or inet6.3 is to permit BGP to perform next-hop resolution.

Put more simply – BGP is the only protocol that will recursively resolve it’s next hop in the inet3. table. So if vMX3 can’t resolve the LSP in inet.3 then what does it do? It looks in inet.0…

admin@vmx3.lab> show route table inet.0 1.1.1.1/32   

inet.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         *[OSPF/10] 1d 01:33:06, metric 2
                    > to 169.254.10.2 via ge-0/0/0.0

admin@vmx3.lab>

And in inet.0 we have a route to 1.1.1.1/32 through OSPF. Since vMX3 thinks that it’s best path to 1.1.1.1/32 is through OSPF it will send a BGP-LU label to vMX5 indicating that it should pop off the label so that it can forward the packet normally. The problem is that once the frame gets to vMX3 – there are still two labels on the frame meaning that vMX4 will receive a label with a single VPN label which it has no idea what to do with.

The fix for this is to get the forwarding information required from inet.3 into the inet.0 table. There are many many many ways to do this, but in my opinion, the easiest and the safest way to do this is by telling the router to use routes from inet.3 for forwarding only…

vMX3 and vMX5

set protocols mpls traffic-engineering mpls-forwarding

After we add that our ping should start working! Let’s now look at the inet.0 table on vMX3 again for the 1.1.1.1/32 prefix…

admin@vmx3.lab> show route table inet.0 1.1.1.1/32 

inet.0: 17 destinations, 20 routes (17 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

1.1.1.1/32         @[OSPF/10] 00:00:38, metric 2
                    > to 169.254.10.2 via ge-0/0/0.0
                   #[LDP/9] 00:00:38, metric 1
                    > to 169.254.10.2 via ge-0/0/0.0, Push 301328

admin@vmx3.lab>

Now we can see that the LDP router (with the lower priority) is in the table and has the # next to it indicating that it’s being used for just forwarding traffic and not as part of any route selection protocol. So now our picture should look like this…

Alright – well that was a long road to get here. But we finally made it and walking through troubleshooting something like this is always helpful in my opinion. Clearly, we had to jump through some hoops because of the way our BGP peering was setup so in the next post we’ll talk through some other options for how to do this. If any of this seemed confusing to you, I’d encourage you to read the post on BPG-LU as well as the one on MPLS VPNs with BGP LU.