The second step of this workshop lab will set up the iBGP so that our routers can exchange internal and external prefixes with each other.
The next step is to configure iBGP mesh between the four routers in each autonomous system. We’ll use a route-reflector set up to handle this, as this is very common practice today, and full mesh iBGP does not scale, as was covered in the presentations.
We will make the core router the route reflector, as is standard practice. The border router will be a client, the access router will be a client, and the peering router will be another client. Before setting up the iBGP route reflector, we need to consider the following:
the border routers connect to transit ISPs, so can in theory receive the entire global routing table from the upstream (either as a default route, or as every individual announced prefix).
the peering routers connect to private and public peers - the peering router should only hear routes that the peers should be able to receive. A common mistake is for peering routers in service provider backbones to carry the full Internet routing table, resulting in bandwidth hijack, misrouted traffic, and so on.
the access routers only connect to end user customers. The router typically does not need to carry the full BGP table from the Internet. All it needs is a default route pointing to the core, and some visibility of the prefixes originated within the rest of the autonomous system.
We will initialise BGP on all the routers in our autonomous system, changing the Cisco defaults to those which are considered industry best practices.
These will enable multi-protocol BGP, set metric calculation to be deterministic1, and fix the BGP default protocol distance so that it is much higher (i.e. less important) than any other routing protocol.
Also note that Cisco IOS assumes that all peers will exchange IPv4 prefixes even if they are running another IP protocol - of course this makes no sense, so we will turn off that assumption too.
router bgp X0
bgp log-neighbor-changes
bgp deterministic-med
no bgp default ipv4-unicast
!
address-family ipv4
distance bgp 200 200 200
!
address-family ipv6
distance bgp 200 200 200
!
We will be using BGP Communities for all our BGP configurations. We want to tag prefixes all prefixes introduced into our network, now and in future lab exercises. For now, we will tag our own prefixes as we introduce them into our iBGP. This way we can manage what we announce in future to all external networks simply by applying community filters, removing the need to maintain prefix-lists, and therefore simplifying the management of the network configuration.
The communities we are going to adopt are as follows:
| Prefix Type | Community | Description |
|---|---|---|
| Our aggregate | AS:1000 | Our assigned address block(s) |
| Subnets of our aggregate | AS:1001 | Any subnets we carve out of our address block(s) |
| Customer independent addresses | AS:1005 | Any address space customers are assigned independently of what they receive from us |
Each group will now set up these community definitions on their Border, Core, Peering and Access routers (not the Customer router, which is not in the group’s AS).
Here is a configuration example using Cisco IOS named community lists:
ip community-list standard aggregate permit X0:1000
ip community-list standard subnets permit X0:1001
ip community-list standard customer-pi permit X0:1005In future labs we will add other community definitions as we require them.
We also need to tell Cisco IOS to use the industry format for BGP communities (as described in RFC1998) rather than the specification standard (as described in RFC1997). The latter represents communities as a 32-bit integer; the former represents communities as two 16-bit integers separated by a colon which is deemed more human readable by network operators.
The following configuration needs to be done on all the routers which have been configured with BGP (Border, Core, Peering, Access). The new-format is actually referring to the community format used in RFC1998, rather than in the specification in RFC1997 - it’s not a new format as such, but an “easier for humans” format.
ip bgp-community new-format
Focusing now on the core router, we will create two peer groups, one for the standard iBGP mesh (passing full routes), the other for use with peering and access routers (passing only local AS routes). As discussed in the presentation, make sure you enter all the IPv4 neighbour configuration under the IPv4 address family, and all the IPv6 neighbour configuration under the IPv6 address family. Cisco IOS will sort out whether the configuration should fit under the preamble or under the relevant address family itself.
Let’s set up the full routes peer-group first - this one passes on all routes it hears from all BGP neighbours, even those from outside our autonomous system. First for IPv4:
router bgp X0
!
address-family ipv4
neighbor ibgpv4-full peer-group
neighbor ibgpv4-full description Full Routes
neighbor ibgpv4-full remote-as X0
neighbor ibgpv4-full update-source loopback0
neighbor ibgpv4-full next-hop-self
neighbor ibgpv4-full password BGPlab
neighbor ibgpv4-full send-community
neighbor ibgpv4-full route-reflector-client
!And then for IPv6 - notice that we use the same name for the peer-group as we did for IPv4, but we include “v6” in the name instead of “v4”. This makes it easier to identify peer-groups with the same function but applying to different address families.
router bgp X0
!
address-family ipv6
neighbor ibgpv6-full peer-group
neighbor ibgpv6-full description Full Routes
neighbor ibgpv6-full remote-as X0
neighbor ibgpv6-full update-source loopback0
neighbor ibgpv6-full next-hop-self
neighbor ibgpv6-full password BGPlab
neighbor ibgpv6-full send-community
neighbor ibgpv6-full route-reflector-client
!Next we set up the partial routes peer-group - this one only passes on routes which are local to the autonomous system. We accomplish this by creating a route-map called partial-iBGP which only allows locally originated routes in our iBGP (we also allow customer independent address space as well):
route-map partial-iBGP permit 5
description Prefixes for partial iBGP
match community aggregate
match community subnets
match community customer-pi
!Then we will set up the peer-group. It is identical to the full routes peer-group, save for the addition of the line including the route-map. First for IPv4:
router bgp X0
!
address-family ipv4
neighbor ibgpv4-partial peer-group
neighbor ibgpv4-partial description Local Routes only
neighbor ibgpv4-partial remote-as X0
neighbor ibgpv4-partial update-source loopback0
neighbor ibgpv4-partial next-hop-self
neighbor ibgpv4-partial password BGPlab
neighbor ibgpv4-partial send-community
neighbor ibgpv4-partial route-reflector-client
neighbor ibgpv4-partial route-map partial-iBGP out
!and then for IPv6 (noting that the route-map can be used in both IPv4 and IPv6 peer-groups):
router bgp X0
!
address-family ipv6
neighbor ibgpv6-partial peer-group
neighbor ibgpv6-partial description Local Routes only
neighbor ibgpv6-partial remote-as X0
neighbor ibgpv6-partial update-source loopback0
neighbor ibgpv6-partial next-hop-self
neighbor ibgpv6-partial password BGPlab
neighbor ibgpv6-partial send-community
neighbor ibgpv6-partial route-reflector-client
neighbor ibgpv6-partial route-map partial-iBGP out
!Finally we apply the newly created peer-groups to the iBGP peers of the core router. First we do this for IPv4:
router bgp X0
!
address-family ipv4
neighbor 100.68.X.1 peer-group ibgpv4-full
neighbor 100.68.X.1 description iBGP with Border
neighbor 100.68.X.1 activate
neighbor 100.68.X.3 peer-group ibgpv4-partial
neighbor 100.68.X.3 description iBGP with Peering
neighbor 100.68.X.3 activate
neighbor 100.68.X.4 peer-group ibgpv4-partial
neighbor 100.68.X.4 description iBGP with Access
neighbor 100.68.X.4 activate
!and then we do it for IPv6:
router bgp X0
!
address-family ipv6
neighbor 2001:DB8:X::1 peer-group ibgpv6-full
neighbor 2001:DB8:X::1 description iBGP with Border
neighbor 2001:DB8:X::1 activate
neighbor 2001:DB8:X::3 peer-group ibgpv6-partial
neighbor 2001:DB8:X::3 description iBGP with Peering
neighbor 2001:DB8:X::3 activate
neighbor 2001:DB8:X::4 peer-group ibgpv6-partial
neighbor 2001:DB8:X::4 description iBGP with Access
neighbor 2001:DB8:X::4 activate
!The only difference between the peer-group ibgpv4-partial and the peer-group ibgpv4-full is that the former has that one additional line only permitting prefixes originated by the local AS to go to that router. So if the upstream provider sends a default route, or any prefixes from the global BGP table, they will now not make their way to the peering router.
We now turn to the border, access and peering routers, and will configure iBGP on those as well. We’ll follow the same ideas as we used for the Core router, only the border, access and peering routers are route reflector clients. Here is a configuration example (applicable to any of the border, access or peering routers).
First off we set up and apply the IPv4 peer-group:
router bgp X0
!
address-family ipv4
neighbor ibgpv4-rr peer-group
neighbor ibgpv4-rr description iBGP with RR
neighbor ibgpv4-rr remote-as X0
neighbor ibgpv4-rr update-source loopback0
neighbor ibgpv4-rr next-hop-self
neighbor ibgpv4-rr password BGPlab
neighbor ibgpv4-rr send-community
!
neighbor 100.68.X.2 peer-group ibgpv4-rr
neighbor 100.68.X.2 description iBGP with Core
neighbor 100.68.X.2 activate
!And then we set up and apply the IPv6 peer-group - use the same principle as we did earlier on the Core Router: use the same peer-group name, just add in v6 to distinguish the two address families:
router bgp X0
!
address-family ipv6
neighbor ibgpv6-rr peer-group
neighbor ibgpv6-rr description iBGP with RR
neighbor ibgpv6-rr remote-as X0
neighbor ibgpv6-rr update-source loopback0
neighbor ibgpv6-rr next-hop-self
neighbor ibgpv6-rr password BGPlab
neighbor ibgpv6-rr send-community
!
neighbor 2001:DB8:X::2 peer-group ibgpv6-rr
neighbor 2001:DB8:X::2 description iBGP with Core
neighbor 2001:DB8:X::2 activate
!
We will now originate our prefixes into iBGP. We will only do this on the core router (common practice is to originate prefixes on the core routers in a network operator’s backbone, never on the peering or border routers).
When we introduce our aggregate into our IBGP we are going to tag it with a specific community. This helps us build our internal policy to flag which prefixes are forwarded by BGP to which peer. Here is an example route-map which sets the “aggregate” community:
route-map set-aggregate-community permit 5
description Set community on Aggregate
set community X0:1000
!We now add in network statements to cover our IPv4 and IPv6 address blocks. Here is an example of the configuration needed, noting that the route-map we just created is applied to the network statement:
router bgp X0
!
address-family ipv4
network 100.68.X.0 mask 255.255.255.0 route-map set-aggregate-community
!
address-family ipv6
network 2001:DB8:X::/48 route-map set-aggregate-community
!
ip route 100.68.X.0 255.255.255.0 Null0
ipv6 route 2001:DB8:X::/48 Null0Don’t forget the pull up routes for the aggregate - the network statement in Cisco IOS only tells BGP to put that address block into BGP if the matched block is in the global RIB - and the simplest way to install it in the global RIB is to set up a static route pointing to the Null0 interface.
Once this is done, check the BGP table. Run the:
show ip bgp communityand
show bgp ipv6 unicast communitycommands. What do you see?
All being well, you should see your IPv4 and IPv6 aggregates displayed in the output. This command shows you all prefixes which have a BGP community set on them.
Then show the actual BGP table entry:
show ip bgp 100.68.X.0and
show bgp ipv6 unicast 2001:DB8:X::/48You will now see an extra line in the output, indicating which community has been set. It should be in the format AS:1000.
We also need to originate the server subnet from the core router. We could have done this using IS-IS, but if we follow the best practice of minimising the number of prefixes carried in IS-IS, we can just as easily carry the prefix around in iBGP.
As before, we create the route-map we need to set the server’s address block community. Note that it is a subnet of our aggregate, so that’s the community we use. Here is an example:
route-map set-subnet-community permit 5
description Set community on local subnets
set community X0:1001
!We now add the necessary network statement to cover the IPv4 and IPv6 subnet used for the server appliance, including the route-map statement so that the community gets set. Here is an example of the configuration needed:
router bgp X0
!
address-family ipv4
network 100.68.X.28 mask 255.255.255.252 route-map set-subnet-community
!
address-family ipv6
network 2001:DB8:X:21::/64 route-map set-subnet-community
!Once this is done, check the BGP table on the Core Router. Again run the:
show ip bgp communityand
show bgp ipv6 unicast communitycommands. What do you see now?
All being well, you should see the server subnet IPv4 and IPv6 route displayed in the output, as well as the aggregate generated earlier.
The peering router is just that, a router that peers with other network operators. It does not provide any transit. The peers should only see the routes that you want them to see (basically what you originate from your AS and any customers). We’ve made sure of this by putting in a route filter on the core router route-reflector so that the peering router can only see locally originated routes.
It is also a good idea to null route the default route, as we will soon be distributing a default route around the AS using IS-IS. So on the peering router we now do:
ip route 0.0.0.0 0.0.0.0 null0
ipv6 route ::/0 null0Now, once we set up the IXP connectivity later on in the workshop, if any of the IXP participants point a default route to the local network, the traffic will simply be dumped in the Null interface of the peering router. Only traffic for specific destinations which are available in the routing table on the IXP peering router will be forwarded to the rest of the network. This is a very important network security requirement.
We will now set up static routes on the Access Router pointing towards the Customer router and have those routes announced by IBGP so that the rest of our AS can see the Customer router (and vice-versa), something that has not been possible up to now.
Take the address subnets assigned for the “End User Space 1” (a /26 for IPv4 and a /52 for IPv6) and on the Access router, point static routes to the Customer router point-to-point link address. Like this:
ip route 100.68.X.64 255.255.255.192 100.68.X.35
ipv6 route 2001:DB8:X:4000::/52 2001:DB8:X:31::1
!Next we create a route-map so that the customer route is tagged with an appropriate community. Note that it is a subnet of our aggregate, so that’s the community we use. Here is an example (same as used for the Core Router earlier):
route-map set-subnet-community permit 5
description Set community on local subnets
set community X0:1001
!And then put the same address subnets into BGP via a network statement, not forgetting the route-map to set the community. Once the static route is functional (the other end of the link is reachable), BGP will announce the subnets.
Here is an example configuration for the Access router.
router bgp X0
address-family ipv4
network 100.68.X.64 mask 255.255.255.192 route-map set-subnet-community
!
address-family ipv6
network 2001:DB8:X:4000::/52 route-map set-subnet-community
!Once this has been configured you should then see the customer IPv4 /26 and IPv6 /52 visible in the iBGP across the whole AS. Check on the Border, Peering and Core routers, and make sure the prefix is visible. Use these commands:
show ip bgp
show bgp ipv6 unicast
To complete the connectivity for the customer end-site we need to set up a static default route (IPv4 and IPv6) on the Customer router pointing to the Access router. In an earlier lab, we already created a source anchor point for the address space we have assigned to the customer using a Loopback interface.
All we need to do now is point the entire IPv4 and IPv6 blocks to the Null0 interface. We have a target to test connectivity to, as well as following best practices by Null routing any address space which is not in use.
Here is an example for the Customer router:
ip route 0.0.0.0 0.0.0.0 100.68.X.34
ip route 100.68.X.64 255.255.255.192 Null0
!
ipv6 route ::/0 2001:DB8:X:31::
ipv6 route 2001:DB8:X:4000::/52 Null0
!
Once this has been completed, test the connectivity across the routers in your AS again. Can you reach them all still? Also check that the customer routes and the AS’s aggregate are known within iBGP as well. Do you remember the IPv4 and IPv6 BGP commands you need to use?
Please consult the BGP Attributes presentation for background on what “Deterministic MED” is and why it is important to set it.↩︎