ACI | Network 172.17.0.0/16 unreachable from APIC

Some days ago I had to configure a radius server on an APIC cluster. This server was addressed in the 172.17.0.0/16 range and it worked well for the spines and leaves of the fabric but not for any APIC of the cluster. From the APIC, we had the “unreachable” faults:

From the CLI, the problem seemed more clear:

APIC1# bash
admin@APIC1:~> ping 172.17.1.50
PING 172.17.1.50 (172.17.1.50) 56(84) bytes of data.
From 172.17.0.1 icmp_seq=1 Destination Host Unreachable
From 172.17.0.1 icmp_seq=2 Destination Host Unreachable
From 172.17.0.1 icmp_seq=3 Destination Host Unreachable
^C
— 172.17.1.50 ping statistics —
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4090ms
pipe 4

admin@APIC1:~> traceroute 172.17.1.50
traceroute to 172.17.1.50 (172.17.1.50), 30 hops max, 60 byte packets
1 172.17.0.1 (172.17.0.1) 3049.452 ms !H 3049.385 ms !H 3049.357 ms !H

The traceroute stopped at 172.17.0.1, not the out-of-band gateway configured.

admin@APIC1:~> route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.200.200.1 0.0.0.0 UG 16 0 0 oobmgmt
10.30.0.0 10.30.0.30 255.255.0.0 UG 0 0 0 bond0.3914
10.30.0.30 0.0.0.0 255.255.255.255 UH 0 0 0 bond0.3914
10.200.200.0 0.0.0.0 255.255.255.0 U 0 0 0 oobmgmt
169.254.1.0 0.0.0.0 255.255.255.0 U 0 0 0 teplo-1
169.254.254.0 0.0.0.0 255.255.255.0 U 0 0 0 lxcbr0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0

The radius servers where in the docker range of the APIC, so the docker route is preferred…. as explained in this enhancement request https://bst.cloudapps.cisco.com/bugsearch/bug/CSCve84297, a service cannot be reached by using the APIC out-of-band management that exists within the 172.17.0.0/16 sub-net.

Solution

The solution is to change the Docker bridge-IP of the APIC, it can be performed from a simple API call (the most complicated action would be to find another /16 subnet available :) ). The detail of the API call is below, the script is replacing the default 172.17.0.1/16 by another IP in the range 10.255.0.0/16 :

POST {{apic-url}}/api/plgnhandler/mo/.xml

<?xml version=”1.0″ encoding=”UTF-8″?>

<apPluginPolContr>

    <apContainerPol containerBip=”10.255.0.1/16“/>

</apPluginPolContr>

Conclusion

Be careful and change the internal network of the docker service before running under production if you have this internal subnet used on your data center network (even if the action of changing it is not impacting the production, we always prefer to spot and correct this kind of problem before).

Benoit

Network engineer at CNS Communications. CCIE #47705, focused on R&S, Data Center, SD-WAN & Automation.

More Posts - Website

Follow Me: