Avoiding TCP/UDP Port Exhaustion Cisco Router

This blog will talk about ephemeral port exhaustion. 

In a  Network Translation and with PAT  ( port address transalation ) , a client  source address and port numbers are mapped to  a single NAT'd source. This is commonly called SNAT ( source NAT'ing ). Doing this for a hand full of clients machine is not critical. Since the src_port is expiring and temporary used and forever changing.

This process of temporary using a src_port is called "ephemeral". So take this machine that I'm typing this blog on;

 sahel01: ~ kfelix$ netstat -an | head
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)   
tcp4       0      0  112.22.1.172.58430      173.194.35.175.80      ESTABLISHED
tcp4       0      0  112.22.1.172.58429      173.194.35.175.80      ESTABLISHED
tcp4       0      0  112.22.1.172.58428      173.194.35.175.80      ESTABLISHED
tcp4       0      0  112.22.1.172.58427      173.194.35.175.80      ESTABLISHED
tcp4       0      0  112.22.1.172.58421      173.194.35.175.80      ESTABLISHED
tcp4       0      0  112.22.1.172.58411      173.194.35.175.80      ESTABLISHED
tcp4       0      0  112.22.1.172.58383      74.125.139.191.80      ESTABLISHED
tcp4       0      0  112.22.1.172.58382      74.125.139.191.80      ESTABLISHED

The port#s in bold are my ephemeral port that's short live. Every time I connect via my browser to a page or link, it will create a new session until that tcp session is torn down. Same if I ssh to an external host or conduct a dns query. The port number are always increasing in a round-robbin fashion.

So if you have a few thousands of machines doing this,  and stacked behind one single /32 NAT host addresss, you can run out of available ports range. The available port-range is technically 0-65536, but common we only use the number 1024 and greater. 

And port #0 is never to be used or seen, since it's reserved by IANA.

So how do we avoid this potential port exhaustion in big enterprise networks ?

Simple, you need to create numerous NAT_pools and spread the SNAT around to  allow for more possible connections.

Take this simple cisco configuration where the source network is a  hughe 10.100.0.0/16. That along is over  64K machines or potential clients. And of then opening numerous ports for their tcp/udp sessions could drain a single ip_address if we where to PAT against one  address ( /32 )


Here's the NAT ACLs for the source networks;

 ip access-list extended mynat_group1 
 
  permit ip 10.100.0.0 0.0.63.255 any 
 
  remark range 10.100.{0-63}.0/24 
 
 ip access-list extended mynat_group2 
 
  permit ip 10.100.64.0 0.0.63.255 any 
 
  remark range 10.100.{64-127}.0/24 
 
 ip access-list extended mynat_group3 
 
  permit ip 10.100.128.0 0.0.63.255 any 
 
  remark range 10.100.{128-191}.0/24 
 
 ip access-list extended mynat_group4 
 
  permit ip 10.100.192.0 0.0.63.255 any 
 
  remark range 10.100.{192-255}.0/24 
 
 !  

And now the rest of the NAT configurations;

 interface GigabitEthernet0/0 
 
  description Link to Internal_CORE
 
  backup interface GigabitEthernet0/2 
 
  ip address 10.100.0.2 255.255.255.0 
 
  ip nat inside 
 
!

 interface GigabitEthernet0/1 
 
  description Link to TrackNetwork WAN Circuit ID:   
 
  ip address xxx.xxx.xxx.238 255.255.255.252 
 
  ip nat outside 
 

! 
ip nat pool ephemeral-group1 xx8.xx4.x8.201 xx8.xx4.x8.201 netmask 255.255.255.0 

ip nat pool ephemeral-group2 xx8.xx4.x8.202 xx8.xx4.x8.202 netmask 255.255.255.0 

ip nat pool ephemeral-group3 xx8.xx4.x8.203 xx8.xx4.x8.203 netmask 255.255.255.0 

ip nat pool ephemeral-group4 xx8.xx4.x8.204 xx8.xx4.x8.204 netmask 255.255.255.0 
 
! 
ip nat inside source list mynat_group1 pool ephemeral-group1 overload 
 
ip nat inside source list mynat_group2 pool ephemeral-group2 overload 
 
ip nat inside source list mynat_group3 pool ephemeral-group3 overload 
 
ip nat inside source list mynat_group4 pool ephemeral-group4 overload 
 
!


ip route 0.0.0.0 0.0.0.0 xxx.xxx.xxx.237 name INTERNET_NEXT-HOP
 
ip route 10.100.0.0 255.255.0.0 10.100.0.254 name STATIC_2_CORE 
 

So in this cfg you will notice we broke down the sources into ranges and then stacked them behind a single /32. The /16 was split into quarters ( 64 class C  groupings ) and then any hosts in those groups would be NAT'd to the corresponding /32.

i.e
10.100.0.1- 10.100.63.255 into group 1
10.100.64.1- 10.100.127.255 into group2
10.100.64.128- 10.100.191.255 into group3
10.100.64.192- 10.100.255.255 into group4


So in the above, that will provide a fair amount of ephemeral-to-client port ratio. And hopefully not exhaust any src_ports upon a pike nat-translations.

We can even tweak port expiration from the nat  process;

i.e

 ip nat translation udp-timeout 30 
 
 ip nat translation syn-timeout 45 
 
 ip nat translation icmp-timeout 30 
 
 ip nat translation port-timeout udp 53 20 

 ip nat translation port-timeout tcp 23 3600 
 
 ip nat translation port-timeout tcp 22 3600 

The above will handle  nat translation expiration to  recover quicker , once these session goes idle and based on the number of seconds idle.

So be very careful of your ephemeral port-ranges to avoid exhaustion. I will show you during the next  blogpost,  how we prevent this with a cisco ASA running code 9.X using the same above networks.

Ken Felix
Freelance Network/Security Engineer
kfelix   at hyperfeed   dot  com