Friday, March 1, 2013

Avoiding TCP/UDP Port Exhaustion Cisco Router

This blog will talk about ephemeral port exhaustion. 

In a  Network Translation and with PAT  ( port address transalation ) , a client  source address and port numbers are mapped to  a single NAT'd source. This is commonly called SNAT ( source NAT'ing ). Doing this for a hand full of clients machine is not critical. Since the src_port is expiring and temporary used and forever changing.

This process of temporary using a src_port is called "ephemeral". So take this machine that I'm typing this blog on;

 sahel01: ~ kfelix$ netstat -an | head
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)   
tcp4       0      0      ESTABLISHED
tcp4       0      0      ESTABLISHED
tcp4       0      0      ESTABLISHED
tcp4       0      0      ESTABLISHED
tcp4       0      0      ESTABLISHED
tcp4       0      0      ESTABLISHED
tcp4       0      0      ESTABLISHED
tcp4       0      0      ESTABLISHED

The port#s in bold are my ephemeral port that's short live. Every time I connect via my browser to a page or link, it will create a new session until that tcp session is torn down. Same if I ssh to an external host or conduct a dns query. The port number are always increasing in a round-robbin fashion.

So if you have a few thousands of machines doing this,  and stacked behind one single /32 NAT host addresss, you can run out of available ports range. The available port-range is technically 0-65536, but common we only use the number 1024 and greater. 

And port #0 is never to be used or seen, since it's reserved by IANA.

So how do we avoid this potential port exhaustion in big enterprise networks ?

Simple, you need to create numerous NAT_pools and spread the SNAT around to  allow for more possible connections.

Take this simple cisco configuration where the source network is a  hughe That along is over  64K machines or potential clients. And of then opening numerous ports for their tcp/udp sessions could drain a single ip_address if we where to PAT against one  address ( /32 )

Here's the NAT ACLs for the source networks;

 ip access-list extended mynat_group1 
  permit ip any 
  remark range 10.100.{0-63}.0/24 
 ip access-list extended mynat_group2 
  permit ip any 
  remark range 10.100.{64-127}.0/24 
 ip access-list extended mynat_group3 
  permit ip any 
  remark range 10.100.{128-191}.0/24 
 ip access-list extended mynat_group4 
  permit ip any 
  remark range 10.100.{192-255}.0/24 

And now the rest of the NAT configurations;

 interface GigabitEthernet0/0 
  description Link to Internal_CORE
  backup interface GigabitEthernet0/2 
  ip address 
  ip nat inside 
 interface GigabitEthernet0/1 
  description Link to TrackNetwork WAN Circuit ID:   
  ip address 
  ip nat outside 
ip nat pool ephemeral-group1 xx8.xx4.x8.201 xx8.xx4.x8.201 netmask 

ip nat pool ephemeral-group2 xx8.xx4.x8.202 xx8.xx4.x8.202 netmask 

ip nat pool ephemeral-group3 xx8.xx4.x8.203 xx8.xx4.x8.203 netmask 

ip nat pool ephemeral-group4 xx8.xx4.x8.204 xx8.xx4.x8.204 netmask 
ip nat inside source list mynat_group1 pool ephemeral-group1 overload 
ip nat inside source list mynat_group2 pool ephemeral-group2 overload 
ip nat inside source list mynat_group3 pool ephemeral-group3 overload 
ip nat inside source list mynat_group4 pool ephemeral-group4 overload 

ip route name INTERNET_NEXT-HOP
ip route name STATIC_2_CORE 

So in this cfg you will notice we broke down the sources into ranges and then stacked them behind a single /32. The /16 was split into quarters ( 64 class C  groupings ) and then any hosts in those groups would be NAT'd to the corresponding /32.

i.e into group 1 into group2 into group3 into group4

So in the above, that will provide a fair amount of ephemeral-to-client port ratio. And hopefully not exhaust any src_ports upon a pike nat-translations.

We can even tweak port expiration from the nat  process;


 ip nat translation udp-timeout 30 
 ip nat translation syn-timeout 45 
 ip nat translation icmp-timeout 30 
 ip nat translation port-timeout udp 53 20 

 ip nat translation port-timeout tcp 23 3600 
 ip nat translation port-timeout tcp 22 3600 

The above will handle  nat translation expiration to  recover quicker , once these session goes idle and based on the number of seconds idle.

So be very careful of your ephemeral port-ranges to avoid exhaustion. I will show you during the next  blogpost,  how we prevent this with a cisco ASA running code 9.X using the same above networks.

Ken Felix
Freelance Network/Security Engineer
kfelix   at hyperfeed   dot  com 


No comments:

Post a Comment