Site-2-Site ROUTED VPN Trouble-shooting & Guide Fortigate

In my past postings, where we configured a lan2lan vpn between a fortigate and juniper-SRX, this is a continuation on t-shooting.

If you are familiar with the webGUI,  you will have ran across this ipsec-monitor at some point and time.

Okay, so what do we do and what do we look at, while trouble-shooting ipsec vpns?

In this blog, we will look at a few various means for  the diagnostics of VPN problems. This post is for troubleshooting vpn-tunnels created in a  interface-mode  ( aka routed  vrs policy ) & those using  PSK for peer-authentication.



PHASE1

We will now look at some of the  troubleshooting and show commands, that can be executed on a fortigate to help diagnose vpn problems.

1st, I want to state,

90% of the problems with these VPNs can be traced back to a mismatch in our configurations. So always double check your configurations, and this mean all aspects of the configurations.

e.g

• Your Pre-Shared-Key  or  Certificate 
• IKE proposal ( 3des md5/sha1/sha256 or whatever your using )
• IPSEC proposal ( 3des md5/sha sha256  or whatever your using )
• peer address  ( is it correct , is it the correct interface )
• expected identification ( FQDN or ip_address or Hybrid )
• fwpolicies
• route reachability 
• etc.......

The  very 1st common error  encountered, typically pertains to  our PSK key entries;  "be cautious of any leters, numbers, lower and uppercase". These are simple to spot in the logs, due to we commonly see "mis-match pre-shared-key" or something to that nature. So always re-key if you believe the PSK is not correct.  This is probably 99.99% of the problems that prevent phase1 setup.

Using the combination of the following cmds;

1. diag debug app ike filter name "phase1-name"
2. diag debug app ike -1 
3. diag debug enable

Will always give you clues to any PSK and other proposal issues;

here's a possible mis-match within our PSK;

Moving on, when the PSK has been confirmed & is correct, then the next most common problem develops around the initial  proposals that's being offered between both parties.

I ( just me personally ) do not like to deploy multi-proposals. It's not required,  can cause confusion by some various  remote-vpn devices. Unless it's a dynamic-vpn, you  DON"T NEED MULTI proposals. Yeap, I'll re say it here again, YOU DON'T NEED MULTI PROPOSALS in a site2site vpn.

Once again  the diag debug app ike  will show you both yours,  and theirs proposals, & what's being  present  by both parties. There has to be a match and agreement before phase1 can ever become established.


e.g ( spotting a mis-match proposals blue is the far-end and what they sent and purple is what I have configured )

NOTE: With a packet capture only, you will also notice very little traffic after the initial connection from the initiator & responder ( why would  2 vpn appliance attempt any further communication if  the keys don't match ? or the proposal didn't agreed in the 1st place )

So even without logging, you can follow the ip datagrams and determine if it's a possible PSK issue exists, & by looking at the ip datagrams from a packet capture. I always suggest doing  this ( packet capturing )  for  a few other reasons;

• You can validate the peers  ip_address that are configured
• are they negoiating via NAT transversal
• confirm that no ACLs are blocking traffic between peers

Next, you need to be aware of  IKE and that it uses src_port #500/udp and dst_port #500/udp unless we are being NAT. If NAT-transversal ( aka NAT -T ) is involved, the dst_port becomes  #4500/udp.

The integral  diag sniffer will expose problems if the initiator or responder are not being reached or fail to respond

e.g ( cmdline using wan1 interface )


diag sniffer packet wan1 "udp and dst port 500"

or

diag sniffer packet wan1 "udp and st port 4500"

Next, once you have confirmed the  PSK and initiator/responder are talking, than we want to ensure the IKE +IPSEC proposal are matching.

With the IKE+IPSEC proposals, the authentication proposal has to match, and the Diffie-Hellman group must match. This determine the key strength during the negoiation of the peers over a unsecured channel.

  

Why such a big harp on proposals and mis-match? 

Will look at  just how many proposals that a modern  Fortigate Appliance running 5.0.X software supports;

What this means? That's a lot of @$#$%  proposals.

Even a juniper or cisco don't support ALL of the listed proposals. Even some firewall have proposals, that even a fortigate don't support.

e.g 

Juniper SRX supports both, PSK and RSA/DSA signatures  certificates, where-as a  Fortigate supports only PSK and RSA

Juniper SRX supports md5 , sha1 and sha256, where-as a fortigate support all of these and  sha 364, 512

Linux's Strongswam support twofish and blowfish ciphers, both of which are not part of a Fortigate lingo or supported ipsec  ciphers  by commercial firewall that I know of

Until recently,  cisco firewall didn't support  IKEv2, where-as fortigate has  had ikev2 support for probably 7+ years or more if I had to guess

Some cisco routers support cisco-only cipher available with cisco routers or phase1 authentication methods that could be totally different.

Eg

 here what a cisco firewall  support ( crack, rsa-sig,psk );

yes Crack :)


Eg 

a common cisco router ;

Next  ( this is  VERY VERY VERY IMPORTANT  PAY ATTENTION! )

" The Phase IKE SAs keylife should always match, and be pretty darn close at it!  "





A mis-match here , will typically not be a show-stop within the initial setup for our phase1 security-association, but the 2 ends not matching, will cause interruptions and problematic issues upon re-keying.

This is due to the keylife expiring out on one vpn device, while the other side still think  it's all is good.

NOTE: DPD and ike-keepalive ( similar but they are not the same ) can help with regards to remote-peers & detection of a failure in connectivity.

Take this SRX and FGT, one party has the keylife set to an exaggerated  120sec ( FGT ) and the other to 180sec ( SRX ). That's the two lowest values that support by these firewall devices btw

Look at what happens when we ping across the vpn tunnel & when we have a mis-match in IKE keylife ?

One last items about phase1 trouble-shooting

If NAT-T is involved, you might need to pin the PAT/NAT session table, or use some of kind of NAT-T keepalive to avoid the expiration of your PAT/NAT translation.

What this means; "  if  the  NAT-T session is maintain by a NAT'ing device such as a cisco router , an upstream firewall, or some other network translation device", any traffic that's does not maintain a certain interval, could cause the IKE session to get hung up and stale. The NAT-T keepalives prevents this from happening.

You can enable NAT-T support,. by each per-peer under your fortigate configurations for phase1 ;

e.g

NAT-T support is passed along during the IKE phase1 negotiations between peers, if they  both support it.


Let's look at another diag debug output from a fortigate  " diag debug app ike 255 ";

Here's we can see the output shows a phase2 proposal mis-match and it's clearly indicated by the above output & using the "diag debug app ike " command.


Let's look at some more diag vpn outputs. One of the most important  option within fortigate diagnostics is our phase1 filters. In a hub or a service provider arena, you might havd a dozen or more  vpn-tunnels. It's not at all uncommon to see 100-1k vpn-ipsec-tunnels nailed to a fortigate.

NOTE:  Use the filtering option at your hub vpn-device,  to avoid the over-saturation of diagnostics outputs & filter on the gateway of interest



Here a listing of my vpn tunels. It provides good  basic information on any IKE and IPSEC SAs, uptime and  peer information. If DPD was enabled we would have DPD details.

and for comparison, here's an active  VPN I have where dpd is enabled &  on just one side;

So sometimes we can quickly find out if both parties are using  DPD,  and if we are going across a NAT-d connection via the remote/local peers destination ports of 500/udp vrs 4500/udp.

NOTE:  DPD DOES NOT NEED TO BE ENABLE BOTH WAY for a vpn tunnel to work, but it should be enable mutually imho.




For negotiation, either parties can negotiate the  Phase1 SA, But let's say you don't want to do this or maybe  your a passive vpn-termination device like those  that I worked within a a few 3 letter gov agencies sub-contractors in the past.

Will have no fear, we can tell your device to never initiate the connection;

The above two screenshots stacked one on top of the other,  shows the Juniper/SRX being the initiator and the  Fortinet/Fortigate acting as a  responder.

PHASE2

Now let's say we think we have our IKE SAs established, but we are still having problems, remember the quickmode selectors ( aka  QM or Proxy-ids ) ? They have to match for both parties & for the IPSEC-SA that are establish under phase2.

Here's our configuration for the FGT<>SRX using a static defined proxy-ids.

NOTE: The local and remote subnets where defined. If you should build your  phase2 configuration like this &  between all devices, than you will never have a phase2 proxy-id mismatch imho. Also if you should happen to change the device on one end to another vpn-hardware appliance, you will not have to any  changes to make on your fortigate vpn configuration.

But in reality, fortinet made a easy button, &  that has been a source of many  major problems & confusion.

By default the proxy-ids are defined as 0.0.0.0/0:0 for both the dst and src subnets:protocol. But this would not work for 99.99% of the vpn-tunnels that terminates to a non Fortigate VPN appliance device.

Yes, try a QuickMode selector of 0.0.0.0/0:0  to a;  cisco ASA/Router, Strongswan, Pfsense,  Checkpoint, etc.....

Your tunnel will not come up ( period ).

Okay we know the easy button  does not fit all applications,  and does not fit all vpn-appliances. I only recommend the 0.0.0.0/0:0  quickmode selectors,  when you have a only-Fortigates on the other end of the vpn-tunnel and you have multiple of networks that you are carrying over the vpn-tunnel.

In reality, I almost never do this and  prefer to build my phase2-interface SA  per dst/src subnet and let me tell you my logic why?

When you build multiple ipsec-phase2 SA, and define each local/remote subnets details, you get statistics per each IPSEC-SA. This can help you with your diagnostics of why a SA number ###  does not work , or what direction ( remember  SA are uni-directional   )

Take this design;

With the above method#1 and #2 & static defined SAs, we will have statistics & for 2ea SAs,  using our diagnostic commands.

And now for some diagnostics output for review;

As you can clearly see, the 2nd proxy-id name srx-p2-1 is not encrypting any data. ( see the text above for our redline underline? )

If you had used method#2 & the one single 0.0.0.0/0:0 define for both src/dst, it would not have been easy to determine the above problem.

PFS the what and why

PFS is one of those configuration items,  that can cause a whole heck of problems. Will have no fear, make sure if you enable it on one vpn-peer, that the other peer also has it enabled. What PFS does in short, is to initialize the IPSEC-SA from a key sessions  that's not generated from the  parent key within the IKE SA.This provides more tighter security against any compromised ike keys.

Keep in mind that with PFS enabled, your ipsec SAs could  use a completely different  encryption schema and ciphers  from that of the phase1 IKE. What that means, you could enable 3DES+grp5 for IKE phase1 and AES128+grp14 for the IPSEC SA phase2.

A complete lack of  any  connectivity at our phase 2 ipsec SAs, could very well be a mismatch in your  IPSEC proposals, or worst you have one SA encrypting data and the other side not de-encrypting or vice-versa. These are hard to identify without mutual eyes, and troubleshooting from both end of the vpn via the firewall engineers. BUT the diag vpn tunnel list , does shed some light into these types of problems.


Moving on,

When the tunnel are up, we have a useful means for gathering statistical data ( the number of packet encrypted/decrypted, SPI identifier , bytes sent/received ) via the "diagnose vpn tunnel  list " command and by following the output. This command is always my 1st command issued, while trouble-shooting vpn diagnostics problems.


One last things that comes up, a VPN ipsec-tunnel  has some overhead. So a 1500 byte MTU is not going to fit with the overhead of the ESP-header, including the additional ip_header,etc.......

Unix offers the means to test  max MTU size by using the DontFrag option and slowly increasing pings with the DF bit set to "1", at some point of the pings failing, this would be your max MTU.

e,g

ping -D ( macosx ) or  ping -M ( linux ), for windows you have a few utilities out that accomplish the same

Conducting packet captures at the src/dst subnets can validation the TCP max-segement size and ip-DF ( don't frag bit  ) settings, these  can shed light  within any path-mtu related issues.  Ping sweeps starting at a low to high packet size, can also some shed light to a vpn-tunnel mtu issues.




A review of the diag commands that are useful for all firewall engineers using a Fortigate security appliance;


diag debug enable
diag packet sniffer
diag debug app ike
diag vpn tunnel list 

NOTE: These are only available from the command line



MISC

Interface based vpns or not?

In this posting we didn't  even tap the various different modes of  vpns.  Within the fortiOS you have 2 possible means for crafting vpns;


1: policy base where we use the key wording "encrypt or ipsec" within a firewall policy action   ( fwiw I NEVER EVER  use this mode & it's highly recommended not to build you vpns in this fashion )


2: routed mode or commonly known as interface mode ( this the best-common-practice and mode most engineers rely on )

Here's the PROs for the latter ( routed mode )

1.  you can run a routing protocol over the vpn
2.  you have a interface to nail policies to just like your ports or wan interfaces
3.  since we have a interface, we can dump or conduct packet captures over that interface using diagnosic sniffer command
4. you can assign ip-address to the inteface
5. you have snmp-ifindex , hence now we can graph and monitor with cacti, solarwinds, OpenNMS, and against a interface.

Guess what, if you created the phase1/2 definitions with the phase1-interface and phase2-interface, you are creating a routed-vpn. And the key word, "routed" means you need a static or routing protocol to learn the remote location dst-subnet(s). So this means you need a route to reach the far end dst-subnet(s)

DIAG SNIFFER

The name you give in that phase1-interface configuration, is the same interface you defined for the route gateway and the same interface you can dump on, with the sniffer to capture packets b4 or after encryption.

NPU issues

You can always disable the NPU from the vpn tunnels if you suspect some weirdness or have concerns with advance  acceleration or other off-chip performance. I never recommend this as a permannet fix, since it drives the encrypted traffic directly at the  cpu.

Using the full keyword in our show, we can see that we are off-loading to the NPU.

The following screenshot shows some of the  crypto stats and these are  helpful with eyeballing traffic at the NPU if your model has any VPN acceleration ;

( continuation )

Show-Full 

Don't forget, that In all configuration details, the "show full" command within the sublevels, provides greater details and default settings.

Clearing IKE-gateways

Some times we need to clear a IKE gateway. For good practices, any changes done in you pre-existing vpn tunnels, should be followed by a clearing

By specifying the "clear " and within the diag vpn ike gateway, we can  flush our  IKE SA.

And lastly & final

Use the diag debug and flow commands. I 'm providing a few link to my earlier posts & with concerning the easy to execute diagnostics debug flows

http://socpuppet.blogspot.com/2013/03/flow-diagnostic-fortigate.html

http://socpuppet.blogspot.com/2013/06/diag-debug-flow-troubleshooting.html

http://socpuppet.blogspot.com/2013/06/diag-system-session-quick-way-find.html


Execution of the diagnostic debug flow, gives you about 90% of the time, the proper direction to take based on the output followed by the diagnostic flow

• the lack of a fwpolicy
• lack of a forwarding route or uRPF checks failures
• policy ordering issues ( maybe you over look policy ID that denying the traffic & ahead of your fwpolicies for your VPN )

By using the diagnostic debug flow commands, you can save yourself time, frustration, and wasted tickets to fortinet-support,  or posts to online forums.



Ken Felix
Freelance Network/Security Engineer
kfelix  at  socpuppets .....dot....com

   ^      ^
=( *  * )=
       o
      /  \