1st To recap the design;
NOTE: This is not ideal in that both tunnels are terminated to the same device, but due to the satefull inspection we are limited on a firewall. You could terminate the tunnels on 2 unique firewall and run a iBGP session between, but you will need to be cautious on asymmetrical routing issues and the statefull inspection failures. Please see import/export route policies included in my bgp configuration section YMMV
The configuration is broken down to interfaces, crypto, bgp and firewall & misc configurations. YMMV and your outcome could be different. This was tested on 5.0.X code
1st the interfaces;
==================== INTERFACES ======================
config system interface
edit "wan1"
set vdom "root"
set ip 12.126.x.x 255.255.255.252
set allowaccess ping
set ident-accept enable
set type physical
set alias "WAN-ISP1 ATT-pl"
next
edit "wan2"
set vdom "root"
set ip 38.x.x.x 255.255.255.252
set allowaccess ping
set ident-accept enable
set type physical
set alias "WAN-ISP2 PSINET"
next
end
tunnels configurations;
config system interface
edit "AWSvpc1"
set vdom "root"
set ip 169.254.255.2 255.255.255.255
set type tunnel
set remote-ip 72.21.209.225
set interface "wan1"
set tcp-mss 1412
set alias "VPC_TUN1"
next
edit "AWSvpc2"
set vdom "root"
set ip 169.254.255.6 255.255.255.255
set type tunnel
set remote-ip 169.254.255.6
set interface "wan2"
set tcp-mss 1412
set alias "VPC_TUN2"
next
end
==================== VPN CRYPTO STUFF ======================
( interface mode vpns aka route-based )
config vpn ipsec phase1-interface
edit "AWSvpc1"
set interface "wan1"
set dpd enable
set dhgrp 2
set proposal aes128-sha1
set remote-gw 72.21.209.225
set psksecret 1blahblahblahmyPSKkey
set dpd-retryinterval 15
next
edit "AWSvpc2"
set interface "wan2"
set dpd enable
set dhgrp 2
set proposal aes128-sha1
set remote-gw 72.21.209.193
set psksecret 2blahblahblahmyPSKkey
set dpd-retryinterval 15
next
end
NOTE: phase2 quick-modes selectors defaults to 0.0.0.0/0:0 which is okay for AWS
config vpn ipsec phase2-interface
edit "AWSvpc1"
set dhgrp 2
set pfs enable
set phase1name "AWSvpc1"
set proposal aes128-sha1
set replay enable
next
edit "AWSvpc2"
set dhgrp 2
set pfs enable
set phase1name "AWSvpc2"
set proposal aes128-sha1
set replay enable
next
end
==================== BGP CONFIGURATIONS ======================
1st we build a few import/export policies via a common route-maps that we will call-out for our peers
We need to adjust the local_pref on 1 peer only
We need to set metrics on the 1 peer that we want to be carrying our traffic
config router route-map
edit "import"
set comments "my import policy apply to one peer"
config rule
edit 1
set set-local-preference 600
next
end
next
edit "export1"
set comments "my export policy apply to one peer only and on the primary tunnel peer"
config rule
edit 1
set set-metric 1147400
next
end
next
edit "export2"edit "import"
set comments "my import policy apply to one peer"
config rule
edit 1
set set-local-preference 600
next
end
next
edit "export1"
set comments "my export policy apply to one peer only and on the primary tunnel peer"
config rule
edit 1
set set-metric 1147400
next
end
next
set comments "my export policy apply to one peer only on the path you don't want to use"
config rule
edit 1
set set-metric 2147400
next
end
next
end
And now for the fun stuff our bgp configs ;
config router bgp
set as 6500
config
neighbor
edit
"169.254.255.1"
set
description "VPC1"
set interface " AWSvpc1"
set
remote-as 7224
set
route-map-in "import"
set
route-map-out "export1"
next
edit
"169.254.255.5"
set
description "VPC2"
set
interface " AWSvpc2"
set remote-as 7224
set route-map-out "export2"
next
end
config network
edit 1
set prefix 10.10.0.0 255.255.255.0
next
end
edit 1
set prefix 10.10.0.0 255.255.255.0
next
end
NOTE: If it's not clear here, we are setting the local_preference higher on tunnel1 to force our traffic out tunnel1 always.
Likewise we are forcing metrics to ensure AWS steer traffic down tunnel1 over tunnel2. This helps maintain our statefull checks. If we didn't do this, we would have to enable asymmetrical routing which is a bad bad things for firewalls
If tunnel1 fails, goes down or if AWS conduct maintenance on the datacenter that houses the VPN termination, we revert to tunnel2.
( a simple fwpolicies )
config firewall policy
edit 0
set srcintf "internal"
set dstintf "AWSvpc1"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ANY"
next
edit 0
set srcintf "AWSvpc1"
set dstintf "internal"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ANY"
next
edit 0
set srcintf "internal"
set dstintf "AWSvpc2"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ANY"
next
edit 0
set srcintf "AWSvpc2"
set dstintf "internal"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ANY"
next
end
==================== STATIC ROUTES ======================
And lastly, it's best practices to place static /32 routes to the AWS vpn-terminate for the respective ISP interfaces;
config router static
edit 99
set device "wan1"
set dst 72.21.209.225 255.255.255.255
set gateway 12.x.x.x
edit 100
set device "wan2"
set dst 72.21.209.193 255.255.255.255
set gateway 38.x.x.x
next
end
NOTE: Amazon seems strange in not wanting to support you if you use a device that's not a router. So YMMV on how well your approach and design & the support you get from the open community.
I hope to have a working for a Juniper/SRX in a few days, so stay tuned :)
If your in the need for a cloud based solution, please reach out to these fine folks at Turing Group. They have been providing cloud based solutions with Amazon for over a year and have over 8 years within the cloud datacenter technologies and strategies
http://www.turinggroup.com
info @ turinggroup <dot> com
If you need remote access solutions please contact me at email below.
Ken Felix
I hope to have a working for a Juniper/SRX in a few days, so stay tuned :)
If your in the need for a cloud based solution, please reach out to these fine folks at Turing Group. They have been providing cloud based solutions with Amazon for over a year and have over 8 years within the cloud datacenter technologies and strategies
http://www.turinggroup.com
info @ turinggroup <dot> com
If you need remote access solutions please contact me at email below.
Ken Felix
Freelance Network / Security Engineer
kfelix ----a---t---socpuppets ---d---o---t---com
^ ^
=( ^ ^ )=
o
/ \
Great article. Did you ever get to set this up on a Juniper SRX? I'm facing the same issue with asymmetric routing if I want the tunnels to be redundant and BGP seem to be the only way to solve this.
ReplyDeleteI'm curious to know why you choose "set metric" vs. as-path prepend to attract the traffic on the right link (if there is a reason).
Nicolas
ReplyDelete1st thanks for the reply.
2nd yes, I did some stuff with juniper & AWS and I will post ablog. Juniper has a few more quirks and issues but the process is about the same.
To answer the question
[QUOTE]I'm curious to know why you choose "set metric" vs. as-path prepend to attract the traffic on the right link (if there is a reason).[/QUOTE]
Traffic within the same AS 7224 would not be effected by the as-path-prepend. The this bgp attribute prepends your AS# X amount of times in the advertisement outbound.
In order to select one uplink path over the other, we set the metric for the path we want to use. This will allow us to route over one link over the other.
Also, for traffic intra-AS, we could advertise to that AS peer, what path to set by metrics when we have a multi-exits. It's aka as MEDs. This metric is typically not carried out of the peer AS.
Please see the follow blog posting
http://socpuppet.blogspot.com/2013/08/non-transitive-path-attributes-bgp.html
Thanks for the extra explanation (I'm not familiar with Fortigate, so please forgive my cluelessness ;-)) so if I understand correctly you're using MED (set metric) to influence inbound routing but this doesn't seem to be supported by AWS (based on this conversation https://forums.aws.amazon.com/message.jspa?messageID=470375 and the linked doco http://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Introduction.html#MultipleVPNConnections where MED isn't mentioned). I'm guessing that it worked as expected because the lowest BGP peer IP address belonged to tunnel1 which is your primary tunnel. So it sounds like AS-PATH prepending is the way to go to control inbound routing (and for outbound we use local-preference, no issue there).
DeleteI'm figuring this out as I go along so I might still be missing part of the picture.
Nicolas thanks for your reply. I will do some testing hopefully this wekend to see if AWS vpc actually honors the metrics. MEDs are typically use intra-as and is not carried out of the peering AS.
ReplyDeleteI will post what I find
Is it possible to setup the IPSec tunnel with AWS without using BGP to advertise the local subnets to the VGW?
ReplyDeleteI'm using static routes on both ends and while I have the tunnels up and running, I can't pass any traffic because there are no routes being 'advertised' as per amazon support. It's almost as if they only use dynamically learned routes in their gateways.
you can add static route in AWS routing table. so just add your LAN subnet and it will be fine.
Deletefyi, AWS does support static route IPsec tunnel. in fact they will ask you whether its dynamic or static when you first create the tunnel.
As of Jan 2016 AWS no longer honors metrics. Don't want anyone to be led to a dead end following these (excellent) instructions. Instead you need to use set as-path-prepend and prepend your AS multiple times to influence less desirable routes. We are also finding lots of the formentioned gotchas with asymetric routing since these are firewalls, its almost mandatory to set the asymetric routing enable command on the Fortigates depending on your design.
ReplyDeleteThanks for the update
ReplyDelete