Wednesday, February 22, 2017

Understanding HTTP redirect traces analysis

When dealing with http servers and SLB, it's common to have redirects, but you need to understand the differences.

Typically 301 and 302 are the most used redirect methods. 307 is another   redirection but not seems that often. These are also called "Server Side redirect" since they typically in the past where used on web-server daemons but in  reality they could be  servers, WAF or reverse-proxies.


Let's look at some redirects examples , but before we do that, " what are redirects"? These are   populate Location: headers that re-points the  web client to the  web content that he/she is looking for. The simple definition

A  location is passed to the User-Agent in the shape and fashion of

Location: http://kenfelix.com/blogger/

Or

Location: https://kenfelix.com/

This new Location header is just that, a header  that points you to the new location.

301 == permanent
302 == Found-A-new-Location
307 == temporary

NOTE: 302 are mistakenly called temporary  redirects , but per the http.definitions, they are not.


reference https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

Here's my bank IBC, this is a very bad redirect flow. It just cause more additional work on the browser , and  additional delay for the 1st byte served or the content requested





Up next here's another example of a redirect flow. I highlighted cookies in this one . Two window servers where involved plus a load-balance

note: notice how Set-Cookies are not encrypted, this  leaks internal host information





One of the best unix tools available for following redirects is the  unix curl with the -L option.

L == Locations


Here's mail.google.com, notice the URL protocol changes  if you initially hit the 1st link via HTTP.





Each redirect  creates a new HTTP.request and adds to the dependency  and  delay factor.


Typical a domain redirect  are very popular;

e.g



And for SSL enabled  site a  http-2-https are commonly used




Jigsaw has a simple and effective  307 redirect and proper use ( notice temporary in the http.response code ?}

https://jigsaw.w3.org/HTTP/300/Overview.html



HTTP/1.1 307 Temporary Redirect  <-----

vrs

HTTP/1.1 302 Moved Temporarily    <-----




Here's how paypal redirects   302 then 301 (a big no no  imho )







reference  http://www.redirect-checker.org/index.php


Good practices are ;

1: keep the redirects as small as possible
2: use  307 redirects when possible & if the  target is Temporary and not Temporarily
3: avoid a redirect loop 
4: if you redirect from HTTP  2 HTTPS on a server, try to use HTTPS on that server
5: check redirects consistency and  remove any stale redirects on a regular basis
6: for web-developers, harvest your site URLs and run them thru a spider or URL optimizer that  minimizes duplications a short HTTP hops between redirects on a local website
7: remember   redirect types 301 are more favorable for SEO



These tips would help you in  creating a good HTTP experience for  the end-user


NOTE: PayPal is really cool if they see your a web-analyzer guy that uses curl , they give you  a X custom header for jobs recruiting in the  response






Ken Felix
NSE ( network security expert) and Route/Switching Engineer
kfelix  -----a----t---- socpuppets ---dot---com
     ^      ^
=(  @  @ )=
         o 

        /  \

2 comments:

  1. If you want to do the same thing in Perl from the CLI, try this.

    #!/usr/bin/perl
    #Author: Lance Vermilion
    #Purpose: Map redirects for webpages
    #Date: Feb 24, 2017
    ####################################
    use strict;

    my $url = $ARGV[0];
    chomp($url);
    my $origurl = $url;

    \$/;;/g'`;tput = `curl -s -L -I -X GET $url | sed 's/^
    my @sections = split(/;;/, $curloutput);

    my $loop = 1;
    my $redirects = 0;
    my $href = {};

    for my $section (@sections)
    {
    my @lines = split(/\n/, $section);
    for my $line (@lines)
    {
    chomp($line);
    $line =~ s/\r//g if ( $line !~ /^$/ ); # remove annoying ^M
    $redirects++ if ( $line =~ /^$/ ); # Count redirect because there is a blank line between redirects
    push(@{$href->{$redirects}->{'header'}}, $line) if ( $line !~ /^$/ );
    my (undef, $LOCATION) = split(/ /, $line) if ( $line =~ /Location: / ); # split line to get location so we can construct complete URL for display
    $href->{$redirects}->{'fulllocation'} = $line if ( $line =~ /^Location: / );

    if ( $line =~ m/^Location:/i )
    {
    if ( $line =~ m/^Location: (http.*)/i )
    {
    $url = $1;
    $href->{$redirects}->{'redirectlocation'} = $url;
    }
    elsif ( $line =~ m/^Location: (\/.*)/i )
    {
    my $uri = $1;
    $url =~ s/\/$//;
    $href->{$redirects}->{'redirectlocation'} = $url . $uri;
    }
    else
    {
    $href->{$redirects}->{'redirectlocation'} = "null";
    }
    }
    }
    }

    print "### REDIRECT MAPPER ###\n";
    print "There are $redirects redirects for the URL: $origurl\n";

    if ( $redirects > 0 )
    {
    #use Data::Dumper;
    #print Dumper($href);
    print "\nOriginal URL: $origurl\n";
    for my $key ( sort keys %$href )
    {
    print "Header:\n";
    for my $line (@{$href->{$key}->{'header'}})
    {
    print " $line\n";
    }
    print "\nRedirected to: " . $href->{$key}->{'redirectlocation'} . "\n" if ( $href->{$key}->{'redirectlocation'} );
    }
    }
    else
    {
    print "\nOriginal URL: $origurl\n";
    print "Header:\n";
    for my $section (@sections)
    {
    my @lines = split(/\n/, $section);
    for my $line (@lines)
    {
    print " $line\n";
    }
    }
    }

    ReplyDelete
  2. Thanks

    I haven't ran the perl script yet but this might come in handy on my day job. We have a lot web engineer that place http.redirects, but really have no insight on redirect traces & on what they are redirecting and any broken redirects.

    ReplyDelete