Amazon ELB and Client side certificates

Amazon-Web-Services-AWS

The problem

Amazon’s ELB only allows you to setup loadbalancer with a normal SSL certificate and does not support 2 way SSL authentication. To overcome this issue, there are certain “hacks” you can make use of. At the time of writting this post, Amazon AWS was in the process of implementing this feature, but it was unsure when it will be publicly available.

The solution

There are few things you have to be aware of, while setting this one up. It’s not easy to scale 2 way authentication SSL. Especially if you want it to be “invisible” to the backend SSL servers.

Few things you have to consider:

  • How will I scale backend servers?
  • Where will I handle SSL authentication?
  • How will I pass the authenticated information to backend servers?

Facts:

  • Amazon supports https protocol, but it’s irrelevant, since client side certificates aren’t supported.
  • Amazon supports Proxy Protocol v1.0
  • Amazon support TCP/UDP Proxying

Taking everything into an account, I figured out that setting up an infrastructure with basic TCP proxy for SSL, enabling Proxy protocol on ELB and forwarding that traffic to backend server, would be an option. You could get real ip of a client from proxy protocol and you could handle client side certificates on the server. Of course if you have multiple instances and using Apache as webserver, this gets a bit tricky. Apache doesn’t support proxy protocol. Only HAproxy and nginx do. Therefore it might be good idea to use HAproxy in front of our Apache webservers, and just for the sake of redundancy, make 2 of them. HAproxy can also handle SSL authentication and add X-Forwarded-For headers for backend Apache to figure out real IP of a client, instead of getting the one of HAproxy or ELB. There is one neat php module (in case you use php application in the backend webservers) called mod_cloudflare. Sure it’s made for filtering out Cloudflare’s ip’s, but it also allows you to replace original REMOTE_ADDR in PHP with X-Forwarded-For variable.

This is theory, now let’s try this in practice…

Setting up ELB

First of all, setup ELB with proxy protocol. You need to use CLI for this, since there is no web interface implementation to enable this. I think you should check Amazon’s website for detailed instructions on howto enable this. Here’s the link: Enabling Proxy Protocol ELB

Second, fire up 2 instances of CentOS or your preferred distribution on EC2, for HAproxy’s. Add them to balancer.

Your settings of ELB should look like this:

Stickiness: Disabled(Edit)
443 (TCP) forwarding to 443 (TCP)

Cross-Zone Load Balancing:    Disabled (Edit)

Load Balancer Protocol    Load Balancer Port    Instance Protocol    Instance Port    Cipher    SSL Certificate       
TCP                       443                   TCP                  443              N/A       N/A
Setting up HAproxy instances

Since HAproxy doesn’t have decent version in CentOS’s repositories, that would have SSL authentication enabled and I wasn’t sure if proxy protocol is supported by default, I just compiled one myself.

wget http://www.haproxy.org/download/1.5/src/devel/haproxy-1.5-dev26.tar.gz
tar -zxvf haproxy-1.5-dev26.tar.gz
cd haproxy-1.5-dev26
make TARGET=linux2628 ARCH=native USE_OPENSSL=yes
make install
cp examples/haproxy.init /etc/init.d/haproxy
chmod 755 /etc/init.d/haproxy
useradd --system haproxy
mkdir -p /etc/haproxy

My config file on both haproxy’s looks exactly the same. I only changed IP address used for binding to the one that HAproxy uses.

Here’s the relevant part of the config:

defaults                                                                                                                                                                                                           
#       log global            # Don't want logging now                                                                                                                                                                                     
#        option httplog                                                                                                                                                                                            
        option http-server-close                                                                                                                                                                             
        option httpclose                                                                                                                                                                                           
        option redispatch # important one, has to be set.
        no option checkcache

frontend ssl_servers
    mode http
    option forwardfor except 192.168.xxx.0/24 # Add real IP in headers, but don't include my internal IP's.
    option httpclose
    option http-server-close
    tcp-request connection expect-proxy layer4 # This one actually picks up proxy protocol that you enabled on ELB
    bind 192.168.xxx.5:443 ssl crt /etc/haproxy/mycert.pem ca-file /etc/haproxy/myca.crt verify required    # This is internal IP of my HAproxy, and it sends default certificate (godaddy signed for example) and CA for client side certificate authentication. You could pass also .crl, but that's optional.
    http-request set-header SSL_CLIENT_S_DN_Email       %{+Q}[ssl_c_s_dn(emailAddress)] # as HAproxy is doing authentication I want it to send required data to my backend servers as specific variable. In this case it's SSL_CLIENT_S_DN_Email.
    reqadd X-Forwarded-Proto:\ https
    default_backend ssl-servers

backend ssl-servers
    mode http
    redirect scheme https if !{ ssl_fc }
    server server1 192.168.xxx.10:443 maxconn 512 check ssl verify none # .10 is one of my SSL webservers, and I don't care if there's anything wrong with backend server's SSL certificate. Don't want to be strict about it, but you can be.
    server server2 192.168.xxx.11:443 maxconn 512 check ssl verify none

Okay this should already be forwarding traffic from HAproxy to backend webservers and connections should be working already. With one exception. All clients that connect to backend webservers are represented as IP of HAproxy… We don’t want that. To fix this, read on.

Backend webservers (Apache+PHP)

We can get neat little module from Cloudflare to get PHP to know about real IP of our clients. I tried others, but failed miserably or was just too much hassle. This one actually did the work and it’s really simple to setup.

wget https://www.cloudflare.com/static/misc/mod_cloudflare/centos/mod_cloudflare-el6-x86_64.latest.rpm
rpm -ivh mod_cloudflare-el6-x86_64.latest.rpm

Now the cloudflare.conf which can be found at /etc/httpd/conf.d/:

LoadModule cloudflare_module /usr/lib64/httpd/modules/mod_cloudflare.so
<IfModule mod_cloudflare.c>
        CloudFlareRemoteIPHeader X-Forwarded-For
        CloudFlareRemoteIPTrustedProxy 192.168.xxx.0/24
</IfModule>

Now, what this module does, is swaps X-Forwarded-For and REMOTE_ADDR variables. If you check it in PHP and var_dump whole $_SERVER variable, you should see real IP of your client in there.

Okay we got PHP to find which IP is real ip, let’s fix our Apache logs also.

In order to replace our HAproxy’s IP with X-Forwarded-For – real IP, we have to modify our default httpd LogFormat line:

#LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" VLOG=%{VLOG}e" vhost
LogFormat "%v %{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" VLOG=%{VLOG}e" vhost

Commented version is the one I used before, and uncommented is my current (the one that gives real IP of a client). Make sure you replace your %h with %{X-Forwarded-For}i in your LogFormat at httpd.conf. Your LogFormat line could look different. Don’t worry. This is just the one I use, becouse I manipulate logging a bit.

This should be it. Restart your Apache and check your access logs. Real IP’s should be there.

Stress test

I stress tested this setup with 2 backend webservers and 2 haproxy servers. I was browsing the web and simultanously shutting down one webserver/haproxy at the time, or even pairs of them. As long as 1 webserver and 1 HAproxy was up, no SSL connection was broken and I was normally browsing our web sites. No need for re-authentication.

Leave a Reply

help-hint.png
Purpose of the commenting system is to share your experience. I encourage you to post feedback with your own suggestions, ideas or optimizations regarding the topic of a blog post. What commenting system isn't for, is asking questions about similar issues of yours and requesting support for it. Blog post is provided as is and I am not here to solve all your problems. Please bear that in mind and try to avoid posting such comments. I do take privilege to remove comment from my blog for any reason whatsoever. Usually I do it when I sense a comment was posted only for spam/seo reasons or is out of blog post's topic. Thank you for reading this, now you may continue :)
 

Your email address will not be published. Required fields are marked *