Nginx + HaProxy gotcha
July 22nd, 2008
I’ve recently switched from using a straight up Apache mod_proxy + mongrel setup to Nginx + HaProxy +mongrel. Basically, mod_proxy is dumb as shit, and should never be used with mongrel + rails, especially with better options like Passenger and HaProxy.
Why does mod_proxy suck? Take a look for yourself:
[8030/5/247]: handling 127.0.0.1: GET
[8031/6/134]: handling 127.0.0.1: GET
[8032/2/160]: handling 127.0.0.1: GET
[8033/0/93]: idle
[8034/10/128]: handling 127.0.0.1: GET
[8036/2/124]: handling 127.0.0.1: GET
[8035/0/110]: idle
[8037/0/101]: idle
Look at this garbage. There are three idle mongrels even though the other five have 25 requests to be processed.
Whats great about HaProxy is that it will queue up requests and only distribute them to idle mongrels. It translates into way better response times under load. Add Nginx, which is way easier to configure that Apache, allowing me to easily setup things like sending known ‘slow’ requests to separate mongrels and it’s a big win. As a bonus, haproxy has a great status tool, which gives a quick overview of your proxy clusters.
One little problem …
The gotcha when I initially switched over was that exception_notifier stopped sending me error emails, in fact the errors were being displayed to the user. What the hell?Exception notifier looks at the incoming IP address. If it’s local (aka: 127.0.0.1), it’s assumed that the user is a developer, so it won’t email the exception to you. haproxy was sending nginx’s IP address, rather than the forwarded client address. The problem was that I had configured both nginx and haproxy to forward headers.
haproxy.conf:
option forwardfor # enable insert of X-Forwarded-For headers
nginx.conf
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
Removing the header forwarding in the haproxy configuration solved the problem. You only need to set the X-Forwarded header in Nginx (or Apache or Pound or whatever front end you choose). Setting it in Haproxy as well will over write the header with one containing the IP of the front end. Since I have Nginx and Haproxy running on the same machine, that IP is 127.0.0.1, which broke the exception_notifier.
This is also published on Learnhub
8 Responses to “Nginx + HaProxy gotcha”
Sorry, comments are closed for this article.
July 22nd, 2008 at 10:14 PM Nice article Wes!
July 23rd, 2008 at 09:55 AM "Why does mod_rails suck? Take a look for yourself:" From the context it would seem that you meant mod_proxy instead of mod_rails. Is that correct?
July 23rd, 2008 at 11:23 AM It's mod_proxy. mod_rails is awesome, Hongli. :)
July 23rd, 2008 at 11:24 AM Hongli: You're absolutely right, sorry about that. mod_rails is great! Fixed!
August 1st, 2008 at 10:21 AM I have a similar setup. I have Nginx frontend with upstream to a haproxy in the same server serving tomcat servers. Did you have any problems servicing pdfs with your setup?
August 4th, 2008 at 11:31 AM @Austin: Nope, no issues serving PDFs. http://learnhub.com/files/0000/0330/OCaml_For_RubyMN.pdf
August 5th, 2008 at 01:06 PM Just curious, what made you choose nginx -> haproxy -> mongrel over haproxy -> nginx -> mongrel? And, is running both haproxy and nginx really necessary? Can't you have one load balancer to communicate with multiple mongrel processes on multiple instances, so you would not need nginx in between? I'm thinking of putting haproxy as the frontend, so I can use hot reconfiguration and quickly be able to switch to another app farm for serving requests. Also, haproxy doesn't support keep-alives. Any issues with this limitation? Thank's for the blog!
August 7th, 2008 at 02:49 AM @Andrej Nginx handles static requests and passes dynamic requests to haproxy. I also use Nginx to filter and send requests to different proxies based on it's URL. It's really useful in order to separate badly behaving controllers (slow, memory eaters, etc) from other requests, so they don't slow them down. Technically haproxy could be left out of the equation since nginx includes proxy functionality, but it's not as good as haproxy's