server: NGINX
module: php5-fpm
os: Ubuntu
The Problem: Browsing to a Website returns 504 Gateway Timeout
A first world problem is having to wait for a website to load. Nobody these days has the patience for this, especially the viewers of your website.
Depending on the server’s configuration, if there is no response after a specified period of time, the server will return a 504 Gateway Timeout error. This means you watch the little circle turn around for 60 seconds or so and get no results.
This also means you lose visitors to your website. They go somewhere else.
Not good at all.
The Solution
Though I’m still working on this problem, this has at least stopped it from repeatedly happening. My suspicion is that there is some remote machine that’s klobbering my servers causing them to get hung-up, and unable to process any other requests.
To immediately get your site back online you can simple restart both NGINX and php5-fpm module.
sudo service nginx restart sudo service php5-fpm restart
This will should get your website back live to the internet… for a time being anyways.
The problem with this is that it’s a temporary fix. As a server admin, you don’t want to be doing this every day.
I want to take special note that after each change you make here, you should test the site to make sure it’s still working. Typically after you restart the server, with this issue, it should at least work for a little while.
I also want to remind the savvy admin to MAKE BACKUPS!
Finally, it is always a great thing to start with the error files that NGINX spits out. The exact location of these can vary, depending on how it’s set. It is usually set inside of the virtual domain configuration files (/etc/nginx/sites-available/*). You may also find that your PHP configuration file has an error output (/etc/php5/*). These error files will often tell you what module is causing the delay, if there is a specific IP that is hanging up, or if it’s a specific site that’s getting attacked.
Edit nginx.conf File to Increase Buffers and Timeouts
Like all servers, NGINX has a configuration file that has inside it various configuration settings for the operation of NGINX. NGINX’s configuration file is located:
/etc/nginx/nginx.conf
If you are not familiar with this file, inside the Configuration file (see NGINX documentation here) there are different blocks with directives (examples: events, http, server, and location).
This article is concerned with the http { } block inside the nginx.conf file. Adding the following lines will increase the functionality of NGINX, but remember, increasing these values too large and you’re opening yourself up to attacks.
client_header_timeout 3000; client_body_timeout 3000; client_max_body_size 32M; fastcgi_buffers 8 32k; fastcgi_buffer_size 32k; fastcgi_read_timeout 300; fastcgi_send_timeout 300; fastcgi_connect_timeout 300; keepalive_requests 50; keepalive_timeout 10;
The exact values you use above can be adjusted to your liking. Test and find out what works best for your instance.
However, I want to alert you to the fact that NGINX’s configuration file should be fairly effect out-of-the-box. Unless you just started doing something like making videos downloadable or you just launched eBay, regular HTML and PHP content shouldn’t require this kind of tweaking.
This means that what we’ve just done here isn’t addressing the core issue that something more aggressive like an attacker is slowing the server down. Of course, it could just be a website that’s getting more popular!
Switch UNIX Socket to TCP/IP
It appears that high load cases using a UNIX socket can cause this 504 Gateway Timeout. Seems kind of funny, but the suggestion is to make the following changes:
- in php-fpm pool configuration replace: listen = /var/run/php5-fpm.sock with listen = 127.0.0.1:7777
pool definitions should be found within the /etc/php5/fpm/php-fpm.conf file. - in /etc/nginx/php_location replace: fastcgi_pass unix:/var/run/php5-fpm.sock; with fastcgi_pass 127.0.0.1:7777;
location ~ \.php$ { will be found nestled within NGINX configuration file: http { server { location { . If you have virtual domains setup, it’s in /etc/nginx/sites-available/*. You will have to change this on each virtual domain you have. It’s a great time now to create an include file for this location, if you haven’t already.
Remember that once you do this, you have to reset both NGINX and php5-fpm again
Conclusion
More testing needs to be done to make sure that this exact solution has worked. Though tweaking of the exact buffers and timeouts on the server can always be done, it is a fine balance in understanding what my servers are serving up, and not allowing a robot to come in and clog up the serving.
It would be nice if there were a more direct way to monitor and blacklist the traffic.
Thanks kindly to the following sources I found ideas of solutions at:
- Codetweet | Solved – 504 Gateway Timeout Error In Nginx Web Server
- Stackoverflow | php-fpm socket -> TCP/IP
- NGINX | nginx.conf HTTP Core Module Directives
- NGINX | nginx.conf fastcgi Module Directives
- Scale Scale | 502 Bad Gateway on NGINX Fix
- TCP Keepalive Overview