Fighting a DDoS Attack – Limiting Requests in NGINX

DDoS = distributed denial of service attack.

server: NGINX
os: UBUNTU

The Problem

A DDoS attack is a pain in the arse. It’s basically some bot out there, or a connection of bots (distributed) that are sending requests to your server in an attempt to overload it and make it really really slow – possibly to the point of causing it to crash.

The Solution

Luckily, this isn’t the first time people have heard of this, nor will it be the last, so there’s security measures in play.

Firewall

Most notably, we have a firewall. The main premise of a firewalls is they block all traffic except the stuff you want through. To enable this, Ubuntu has a built in firewall called Uncomplicated Firewall ‘ufw’ that should already be ready for you to enable. The manual is here for ufw. The ufw Community Help Wiki is here.

The community help wiki is extremely helpful at starting this for you, but basically, you’ll want to:

ufw allow ssh <- allow ssh connections to your server
ufw allow http <- allow http connections to your server
ufw enable <- enable ufw itself now that you’ve allowed the connections you’ll still need to access it
ufw status verbose <- have a look at what settings you’ve set

This is just simply smart computing, especially with a server, which allows you to ensure that all your ports are blocked so that nobody can get in that you didn’t really want in.

However, this does nothing for limiting the amount a single IP can ping your server, so we still haven’t stopped the DDoS attack.

NGINX Limit Request Module

NGINX has a built in module called limit_req – Limit Requests – that’s does exactly what one needs to do to limit the effectiveness of a DoS attack. If a DoS attack is limited to so many requests per second on your server and then is bottlenecked for a certain time, it becomes a very inefficient DoS attacker. When this happens, a log is created of the event which we’ll be using later with Fail2ban.

The NGINX manual for limit_req is here, and there is a great article on EasyEngine on configuring the limit_req module, but you’re going to be opening your NGINX config file:

sudo vi /etc/nginx/nginx.conf

and adding the following inside the http{ … } block:

limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;

What you’re doing here is setting up a limit_req_zone configuration that you’ll refer to later by the zone name: zone=one. You can change it to: zone=superstrictrule, or anything of your choice. $binary_remote_addr is the binary IP address of the out of line attacker. 10m is the size of the file that stores the zone states in. rate=1r/s means that this zone’s limits are 1 request per second. You’ll want to configure this zone file as you see fit, particular with the rate, and you can use (s) for seconds, (m) for minutes, and NGINX makes clear if you’d like a rate of less then 1r/s you must go towards: 30r/min, which is kind of equivalent to 0.5r/s.

Then, inside the unique server {…} block (or a virtual server if you’ve deployed them):

limit_req zone=one burst=5 nodelay;

To set the actual limit_req rule. This doesn’t necessarily have to be inside a server { } block, it can be inside a location { } block if you’re really getting specific. nodelay makes sure that as soon as the limit is hit, an error is thrown. zone=one will identify which specific limit_req_zone the limit_req is for, and burst=5 is the burst size that’s allowable (my suspicion is that 5 is of the rate already defined, so for this example, it would be 1r/s is the rate, burst would be allowable 5r/s, I’d love is somebody could confirm this in the comments).

Fail2Ban

I’m not quite sure the source of the name, but Fail2ban is pretty brilliant tool. It scans your logs and will create specific blocks based on suspicious activity. So, if the same IP is hammering your server, it’ll see it and block it. Another cool thing is that it works for any logs, be it SSH, mySQL, HTTP, etc. Within the jail.local file that one creates upon installation, you can activate whichever monitors you wish.

The Ubuntu Community Wiki help file on Fail2ban steps you through installation and how to enable each module, and there is also this tutorial at EasyEngine on how to configure Fail2ban for NGINX that’s really helpful, but the steps are:

sudo apt-get install fail2ban <- install Fail2ban
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local <- create a local configuration file
sudo vi /etc/fail2ban/jail.local <- edit the local configuration file and add the section below:

[nginx-req-limit]

enabled = true
filter = nginx-req-limit
action = iptables-multiport[name=ReqLimit, port=”http,https”, protocol=tcp]
logpath = /var/log/nginx/*error.log
findtime = 600
bantime = 7200
maxretry = 10

sudo service fail2ban restart <- restart fail2ban with the new configuration settings
(might have to do ‘sudo /etc/init.d/fail2ban restart’  instead of above)

note: logpath above should be the folder that your nginx config file has been set to for it’s errors, according to the error_log directive. This may sometimes be inside the nginx.config file itself, or if you have virtual servers and have declared it inside the virtual server itself, it will be in there: /etc/nginx/sites-available/mywebsite. The * in front of the error.log means that any log file that ends with error.log will be included in the rule. So, mywebsite.error.log will be included, so will myotherwebsite.error.log. It is handy to have each website, virtual server, creating its own error log so that you can much easier monitor each site.

Once you have Fail2ban running, you can monitor the log file that is actually remarkably clear on what it’s doing (compared to some very cryptic log files):

sudo vi /var/log/fail2ban.log

If it’s not there, you might have to go looking for it, using the command:

sudo ls -la /var/log/

To browse the folders to see what’s in them (hint: you’re looking for a file named fail2ban).

After a while, you may want to see a report from Fail2ban to see if it’s actually working. The command:

fail2ban-client status nginx-req-limit

will give you an output that will tell you how it’s working.

Conclusion

Well, I hope this helps some of you start your path towards higher security on your webserver. Sometimes it can feel almost like you’re hoping everything goes right and when something goes wrong you’re caught with your pants down. Truly, this solution feels a little bit like magic.

I’d love if some of you who have done this for a while could share below your configurations so I could get a look at optimal settings, or if you’ve written your own similar blog, please do share a link. Better security is always becoming more widely versed in secure practices.

Further Links




Defensive Data – An In-Depth & Technical Look at Online Data Security

Software Suggestions
(see bottom of post)

I need not list of the myriad of reasons one should be cautious of data security, but I should like to take a few moments to talk about the ways in which one can become more savvy in securing their data.

What is Data

  • n.
    Factual information, especially information organized for analysis or used to reason or make decisions.
  • n.
    Computer Science Numerical or other information represented in a form suitable for processing by computer.
  • n.
    Values derived from scientific experiments.

With regards to your person, data is all points of interest in you. Some really easy to understand data points on your person are: birthdate, full name, address, social insurance number, passport number. However, digging a little deeper, we can begin to see a much wider array of data on your person that could potentially become valuable information to somebody.

I want to iterate the fact that you don’t know who that somebody is. You don’t know what scheme they’re getting up to, or how they’re able to use the data of your person to their advantage.

Some more abstract ways that your person creates data are gps locations, browsing history, search terms, pattern keystrokes, facial features, facial responses, habits, grocery store foods commonly purchased, favourite websites, car colour, various usernames and passwords, etc..

Data then becomes any measurable or observable point of information (from you or anything else). If somebody is sitting at the corner of the street counting the colours of pants they see, this is data. Those two rubber lines one sometimes drives over on the street are collecting data on the number of cars driving by and the speed at which they’re driving. Data is the number of sales per day.

In our modern time of technology, we are very data rich and thus we’ve all become rather used to data being collected on us, and oblivious to the myriad of ways at which we are being observed – that is, we are becoming numb to our data points.

Securing Computer Data

Thus, a need arises to become more savvy in securing our data to make this unnecessary data collection harder on prospective snoopers who have no good intentions in their data collection. Every single level of security that you bring into your workflow will make it harder for somebody to steal it, and consequently will make it less likely they’ll go after you. In statistics, the bad guy usually goes after the easy pickings.

I will now begin a list of ways in which one can systematically become more secure in our data-centric world. I will first start off with fairly general security guidelines for every day application, and then I will drill down into some specific tactics for computer security.

General Online Secure Guidelines

  • Acting cautiously in the physical and digital world – this is really an obvious one. When somebody phones you up from who-knows-where saying they’d like to talk to you about something or other, and they just need your phone number, date of birth, postal code, and mother’s maiden name to begin with questions…  you might want to be a bit cautious about sharing this information. It’s not bad to say: “I’m not comfortable giving this information to you.” I do it all the time, and the caller is then forced to say: “Ok, no problem. Good bye.” In the digital world, this means not filling in all the fields if they’re not necessary (usually necessary ones are indicated by some sort of red asterix), or perhaps creating a secondary email that you can give to not-so-secure websites that are not-too-important to your person (more on this below).
  • Learn about encryption – encryption isn’t a scary thing, it’s actually really cool. Leonardo da Vinci wrote upside down and backwards as a form of encryption 500 years ago. Today, we have secure options that are password protected inside of encrypted database files.
  • Everywhere you enter information is a possible vulnerability – any/all of those websites that you enter in your name, password, family members name, mother’s maiden name, address, postal code, SIN, etc. are all possible vulnerabilities. If any one of them gets hacked by somebody then all of your information you entered is now in the hands of somebody who may or may not use it maliciously. This means at online stores where you enter your information, at the local swimming pool website, the transit card website, etc. Any of them are subject to attackers. I’m not suggesting never create an online account anywhere, but maybe think about the information you’re handing over as a general rule.
  • Never ever use the same password – never. No excuses. Hopefully explanations below will help with this one.
  • If it’s connected, it’s vulnerable – anything that is connected to the internet is vulnerable. Almost every single website or software or computer operating system has been cracked. Who and how, you don’t want to know. Internet of Things refers to those devices like a webcam or baby monitor that’re connected to the internet but don’t function like a computer. News every day comes out exposing how hackers are using this vulnerable equipment to their advantage.

Data Encryption Techniques

So those are general rules to consider as you enter upon your data security. The first big step towards really making change in your flow of things is data encryption.

Having data encrypted means that unless you have the proper password to decrypt the data, it is virtually un-accessible (or at least you’d need a few days and super computers working at cracking it). Thus, if you have to keep sensitive data on your computer or stored on a hard drive, the defensive security approach to this data would be to encrypt it!

You can encrypt a drive, folder, or a single file (to keep this explanation simple).

Whatever way you encrypt though, if you’re browsing on your computer to the location of the encrypted data file, it will look as a single file of unknown type and seem like no program will be able to open it. So, if you were to name your encrypted drive Secret Information, it will show up as just a file with the name Media of about the same size as you created it.

Encrypting a Drive or Folder

Encrypting a drive or folder are about the same. The theory is that you first create an encrypted drive or folder, then you mount it with your encryption software as if it’s a virtual drive, and then use it as if it’s a regular drive attached to your computer with supposedly no data transfer speed sacrifices. In a sense, it’s very similar to plugging in a USB stick. With the encrypted folder, you open up the software, select the encrypted data file you’d like to mount, mount it (this is like inserting a CD or USB stick), enter the decryption password, and then it’s mounted and fully accessible like any other drive on your computer.

Data Encryption Caveats

Encrypting data is essentially making more trouble for you, the user. It’s not as simple as just saving files to your computer, and you’re now forced to think about where you’re saving all your files. But, you’re adding steps to your workflow that will hopefully make it harder for thieves to get access to it.

One problem I constantly have is with an encrypted USB drive that I then plug it into another computer without the encryption software or the password to decrypt it. This makes this USB drive virtually an un-usable drive now. You can’t mount it to gain access to the information without the software (which you can technically download to the new computer). My current work-around is to have another un-encrypted USB device to easily plug into other computers. In this way, my secure USB drive works like an offline drive for my computer. This is the ultimate way to ensure there is no stealing of my data; if it’s not online, it’s not available.

Encryption Software Choices

VeraCrypt is the software I have chosen to use.

Password Security

Password security is probably the biggest issue right now in the online world. There is no secret that many many users are using simple passwords like: hotlegs2, bigdaddy69, etc. Further, many people have a hard enough time remembering their passwords, so they just keep repeating their same passwords for all websites they approach.

Not good.

If one of the websites compromises your password (and most likely all your other user data) then all of the sites you’ve ever used your data on with that same password are virtually accessible.

Like I said, not good.

Password Techniques

The longer the password string the better. The more random the password strings the better.

The way I see it, is that I only ever need my passwords if I’m on my computer, so if I make a digital safe that’s encrypted with all my passwords in it, I’ve just made life a whole lot easier for me. No sense trying to remember all the passwords in the world I’ve made (and yes, there’s a lot).

So, use software to remember all of your passwords, so that I only need one password to access that software that has all my passwords in it. This provides flexibility of password choices and to copy/paste the passwords and usernames it’s just a double click away.

A benefit of this is that most password software will also give you password suggestions. In this way one doesn’t even have to wrack their minds over their new password, you just click that suggest button and done. Further, using software, most of them will be able to remember password history (you can set how many of the last ones you’d like to remember). Further, in the unlikely event that you are held hostage and demanded to hand over the password to something, you can honestly say that you don’t know it, because you won’t. You’ll know the password to get into the password database, but you won’t actually know the password itself. (This might make more sense if you watch pwsafe.org’s quick start guide.)

All of the passwords, rest assured, are stored in a single encrypted database file. This file can be stored anywhere, and accessed by only the same software that made it in order to decrypt it. To unlock it, you need one password, and then you’re in and have all your passwords.

Sounds much more secure then a little book you carry around with you and sometimes forget it on your chair, right?

Password Encryption Caveats

Almost everybody these days has more then one smart device: laptop, smartphone, computer, 2nd laptop? Who knows which device you’ll need which password on. It would be silly to depend on having the encrypted password database file on one single device, for the passwords are long and not easy to enter in manually. Imagine having to read off your computer and then type into your phone this password: xkhs8H7shfp)-)8sjk%yHk. You don’t want to do that.

To get over this problem, one can transfer their encrypted password database file to all of their devices. My method of choice is to use a cloud file repository like Box or Dropbox. I’ll upload the encrypted database file to Dropbox which my other devices will also have access to. I then ensure that that file is available offline by that device because some programs that you may have passwords to don’t necessarily need the internet to run.

Password Encryption Software Choices

I use PwSafe. It is recommended by Bruce Schneier who’s one trusted expert.

Email Security

Emails easily have the most sensitive information that’s most vulnerable in it. Personal conversations are where the juicy stuff is! Each email usually has a thread of a whole long conversation about specific topics of discussions. And more. Your email address alone can become compromised, meaning somebody gets access to all your emails. And if somebody else gets access to your emails, they can do all sorts of damage.

There is a benefit to using a service like Gmail: they are almost too big to break (I’m sure somebody somewhere is snickering about this). Sure, this may go against the axiom: the bigger they are the bigger they’ll fall, but compared to your own personal email server, they’re most likely on top of data security. On your own email server, when was the last time you did a security audit on it? Further, if you’re using an email at your own domain, what happens if somebody actually gets access to your domain itself and re-routes all your emails to their own inbox? The likelyhood of somebody getting access to google.com is pretty slim, whereas somebody getting access to a domain name registered through a website with very weak password recovery options is a lot more likely.

This extends to company email systems. One would hope that they’re secure, but security is an expensive business to be savvy in.

It is not a bad thing to have multiple email addresses. There are some email solutions that pride themselves on being very secure end-to-end encrypted email providers (think Proton Mail), and then there are some others who have no encryption. In fact, there’s probably a list somewhere on the internet of email providers who are notorious for being watched, hacked, and easily accessible to just about anybody with that intent.

So, when you’re creating an account on a website, keep in mind what email address you’re using. My rule of thumb is that if it’s a website related to anything un-professional, I’ll use a secondary email address for spammy things that is definitely level 2 or even 3 importance. This includes online shopping stores and website newsletters or blogs. If it’s a government, banking, professional website, I use the most secure email address I have.

Email Archives

When the Germans, meticulous record keepers they are, realized they were losing the war, they started burning everything. All their records destroyed. Why? Because records can and most likely will be used against you.

In our modern day of data richness, it is almost counter-intuitive to delete emails.. any data for that matter, especially emails you’ve been holding on to for the past 10 years. But they’re a security breach.

My suggestion is two fold.

The first is to just delete all the garbage emails that have no significant value in storing. Easy things are ads, newsletters, flyers, promotions, etc. Harder ones are emails from family members with some form of communications. Find a level that you’re comfortable with. Some people want a history so that if their grandchildren want to read their old communications in 50 years, they can. Others don’t plan on having kids and expect nobody will ever read a thing. Your choice.

The second suggestion is to practice archiving old emails, putting them into a local (encrypted) folder. Think of it similar to a filing cabinet. It’s no longer on your desk of important letters to attend to, it’s been properly filed away and most likely never to be looked at again.

It Takes Two To Email

Unless you’re emailing yourself, you’re having an email conversation with another person. This could be your biggest vulnerability. It is kind of up to you to make sure they are taking the necessary precautions to avoid compromisation. And to further this, it is up to you to avoid conveying any sensitive information to a person who doesn’t need it. If you’re in the habit of conveying sensitive information, it is imperative to take the necessary precautions to protect your data security. If the other person doesn’t want to be secure, then don’t send them secure information, your security depends on it.

Conclusion

The road to online data security can get quite long. The main idea with security techniques is to make it harder for others to access your data. Every practice you implement makes you much less likely to become a target. The day may not be now that you’re a target, but if we think of a long term strategy, the long game, starting your data security game now will save you a lot of trouble and headache in the long run.

Hope this helps!

 

Additional Data Security Resources

  • Online Privacy Guide for Journalists 2017 <- if anything I’ve written is at all interesting, this is a very in-depth article on all privacy techniques.
  • Signal Messenger <- look in the app store of your phone, it’s secure, end-to-end encrypted messaging
  • WhatsApp <- also secure, peer-to-peer encrypted messaging
  • Tor Browser <- ultimate online web activity security using built in proxies
  • Password Safe <- my favourite secure password database software, available for computers and most smartphones
  • Bruce Schneier <- security expert who puts out really interesting articles worth a read (that means sign up to his newsletter)
  • Proton Mail <- free secure email service with built-in end-to-end encryption and state of the art security features (like gmail on high alert).
  • VeraCrypt <- free disc encryption software



Mount: Can’t Find /dev/xvdg in /etc/fstab or /etc/mtab

os: Ubuntu / Linux
terminal: SSH

mount: can’t find /dev/xvdg in /etc/fstab or /etc/mtab

The above is the exact response that my terminal in Ubuntu gave me when trying to send the command:

sudo mount /dev/xvdg

For anybody knowing anything about the mount command in ubuntu, one must not only specify the device they would like to mount to the system, they must also specify the name to give the drive once it’s mounted.

So, the proper command would have been:

sudo mount /dev/xvdg /folder_name_here

 

A few tips that might help you in this process:

To view attached volumes: sudo fdisk -l

To view mounted partitions: sudo df -h

To view a list of folders: sudo ls -la /




NGINX Error: could not build the server_names_hash

server: NGINX

could not build the server_names_hash, you should increase either server_names_hash_max_size: 512 or server_names_hash_bucket_size: 64

The Problem

As my server grows, so do the number of server names. You may ask yourself, how many server names could one person have?! www.teition.com and teition.com is only two!

NGINX has a scaleable feature known as Server Blocks (Apache uses the name Virtual Host) (see NGINX examples of server blocks here). These allow a single server to serve more than one domain name. In this way, a single server can have one IP and multiple domain names. I won’t get into the specifics of a deployment process, but rest assured, there are benefits and naturally some drawbacks to doing this.

So, I went to add two server names to a server block under the server context, greatly simplified:

server {
server_name servername.com newserver1.com newserver2.com;
}

Simple enough change, I’ve done it to many blocks before. Of course, after doing this, one must restart NGINX in the console:

sudo service nginx restart

which promptly [fail]ed.

I checked the NGINX server error logs, by default located in \var\log\nginx\error.log and found:

could not build the server_names_hash, you should increase either server_names_hash_max_size: 512 or server_names_hash_bucket_size: 64

The Solution

Reading the above error message, seems pretty easy to understand one must increase the size of one of the two. According to NGINX hash documentation, whichever element shows up first in the error log, it is suggested to increase that one.

Thus, for me, this meant changing the server_names_hash_max_size directive. This is located inside the http context, the \etc\nginx\nginx.conf file.

However, when I went to look at the nginx.conf file, there was no existing line: server_names_hash_max_size, but I did find the line: server_names_hash_bucket_size 64; (which was octothorp’d out meaning it was set to default according to the NGINX hash documentation).

I then, just above the ‘server_names_hash_bucket_size 64;’ line, added:

server_names_hash_max_size 700;

Making it just slightly larger then the default size of 512.

side note: if you feel very strongly that I should have set this to 1024, please comment below your reasoning. I fail to believe that a whole 512 block will be reserved for the new hash and and only use the additional 188 that I’ve set above. I like to believe the designers of NGINX wouldn’t allow such a block to be left… But I’m open to correction.

Once this has been changed, a server restart was required:

sudo service nginx restart

Love to hear if this worked for you!

 

Resources:




AUKEY 30000 mAh Power Bank

AUKEY 30000 mAh portable battery power bank

I’ve been experimenting with various power banks over the past years. What this means is both recharging my devices with portable power banks, and also trying to switch my devices over to USB rechargeable.

Without a doubt, this AUKEY Power Bank is pretty damn impressive. It’s got 30000 mAh, which means about 15 phone charges for my Samsung phone. Comparatively, those smaller ones you see that look like a lipstick container can probably recharge the phone once. This AUKEY Power Bank can very easily fit into my day bag and is hardly any heavier than most other power banks out there.

Specs

Capacity: 30000mAh
Micro-USB Input: 5V 2.4A
Lightning Input: 5V 2.4A
Output 1 (Quick Charge 3.0): 3.6V-6.5V/3A, 6.5V-9V/2A, 9V-12V/1.5A
Output 2 (AiPower): 5V 2.4A
Dimensions: 5.9” × 3.3” × 1.1”
Weight: 20.46 oz

Off of Amazon, this will cost $60 CAD.

AUKEY 30000 mAh portable battery power bank

A common question I always field about a mobile power bank is basically: why? Why do I need to carry this when there’s almost always power somewhere out there I can plug in to. The answer is really an understanding of where one finds power. For instance, if I’m on the road.. let’s say on the ferry. I can wander the ferry for 15 minutes looking for one of those power outlets that’s definitely going to be out of the way, stretch my power cable over whoever is undoubtedly sitting right next to it, and then sit there with my phone as it charges up… -or- I can just plug right into my power bank that’s in my day bag and continue enjoying the beautiful day on the ferry without much inconvenience.

Needless to say, living life as a nomad, with electronics (phone, camera, speaker, flashlight, tablet) having a power bank is indispensable. Nobody wants to be stuck running out of time, power dying on the phone, and the only directions to the place tonight is on the phone.

To charge the Power Bank, I plug the Power Bank itself into the microUSB connector end of a cable, and the other end goes right into a charge source (likely a wall plug or car adapter). Nearly the exact same as a smartphone. You can even use a phone power connector (unless you’re using an iPhone). Charge time will take about ~ 6 – 8 hrs from a wall socket, depending on the capacity of your charge connector you’re using. Remember, when fully charged, this can recharge a smartphone about 15 times.

AUKEY 30000 mAh portable battery power bank

The benefit here comes in the fact that one charges up the Power Bank so that it can be used whenever devices run out of power. So, if I were to fully charge my phone and camera with it, then leave the Power Bank plugged into the wall while I went out to get groceries or a walk along the river, I could still have my electronics (fully charged at this point), and be recharging the Power Bank at the same time!

I like to think of this as a paradigm shift in how we use electricity, but perhaps for a lot of us it does sound very logical already. The availability of power storage is the basis of all my future electronic considerations.

AUKEY 30000 mAh portable battery power bank