monitoring tool: NGINX Amplify
monitoring tool: ngxtop
monitoring tool: access_log
Necessity of Monitoring a Server
As a professional, it’s essential you monitor the status of your tools. If you’re building a bridge, you check for cracks; if you’re running a server, you monitor logs. It’s important. If people come to your website and may get turned away because of some mis-configuration or server fault. In this case, there is no point in having a website at all if all the traffic is getting sent away. Further, if your website is working extremely slow, then you’re also going to have visitors very quickly going to find another resource on the internet.
So, we know that it is very important to monitor the status of your web server. I will provide some of the tools that I use in this post.
Warning: Making your server run fast and cleanly is addictive, once you begin down this road, it is like a rabbit hole and you will not return the same (lazy web administrator).
By far, one of the easiest to use and elaborate monitoring tool is Amplify by NGINX (they may still be in Beta, but beta is a great thing when the developers respond immediately to issues). Amplify goes into much greater detail then I will here about how to install and configure Amplify on the first of three configuration blogs: Setting Up NGINX Amplify in 10 Minutes. The default metrics that show up on your Amplify Dashboard are very good for monitoring your server, more so now that they’ve done some UI changes, but it’s worth the time to also go into the Alerts and Custom Dashboards sections to see if they are giving some suggestions to change your server configuration (NGINX itself), as well as custom graphical metrics.
Stub Status must be turned on for some more back end information, so don’t miss that in the configuration step, but more importantly I feel is setting up the access log configuration acutely.
They have put together three full posts on how to configure your Amplify here:
- Setting Up NGINX Amplify in 10 Minutes
- Improving Server Configuration with NGINX Amplify Reports
- Using NGINX Amplify Custom Dashboards and Filters for Better NGINX Monitoring
NGINX Access and Error Log
NGINX’s built in access_log and error_log directive should most definitely be turned on. This provides all kinds of essential information regarding your NGINX server. Without this critical information, you’re kind of shooting in the dark when it comes to troubleshooting your server errors.
Using the access log, you can also set the log format, this can be customized to give some detailed information about what’s happening on your server in regards to error codes, request URI, referrer, etc.
You set the access_log format inside the nginx.conf file (typically located in etc/nginx/nginx.conf on a Ubuntu deployment), but you can also set the access_log parameters inside each virtual server configuration block so that you can have a unique log files for each virtual servers (typically located in etc/nginx/sites-available/siteName). Virtual servers are usually used if you have multiple websites hosted by the single NGINX server.
A caveat that should be noted is that if you define access_log within each unique virtual server, Amplify can provide a breakdown of each websites access_logs.
For a full list of access_log log_format descriptors, see NGINX Metrics & Metadata documentation.
note: I’m still working on ways to customize my access_log file results so I can better identify exact URL requests for 4XX and 5XX errors, so if you have a resource or some tips, I’d love to hear them below.
From the command line within your server you can run ngxtop to get very detailed breakdown of what’s happening on your server in real time. There are multiple commands that you can customize, so do have a look at the GitHub documentation to get a better idea of what you can do with this powerful tool. This is more of a real-time monitoring tool (as the Ubuntu top command is, too).
I say Python script because Python is quickly becoming my favorite programming language, but you could probably use any language you like to parse the access_log files for basically anything you’re looking for. If one was so inclined, they could even turn this into an hourly cron job that sent results to you via email.
Working on this code now. 🙂
Hope some of this helps, and I’d love to hear if you have any other suggestions that are essential for you and your server administration!
What is a Color Profile or Colorspace
Every single device or medium out there has an ideal color profile for viewing pleasure. That is, your phone, your laptop, and your printer all have a recommended color profile for optimum color representation that will look good to your eyeballs (or the professionals that decide it will look good for you). There are a few main color profiles that are most standard in the world of images:
- ProPhoto RGB
There are dozens more, and each of these color profiles has its benefits and detriments that, in fully understanding where you’re using the images, will help you understand what color profile you want to use.
So, what this means is that there are different color profiles that show and define color differently. It’s like a new deck of playing cards, sure, they’re all aces and kings, but the graphics and size of letters on them are all different. Some are better for old folks with trouble seeing tiny numbers, some are better for young kids who like to see more wild illustrations for the royalty.
Why is my color dull on my computer?
If you’ve just scanned your beautiful painting or transferred your beautiful photograph from your camera that looks so vibrant on the back of your camera, and you then you open up Photoshop or Lightroom all excited to see how it looks after scanning and notice that it looks a bit dull and desaturated, then there’s a good chance that you’re not viewing the image in sRGB color profile.
sRGB Color Profile
sRGB is the standard color profile for anything being viewed by a monitor, so this means laptops, televisions, smartphones, and even the person sitting next to you on the airplanes laptop screen. Almost all monitors and phones across the board are set up to in the sRGB color profile. Unless you’re doing fancy things and have a special program that has converted your monitor to display another color profile, you’re going to get the best viewing pleasure from your images if your in the sRGB color profile.
What this means for you as an image creator is that any image that you put onto the internet should be in sRGB color space.
Read more about sRGB on Wikipedia
AdobeRGB Color Profile
AdobeRGB is the standard color profile for anything being printed. A growing number of printers request your images in the sRGB color profile, but an AdobeRGB image will have a wider range of colors that a professional quality printer will be able to use.
What this means for you is that if you’re printing your image, you most likely want to be in AdobeRGB color space, but I would highly suggest to have a conversation with your printers about what color space they desire for your submitted images.
Read more about AdobeRGB on Wikipedia
ProPhoto RGB and CMYK
ProPhoto RGB is another color profile that is designed for photographs. It is often the default color space of Adobe Lightroom. ProPhotoRGB has a very large gamut, which has its benefits in preserving exact colors.
CMYK is the exact colors that almost all printers use. Cyan, Magenta, Yellow, and Black. When you order toner for your printer, these are the options you have. Using this color profile has it’s benefits for printing, but once again, ensure that your printer wants color in this space before you send them the print file.
Read more about ProPhoto RGB and CMYK
Which Color Profile is Best for You
Hands down, using sRGB will give you the best most vibrant and alive colors for anything in the digital world. If it’s staying digital, use sRGB. Go ahead, export the exact same image in sRGB and AdobeRGB and compare them side by side on your monitor. Leave comments below which one looks best on your monitor!
If you’re printing, do ask your printer what color profile or space they desire. Often times professional printing houses will have information online (on their website) of how to prepare images for them. I have never met a printing house that didn’t enjoy having a conversation on the phone about what color profile to send them my art in.
Most of the time, they’ll want AdobeRGB. Note, this isn’t the same conversation as what’s better: TIFF or PDF. If they want sRGB, they very well might be converting the images once they receive them and prepare the image for proofing.
Hope this helps!
software: Adobe Photoshop
I have previously written about the differences of Color Profiles, but here I would like to talk about how to convert between the Color Profiles in Photoshop.
To begin with, ensure that Photoshop is open, and the image you’d like to convert is also opened within Photoshop.
From here, select: edit -> convert to profile
Edit and Convert to Color Profile in Adobe Photoshop
From here you are given a menu as shown below:
Convert to Profile selection window in Adobe Photoshop
There are a few items that should be of interest to you inside of here.
First, take a look at the color profile or color space that your image is currently in, ie. the Source Space. If you see above, the image is in sRGB color space.
Next, you can see there are many choices for Destination Space , which only you will know which to pick now!
For more information on what color space or profile you should be using, have a read of this article on the differences in sRGB, and AdobeRGB color profiles.
Hope this helps!
web server: NGINX
application: NGINX Amplify
NGINX Amplify is a great tool for monitoring the status of your NGINX deployment. It’s a new thing they’ve been working on and perhaps even still in beta mode.
Recently I was upgrading my Ubuntu based server, and the NGINX Amplify agent wouldn’t upgrade, giving an error along the lines of:
sudo apt-get upgrade
Setting up nginx-amplify-agent (0.40-1~trusty) ...
dpkg: error processing package nginx-amplify-agent
subprocess installed post-installation script returned error
exit status 1
Setting up libcurl3-gnutls:amd64 (7.35.0-1ubuntu2.10) ...
Setting up libcurl3:amd64 (7.35.0-1ubuntu2.10) ...
Setting up curl (7.35.0-1ubuntu2.10) ...
Processing triggers for libc-bin (2.19-0ubuntu6.9) ...
Errors were encountered while processing:
E: Sub-process /usr/bin/dpkg returned an error code (1)
Bummer. I didn’t really think much of this until I got an email from a fellow over at NGINX, so I thought I’d tell him this was happening. He asked me to share with him some more information, and we worked our way through a few things that I’ll outline below.
We tried a few things, but essentially the only thing that worked was:
I suggest you add hostname to the agent configuration manually
hostname = yourhostname
In this place, for yourhostname you may use any valid hostname, except local hostnames (‘localhost’,
‘localhost.localdomain’, ‘localhost6.localdomain6’, ‘ip6-localhost’).
After this, running:
sudo apt-get upgrade
worked, and everybody was happy again.
A note from support at NGINX Amplify:
Please note that current version of amplify-agent (0.40-1) contains a bug in the hostname detection logic, this is the reason why some users have problems with the upgrade. So, explicit assignment of hostname in the configuration is just a workaround.
Hope this helps!
In this article I’m going to drill into many technical nuances of what data structure names are on your computer. This will involve owners, permissions, folders, directories, any many quircks in between, also comparing the differences between Windows and Linux and Mac.
Most of us know what a folder is. The analogy is straight forward enough to know that we put files inside of folders. The files can be anything, from photographs to Word documents to more folders to random data files we have no idea what they’re for!
Everything on your computer sits inside some kind of folder. Even cloud repositories have folder systems.
Using folder systems allow the user to organize their data. Once again, the analogy is very clear here. Imagine a desk with files all over the place in little piles that overlap and mix with other piles and all it takes is one funny gust of wind to the desk and everything is scattered around the room. We don’t want this, so we organize our documents on our desk into file folders which then get sorted into a cabinet for future (easy) reference.
The trick here, of course, is to make effective folder systems. This is a conversation I’ll leave to another time. If you’re interested in this subject, it’s called ‘Data Structures’. There’s University courses on the topic and many a text books.
Directories or Folder Structure
C: folder directory
The term directory refers to folder structure on a computer. In the image above, I have shown the C: drive directory on my Windows computer. When I say C:, you can read this as C drive. The : implies it’s a drive. What’s before the : is the letter assigned to the drive, what’s after the : is a directory, or a folder list that resides on that drive. A drive is also known as a Hard Drive. It is possible to have drive partitions within one hard drive to make that single drive appear to the computer as two separate hard drives. In this case, when you mounted or plugged in your hard drive to your computer, it would show up as F: and G:, not just F:. On Windows computers, most of them now come with C: and D:. This isn’t physically two different hard drives, this has been made with a hard drive partition. Technically (read: usually) your computer will start assigning drive letters at E: and continue down the alphabet for each new connected hard drive. It does get confusing when you start mounting multiple hard drives, because one day it may show up as the G:, and the next day it may be the I:.
On a Linux system, this drive will get mounted as /dev/sdf (or /dev/xvdf on AWS instances).
Hard Drive vs. Virtual Drive
If you’re really getting fancy, you can create a Virtual Drive using a tool like Daemon Tools. This is handy when you have what’s known as an image file (a standard file format is *.iso). It is basically a snapshot of a drive or disc that gets saved into an *.iso file. You can then virtually mount that *.iso using Daemon Tools to view that entire original disc just like it was actually plugged into your computer. Almost exactly the same as what happens when you put a disc into your computer. Virtual drives are powerful because they’re usually ‘read only’, which means you can only read it, you can’t write files to it. So, things like DVDs or Video Games are often downloadable as a *.iso, which you mount using Daemon Tools, and carry on just as you would have if you put the DVD into your computer. After you’re done with it, you unmount and can go so far as to delete the file without actually having to throw a DVD out. We like less waste.
Root & Branches
The root of your directory is the base of the tree (there’s an analogy there). If you think of your C: as the base of your tree, you can imagine all of the folders within it branching out as you go deeper into each folder. So, let’s say inside the C: you have 10 folders, and in each of those folders you have another 10 folders,. We’re now at 100 folders. Inside of each of those 100 folders you have another 10 folders,. Now we’re at 1000 folders. Like a tree, the further you go up, the more branches you have. Always at the end of your folder digging you’ll find files, which you can think of as leaves or berries. The files hold the information, the folders just organize things.
The root is important to know because its basically the first door your computer is faced to look into, that is, is it C:? D:? E:? The root of the E: would just simply be E:\. While writing computer code, if you don’t explicitly define the root – like C:\python\program1\ – then you have to depend on the Operating Systems natural way of searching. A typical way you could expect this is to be local within the folder you’re working in, and then digging into the System Environment PATH locations. This is handy sometimes, but typically, to avoid any sort of confusion, it is very smart to define exactly where you want your program to run from.
It gets a bit tricky when we’re talking about programs here though, because the root folder for a program could just be the first folder for that program. So, if I’ve installed Microsoft Office on my computer, the root folder of that would be C:\Program Files\Microsoft Office. On older versions of Windows it was easier to see this location, but still can by clicking on the address bar in the top of your Explorer window that will automatically revert back to this format, as shown below:
Microsoft Office Folder
Local vs. Remote or External Drives
Every computer that is out there today has local hard drives inside of it. Local to the computer. Remote in this sense means not packaged directly inside of the computer, so external. A USB stick is a remote or external drive, your internal hard drive is local.
Your operating system runs on your local hard drive, no matter if it’s Mac, Linux, or Windows. You can do some fancy business by having an operating system on a USB stick and booting your computer onto that, which wouldnt’ be local, but that’s also another discussion for you advanced nerds out there.
Typically on Windows computers, the operating system resides in the C:\. On Linux, the operating system resides somewhere in the /etc/ folder, but also spreads out to many other folders, that is, anywhere under /. On a Linux machine, you have the capacity to manually mount a hard drive that you’ve attached to it (technically you can also do this on Windows). On a Linux machine, you can find the attached drives or devices under the folder /dev/. Depending on the year and the OS, the newly attached hard drive will show up as something like /dev/sdf , sdf being the name given to that now attached drive. So, using the command:
sudo mount /dev/sdf /myDrive
You give a name to your external hard drive “myDrive”, and reference it that way.
Permissions and Users
Operating Systems today are getting smart in that they make various levels of permissions to do various operations on your computer. For example, if you want to install a program, you usually need some kind of Super User or Administrator/Admin permissions to do so. If you’re not an Admin on your computer, your computer will prompt you to enter your username and password to install a program, or flat out deny you ability to install a program. Some of you may have seen this. It’s saying: “You’re doing something that will make some serious changes to your computer, are you super sure you want to be doing this?” This is a good feature that’s been around for a few years now. Some of us may have nostalgia about the first time, years ago now, being prompted upon start up to pick a user you wanted to log in as. It marked a big change in computing.
This all means that generally, your user isn’t an Admin. It’s kind of good so that you can’t do too much damage to your computer if you’re just messin’ around, but a pain in the arse if you’re trying to install any new programs. Your program wont install, and you’ll be told you don’t have proper permissions to do so.
As the Admin , installing programs, you’re usually asked if you’d like to make the program available to specific users on the computer, or only to the Admin. This means that as the Admin of a computer, you can specify who can read (r), write (w), or execute (x) specific files and folders.
- Reading means you can view the contents of the file, but not edit them.
- Write means you can read the files, and also change the files and save them with the new content in them.
- Execute means that you can run the files, so it’s really only relevant for program files, not really text files.
So let’s say I’m working as User1, and I’m trying to edit a Word document, but I’ve only got read permission on the file. When I go to save it, I’ll either get an error saying I do not have permissions to save that file, or to save it as a new file. Saving it as the old file name will not be allowed, User1 won’t be allowed to do that because it doesn’t have the right permissions.
Understanding this, one can see how certain security features could be added to files that are being shared among various people. This is also the basis of how file repositories work, specifically online clouds like Google Docs. If one person is editing the file, it’s locked to only Readable for anybody else with access to it. They can read it still, but they can’t make changes to it. Google Docs have become pretty powerful now, in that you can actually see edits being made in real time to documents that this locking has almost become invisible.
User’s Home Folders
Each user has a home directory. They load unique for each user. So, unique users will have unique Music, Pictures, Videos, Downloads, etc. folders. Remember, the Admin user will have access to all of these folders, but other users on the same computer will not be able to change other users’ files. Other users may be able to dig and read them, so it’s not enough to depend on this level of file security by just changing users. File encryption is highly suggested here.
Each operating system will have a different technical definition of Home Directory, but usually it’s with respect to the specific user. This term is more of a Linux or Mac term, and the home directory is actually labelled as home: /home/user1/. So, inside the folder /user/ will be the home directory for that user.
This term could also be applied to the home directory for a program, such as /etc/nginx/, or C:\\Program Files\Microsoft\.
Windows vs. Linux and Mac OS
All of this gets a bit complicated when you switch operating systems. Above I’ve mostly referenced a Windows Operating System. However, Linux runs much the same, except for assigning drive letters.
Instead of drive letters, Linux and Mac usually assign a name and load it into the /dev/ directory.
For example, when you plug in your USB stick to your Windows computer, it will automatically mount as, let’s say, F:. When you plug in your USB stick to your Linux computer, it will automatically mount as just the USB sticks name (which might be something like ‘Seagate Backup Plus Drive‘) which will show up on the desktop, and also inside the dev folder as: /dev/sdf.
Formatting, Encrypting and Naming Drives
Formatting a drive means that you’re re-writing the entire drive structure to be like a clean slate. You’re erasing all the content and leaving it empty. Like painting a wall, emptying out the garbage can, or even buying a new hard drive. New hard drives usually come pre-loaded with a bunch of files though, some will be security programs, some will be technical documents, some will be backup programs. It’s up to you if you like to keep all this stuff, but I prefer to use my favorite programs instead of the supplied ones.
I now encrypt all of my hard drives using VeraCrypt software. They have extensive documentation on how and why. Encryption must be done before you put anything on a hard drive. When you encrypt a hard drive, it also formats it with the option to give it a name. But generally, when you mount the encrypted hard drive you assign it a name then, and mounting is the necessary step for encrypted hard drives because when you mount an encrypted hard drive, it then prompts you for the password to unlock the encrypted hard drive. This is same for Windows and Linux.
If you’re formatting your drive, you can give it a name of your choosing, but most folks who get a new USB drive just plug it in and go right on using it as it came from the factory.
As discussed previously, when you mount a hard drive in Linux, you can mount it with any name you wish. Mounting doesn’t re-name the actual hard drive itself, nor does it format it, it just gives an additional reference name to that device that your computer will recognize.