Shell script to monitor server load

Don’t feel like shelling out $$$ for NewRelic or Blackfire.io? Got a micro instance hosting a blog that nobody reads that doesn’t have the memory to support enterprise monitoring anyway (*cough* this blog *cough*)?

Here’s a shell script that will email you when server load gets above whatever threshold you specify. Would be pretty easy to adapt to monitor memory using the ‘free‘ command as well. Just schedule it using ‘crontab -e‘, like ‘*/5 * * * * /path/to/script.sh‘ for every 5 minutes, and you’re set.

Server load monitoring on a budget!

#!/bin/bash
# requires bc library - yum install bc

# config
# alert threshold, as decimal
ALERT=.7; # 70% CPU utilization
# admin email
EMAIL='[email protected]';

# add /usr/bin to path so cron works
export PATH=$PATH:/usr/bin;

# get number of processors
NPROC=`nproc`;
# get first utilization metric
UTIL=`cat /proc/loadavg | cat -d ' ' -f 1`;

# divide util by number of processors, accounting for 0.00 util
RESULT=`bc <<< "scale = 2; $UTIL / $NPROC"`; 

# email alert if util is greater than alert threshold 
if [[ `bc <<< "$RESULT > $ALERT"` -eq 1 ]]
then
  # calculate a percentage
  PERC=`bc <<< "scale = 2; $RESULT * 100"`;
  echo "CPU utilization is above threshold at $PERC %";
  # add top output to email
  TOPOUTPUT=`top -n 1 -b`;
  `/usr/bin/mailx -s "Utilization high on $HOSTNAME" -r "$EMAIL" "$EMAIL" <<< "CPU on $HOSTNAME is at $PERC % 

$TOPOUTPUT"`;
# else
  # echo "Utilization is only $RESULT";
fi

You can test it by generating CPU load with the stress utility if you like.

Google Analytics – take daily with a grain of salt

Haven’t messed with Google Analytics in literally years, but since I’m getting a new site up I logged in. I noticed these spikes on all of my accounts for different sites, even sites that do not exist anymore:

WtfGoogleAnalytics
Most of the traffic was from Great Britain. I’m sure there’s an explanation… that I don’t have the time or motivation to uncover.

My point is – approach your analytics data with a touch of healthy skepticism.

Real Life Security Fails – How a Lost Password Sent Thousands of Magento Credit Cards to Eastern Europe

All of the server security on the planet is useless if you don’t keep track of your passwords.

One of the most effective yet hardest to detect attacks against Magento I’ve seen to date requires nothing but a medium-level admin login and the ability to copy/paste an 8th grade level piece of code. Best (worst?) of all, you won’t even know if you’re infected, because it causes no obvious changes in the site, neither creates nor edits any files, slides under every website malware checker I know of, and doesn’t even show anything shady in webserver access logs.

I have seen this exploit quietly shuttle customer identity and credit card data to attack servers for literally *months* before anyone noticed.

Also, I don’t mean to pick on Eastern Europe, but if we’re being honest, most good malware comes out of Russia/China/Eastern Europe (Ukrainian Magento developers are some of the best). Blame a high cultural importance placed on STEM education and a fluctuating economy. The times I’ve encountered this hack in the wild, it’s usually been shuttling data to that general area.

The hack goes about like this:

1. Attacker gets a hold of an admin login, maybe by brute-force password guessing, but more likely by getting a hold of an email or some other electronic correspondence, probably from a trusted or formerly-trusted 3rd party like an overseas developer.
2. Attacker logs into admin.
3. Attacker goes to System > Configuration > Design > Footer

MagentoAddMiscFooterJavascript

4. Attacker pastes in a piece of javascript that gets loaded in the footer of every page, that basically simply says

if this is the checkout page
when the submit button is pressed
grab the customer’s data,
create a URL like http://attacker-domain.com/collector.php?firstname=John&lastname=Doe&CcNumber=1234…
and send a request to that URL.

The most devious part of this, besides the fact that it takes minimal technical skills, is that the script doesn’t actually load anything into the page, so malware scanners won’t notice it. All the attacker has to do is write a script on their own server that grabs the data out of the URL parameters and they’ve got all the customer’s identity information and their credit card number.

Counter Measures

1. AFAIK, the only 100% effective countermeasure against this hack is to do a checkout with your browser running through a proxy, record all network traffic, and then verify the identity of all the servers involved.

2. A better-than-nothing test is to request the page via script, preferably by driving a headless browser like PhantomJS, use some regex magic to parse out any URLs, and then verify the server location via IP-to-country API.

3. Rotate passwords and keep track of who is logging into your Magento.

  • I wrote some shell scripts that perform the second option, but unfortunately I don’t own the code so I can’t release them 🙁
  • I’m working on a Selenium/NodeJS system that performs the first (superior) check, and I promise I’ll either open source the components or provide a free version 🙂
  • I just finished an early-but-functional version of a free extension that logs all of your admin area logins -https://github.com/siliconrockstar/magento-admin-login-logFuture versions may support IP geolocation. I’ve also got plans in the works for other extensions that add enterprise-level admin user security features like password rotation enforcement, adjustable password difficulty enforcement, etc.