Bad SD cards in a NEMS Server Raspberry Pi

With the coming release of NEMS 1.3, the retroactive stats system was implemented to allow me a way to see how many systems have been upgraded, and if any are having trouble.

Now, behind the scenes, I also developed the anonymous stats system as a way for me to gauge system load on various platforms.

Why would I do that? Because NEMS is branching out onto other SBC’s, starting with the ODROID XU4.

The fact remains, I often get the question “How many hosts can I add to NEMS” and the truth is, I have no idea! I’ve received great feedback from users who are monitoring 100+ hosts on a Raspberry Pi NEMS server, but otherwise I don’t really know much about what’s actually going on in the real world of NEMS.

So the stats system shows me, anonymously, how NEMS is being used, and how it is performing.

So, that system has been running for a few days, and today I began seeing some alerts: some users have bad SD cards!

This is causing their NEMS server to run very, very slowly.

If your NEMS server is running slowly, you may be one of these! Remember, the stats are anonymous, so I can’t contact you.

I’ve considered making the stats system able to send back a response to your NEMS server… that way, even though it’s anonymous and I don’t know who you are, I can have the system programmatically warn you that your card is failing.

It’s really quite cool for me to see how NEMS is being used. So far more than 100 NEMS servers have reported in with their current statistics, and many are running more than 100 nagios monitors with a 0.01 load average.

So back to my comment about branching out to other SBCs and knowing how many hosts we can handle on NEMS: the fact is, now that I will have real world statistical data, I will be able to put together proper information about how many services can be running on NEMS on various platforms! It’ll start with the Pi 3 obviously, but I’ll be able to see for example, how a NEMS server with 500 hosts has a higher load than one with 100, so we can therefore recommend if you have 500 hosts or more, you should install on an ODROID XU4 (for example). I’ll also be creating new graphs that show comparisons of various platforms, performance graphs over time, averages based on user deployments, and so-on. Really exciting stuff!

Thanks for using NEMS! If you have any questions, I’m here for you!

Robbie

Why support may ask for your backup.nems file, and why you should never share it with others

During support sessions I often request either SSH access or a copy of the user’s backup.nems file. SSH access, it should be obvious, should not be shared with just anyone. Also, you should never, ever, ever, open SSH to the world on your NEMS server if you have not yet initialized it. This is because there are botnets that look for Raspberry Pi computers which use the default “raspberry” password, and then compromize them. Continue reading

Configuring email notifications in NEMS

A step-by-step guide to configuring Email Notifications on your NEMS Server. Continue reading

NEMS 1.2.1 Technical Information

The following is the technical information outline of NEMS (Nagios Enterprise Monitoring Server). Continue reading

NEMS Changelog Archive

This is the historical changelog. For the latest version of NEMS please visit baldnerd.com/nems
Continue reading

Backup your NEMS configuration files automatically

One of the worst things that can happen to your NEMS deployment is having your SD card fail. So keeping a current backup of your NEMS configuration is a smart idea.

Your NEMS Migrator snapshots are always accessible at http://NEMSIP/backup/ and will automatically generate and send a backup.nems file, which contains all the NEMS configuration settings, logs, data, etc. to allow an easy recovery by restoring to a new NEMS deployment.

Knowing this, it’s easy to add a NEMS backup to your daily backup script.

From your Linux server (where your backups run), simply add this to your backup task:

… where /backup/backup.nems is where you want it to output the download, NEMSIP is the actual internal IP address of your NEMS server and YOURUSER and YOURPASSWORD are those you set during nems-init. From there, I recommend you have your backup script run an rdiff-backup of your /backup folder (in this example) to allow for versioning.

Setting up NRPE on Windows for NEMS

Please Note: As of NEMS 1.2 NSClient++ is optional for monitoring of Windows computers (thanks to the addition of WMIC). If you’d like to use it, please follow the directions below.

  1. Grab the latest Windows client at https://www.nsclient.org/download/
  2. Install the client with the following settings:
    • Select to install the “Generic mode” NSClient++.
    • Choose “Complete installation” and if asked, choose to save config to ini file.
    • Under “Allowed Hosts” it should read 127.0.0.1,NEMSIP (where NEMSIP is the IP address of your NEMS server)
    • Clear the Password field for ease of deployment. NEMS sample scripts are setup to use NRPE without a password because I’m making the assumption that this is being deployed in a trusted LAN. If you do not blank the password here, you will have to edit all the scripts before NEMS will be able to communicate with this computer.
    • Enable all modules and change the NRPE mode to Legacy. NEMS uses Nagios 3.5.1 at present, and I suppose that’s technically “Legacy”. 🙂
    • Screen should look a little something like this:
      nsclient-setup
  3. Add your Windows host to NEMS. If you are using NEMS 1.1+ you can use the template “ourwinserver” in nconf. Just change the hostname and the IP address.

Please note: If you have a software firewall running on your Windows machine, setup an exception for your NEMS server IP to gain access through ports 5666 and 12489.

NEMS Migrator: Upgrade NEMS 1.0 or nagiospi to latest NEMS

Thanks for being an early-adopter! Whether you’re coming from NEMS 1.0 or its predecessor, nagiospi, I want to make it as easy as possible for you to get the latest and greatest, without having to reconfigure everything. It’s been exciting to see the NEMS project really catching on, and I endeavor to make it the best it can be. Your suggestions along the way have helped me focus on some great features for as NEMS continues to evolve.

NEMS 1.1+ has a nifty backup and export tool called NEMS Migrator. While it comes pre-packaged in 1.1+, I designed it specifically to run on legacy builds as well (NEMS 1.0 or nagiospi), giving you the opportunity to export your old configuration, deploy the latest version of NEMS, and then restore the configuration to NEMS. Easy peasy!

Here’s what you need to do:

Note: These instructions are for NEMS 1.0 or nagiospi only. Do not do this on NEMS 1.1+ as the tool is already built-in.

  1. SSH into your NEMS/nagiospi server.
  2. Become root: sudo su
  3. Update repository data. Type: apt-get update
  4. Install Git. Type: apt-get install git
  5. Install NEMS-Migrator in /tmp. Type:
    cd /tmp && git clone https://github.com/Cat5TV/nems-migrator
  6. Create the backup config on your NEMS/nagiospi system. Type:
    • If on NEMS 1.0: cd /tmp/nems-migrator && ./backup.sh
    • If on nagiospi: cd /tmp/nems-migrator && ./nagiospi2nems.sh
  7. Download the backup to your computer by opening it in your web browser. In your favorite web browser, simply add /backup/ to the end of your NEMS/nagiospi server address. Eg., http://10.0.0.5/backup/
  8. Now that you have your backup.nems file, follow the instructions here to restore your configuration to a new version of NEMS.

NEMS Migrator: Restore

The NEMS Migrator tool allows you to export/backup your NEMS configuration (backup.nems) as well as import a previous backup (through the Restore option).

The NEMS Migrator’s backup and restore options are great for keeping a safe backup without having to shutdown your NEMS server. Just download the file once in a while, or back it up automatically with your daily backup script.

NEMS Migrator is also helpful when upgrading from previous versions of NEMS, saving you having to reconfigure your NEMS deployment just to get the latest features.

Important Note
I am a firm believer in redundancy, and protecting your data. What I’d like you to do is, export your migration file, then install NEMS on a new MicroSD card. Then boot from that and restore your NEMS Migrator backup. Once you’ve confirmed everything worked well, you can deprecate the old one safely. However, if something went wrong, you can contact me to fix it for you, and continue running from the old SD card in the interim.

How to Restore a NEMS Migrator Backup
Requires NEMS 1.2+

  1. Place your backup.nems file on a USB flash drive. You can access this directly from your web browser at http://NEMSIP/backup/ where NEMSIP is the IP address of the NEMS server you wish to backup.
  2. Deploy the version of NEMS you wish to restore the backup to. Please heed my Important Note above.
  3. Boot the new NEMS deployment and mount the USB flash drive.
  4. Determine the location of backup.nems in relationship to your mountpoint. For example, if you mounted the USB flash drive on /mnt/flash you may determine the location to be /mnt/flash/backup.nems
  5. Armed with that information, run the following command (use the full path to your backup.nems file):
  6. Follow the prompts on screen to restore your configuration to the new NEMS deployment. If it fails for any reason, you can safely shut down and replace the SD card with your original deployment.

If you have any problems (or praise) please comment below.

Installing the Nagios NRPE Client Agent on Debian / Ubuntu

The Nagios Remote Plugin Executor (NRPE) allows your Nagios Enterprise Monitoring Server to communicate with the Linux machines on your server to determine things like free disk space, CPU load, and detect possible issues that a simple ping can’t determine.

There are countless instructions online to download tar.gz files and install manually, or use a PPA to install via apt-get, but you’ll be surprised to note the needed packages are in fact already in your Debian (and by proxy, Ubuntu) repositories.

To install the needed NRPE client on Debian / Ubuntu / other Debian-based Linux operating systems:

Don’t forget that you need to be root (Debian) or use sudo (Ubuntu).

Next, we just have to tell NRPE that it’s allowed to communicate with our Nagios server. On the client system, open the file /etc/nagios/nrpe.cfg

Find the line that reads: allowed_hosts=127.0.0.1

Now there are a few ways we can allow our server. First (and most obvious) is to add its IP address like this:

Where 192.168.0.5 is our Nagios/NEMS server.

Alternatively we can tell NRPE that it’s allowed to communicate with any local system:

Now, save the file and restart NRPE as follows:

And there we have it! Your Nagios/NEMS server should now be able to see your Linux machine.

Looking for a lightweight, affordable, easy-to-deploy enterprise monitoring server? Check out Nagios Enterprise Monitoring Server for Raspberry Pi 3!