NEMS 1.2.1 Technical Information

The following is the technical information outline of NEMS (Nagios Enterprise Monitoring Server).

NEMS 1.2.1

Release Date: TBD

Download Latest Version: baldnerd.com/nems

General Application Versions

  • Debian: 9 (Stretch)
  • Linux Kernel: 4.9.28
  • Nagios: 3.5.1
  • PHP: 7.0.19
  • Apache: 2.4.25
  • MariaDB: 10.1.22
  • NEMS nConf: 1.4
  • Webmin:
  • Check_MK Multisite:
  • pnp4nagios:
  • nagios-api: 1.2.2
  • WMIC: 1.3.14
  • WMI Plus: 1.62

Additional Included Software Commands

  • Web Browser: w3m

Nagios Live Data Socket

  • Located at /var/lib/nagios3/rw/live.sock

nagios-api

  • The JSON API is active by default on port 8090.
  • If desired, you can disable nagios-api by editing the root cronjob which enables the API at NEMS boot.

Default Passwords (And How To Change Them)

  • Web access password is created during initialization. To initialize NEMS 1.2, use the command: sudo nems-init
    The easiest way to change this password after NEMS is already initialized, simple re-run the initialization script. If you wish to leave your pi user password as is, just enter the existing password again during init, and you can press CTRL-C when prompted to abort expanding the filesystem (as it’s unnecessary to do it again).
    If you’re more adventurous and prefer to do it yourself, you can instead add an htpasswd to /var/www/htpasswd and then change the username in both /etc/nagios3/global/contactgroups.cfg and /etc/nagios3/global/contacts.cfg to give the new user access to Nagios Core. But again, all this stuff is automated with sudo nems-init so you can save yourself some trouble by just doing that.
  • NagVis: admin/admin
    Login using the default password, and immediately choose User menu→Change Password
  • MySQL: root/nagiosadmin
    Best leave this as is since changing it would break a lot of stuff. Just never open up your MySQL port to the world, that’s all.
    Also important to note that if you manually add any databases or tables, you will lose them if you run a nems-init or a NEMS-Migrator restore. Best leave MySQL alone, and put in a feature request in the comments if you need to add something.

WMIC

  • Quick way to test if WMIC is working from within a NEMS SSH seession:

    Where WINDOWSIP, WINDOWSUSER and WINDOWSPASS are the actual IP of the Windows computer, and the administrator user/password for that computer.
    Run it twice, and then if it’s successful you should see something like this:
  • WMIC takes up a fair bit of space on the Pi’s SD card, which resulted in NEMS 1.2 minimum system requirement being 8 GB.

NEMS Changelog Archive

 

This is the historical changelog. For the latest version of NEMS please visit baldnerd.com/nems

Version 1.1 – November 13, 2016
– NagVis upgraded to 1.8.5. (1.9 is still beta).
– Check_MK livestatus upgraded to 1.2.8p13.
– Added Check_MK Multisite 1.2.8p13.
– Added PNP4Nagios 0.6.16-2.
– Added a few sample configurations to NConf to help users figure out the initial setup and/or to use as templates. Included samples are: test if an external web site is up via ping, monitor a Linux server, monitor a Windows server.
– Created NEMS Migrator. New feature allows backing up and restoring your NEMS configuration, making migration or recovery a breeze.
Upgrade from NEMS 1.0.
– Added sendemail 1.56-5 and setup email config in /etc/nagios3/resource.cfg (you’ll need to add your SMTP info as per the instructions above).
– Added git, htop.
– raspi-config upgraded to 20161108.
– Linux kernel upgraded to 4.4.26-v7.
– Apache2 upgraded to 2.4.10-10+deb8u7.
– OpenSSL upgraded to 1.0.1t-1+deb8u5.
– MySQL upgraded to 5.5.52-0+deb8u1.
– Exim4 upgraded to 4.84.2-2+deb8u1.
– PHP upgraded to 5.6.27-0+deb8u1.
– PHPMyAdmin upgraded to 4.2.12-2+deb8u2.
– Python upgraded to 2.7.9-2+deb8u1.
– Network and Bluetooth firmwares upgraded.
– Various system components upgraded.
– Reduced the amount of memory dedicated to Raspberry Pi graphics adapter.

Version 1.0 – May 8, 2016 (Discontinued November 13, 2016, Downloaded 409 Times)
– Initial release. Built and tested on Raspberry Pi 3. Based on Raspbian Jessie. Inspired by NagiosPi, which in April 2016 was still running on the old Raspbian Wheezy. I started this new distro since NagiosPi seems to be out of date, and I want to have an easy drop-in Nagios img for the Raspberry Pi. Figured I’d share it with the world while I’m at it since there are probably others (possibly less tech savvy) who might want the same thing. I decided to leave most of the settings the same as NagiosPi (eg., usernames, passwords) so those coming from that distro can seamlessly transition, or so if NagiosPi wants to use our build to bring things up to date, they may do so with minimal effort.
– This initial build is using default repositories in a lot of cases and is meant to be rock-solid, not bleeding edge (eg., Nagios 3.5.1 instead of Nagios 4.1.1.
– Using the rpi-4.4.y Linux kernel tree (Currently 4.4.7-v7+ #876 SMP), firmware updated to 1e84c2891c1853a3628aed59c06de0315d13c4f1. Use rpi-update to check for upgrades, if needed.
– Includes rpi-update tool – an easier way to update the firmware on the Raspberry Pi – See https://github.com/Hexxeh/rpi-update
– On-board Bluetooth disabled due to potential stability issues. Use rpi-update to check for kernel updates and see if it is fixed, and then edit /boot/config.txt to re-enable. Until they fix it, use USB Bluetooth dongle if needed.
– Installed and configured: mysql-server mysql-client phpmyadmin apache2 nagios3 nagios-nrpe-plugin
– To keep things consistent for those coming from NagiosPi, I have used the same passwords. MySQL is: User: root Pass: nagiosadmin
– Installed w3m web browser to allow local testing in terminal: w3m localhost/phpmyadmin
– Manually installed NConf 1.3.0-0 “Final”, an Enterprise Nagios configuration tool. This tool was broken on NagiosPi’s instructions due to a missing symlink at /var/www/nconf, so I fixed that in my version. Access NConf via the “Configure Nagios” link on the main menu.
– Includes NagVis 1.7 – want to do 1.9 but not until out of beta.
– Built and integrated the first version of our menu system, which includes the first version of a custom Nagios skin to begin integrating a more modern interface. Menu accessible at http://nems/ (or http://IPADDRESS if that doesn’t work for you)
– Added a nice little MOTD.
– Added a simple cronjob to check our web site for the currently available version and warn you if yours is out of date.

Add A Drive to Linux and Encrypt It

We’re going to be featuring encrypted removable backups on Category5 Technology TV, and here’s a quick list of the steps we’ll be using to prepare the hard drive.

Everything we’re about to do requires running terminal as root.

In Debian, become root as follows:

In Ubuntu (and other “sudo”-based environments):

First things first:
Create a partition on your device, which we'll call /dev/whatever1. If you run luksFormat against the device itself (as opposed to a partition on the device) you will receive the error "no key available with this passphrase" when you try to run luksOpen, and will not be able to open the volume. If you accidentally do that, you can use a partition editor like gparted to change the drive to "unallocated" and try again.

Install cryptsetup:

Make the drive encrypted (destructive), 512-bit :

Show result:

Map the drive:

This will ask you for the passphrase and then creates a new mapper at /dev/mapper/backup

The reason I first go to /tmp is just in case there is a ./backup folder where I am currently situated within the filesystem. This could cause problems, so moving to /tmp removes the risk (unless there is a /tmp/backup, of course).

Create the filesystem (format):

You can now test mounting the drive if you like:

Create a key file so you can auto-mount the drive (without having to enter the keyphrase). Only root should have access to this file:

Add the keyfile to our LUKS drive:

Enter your passphrase when prompted.

Do another dump and you should now see Key Slot 1 has a key (from your key file):

Now we need to determine the UUID of your LUKS-encrypted partition. This will be different than the actual physical UUID, so we have to use cryptsetup to find it:

 

Setup a crypttab entry:

Add the following:

Start the crypto disk (replace backup with whatever you called it in the crypttab file):

Create your permanent mountpoint wherever you’d like and make it so you can’t write to it unless it’s mounted. For my example I’ll place it in /home/robbie/backup

Open your fstab file for editing:

Add your encrypted partition to the permanent mountpoint by adding this line:

nofail means if the drive is not present, keep booting. noatime means access times are not updated when a file is read (read operations are read only: don’t use resources or reduce the life of the drive with write operations when not necessary). Our x-systemd.device-timeout setting means the mount will skip the drive if it is not plugged in after 5 seconds. The default is 90 so this speeds up boot big time.

Test to make sure everything worked:

Do not reboot until you get a good result. 😀

Side note: If the drive is a USB drive, make sure you disable usbcore autosuspend, which will periodically turn off your USB, thereby breaking your mountpoint. On Debian I did this by editing /etc/default/grup and adding usbcore.autosuspend=-1 to GRUB_CMDLINE_LINUX_DEFAULT – you can confirm it worked by rebooting and then typing: cat /sys/module/usbcore/parameters/autosuspend – Here is some great info for other distros: http://unix.stackexchange.com/posts/175035/revisions

I hosted an episode of The Pixel Shadow!

I’m so not a gamer, but over the weekend I got the fun chance to record an episode of The Pixel Shadow with me as host.

It was a blast to share some of the things I’ve learned about Minetest in a fun way. The kids won’t even realize they’re learning stuff!  😀

Check out minetest.tv for more and to download the free game.

UPDATE
That episode was such a success (Over 5,000 views and 108 likes in under a week!) that we decided to do it again!

Looks like we’re re-shaping The Pixel Shadow a bit and I’ll get to host several upcoming episodes as my BaldNerd character 😀

Backup your NEMS configuration files automatically

One of the worst things that can happen to your NEMS deployment is having your SD card fail. So keeping a current backup of your NEMS configuration is a smart idea.

Your NEMS Migrator snapshots are always accessible at http://NEMSIP/backup/ and will automatically generate and send a backup.nems file, which contains all the NEMS configuration settings, logs, data, etc. to allow an easy recovery by restoring to a new NEMS deployment.

Knowing this, it’s easy to add a NEMS backup to your daily backup script.

From your Linux server (where your backups run), simply add this to your backup task:

… where /backup/backup.nems is where you want it to output the download, NEMSIP is the actual internal IP address of your NEMS server and YOURUSER and YOURPASSWORD are those you set during nems-init. From there, I recommend you have your backup script run an rdiff-backup of your /backup folder (in this example) to allow for versioning.

Setting up NRPE on Windows for NEMS

Please Note: As of NEMS 1.2 NSClient++ is optional for monitoring of Windows computers (thanks to the addition of WMIC). If you’d like to use it, please follow the directions below.

  1. Grab the latest Windows client at https://www.nsclient.org/download/
  2. Install the client with the following settings:
    • Select to install the “Generic mode” NSClient++.
    • Choose “Complete installation” and if asked, choose to save config to ini file.
    • Under “Allowed Hosts” it should read 127.0.0.1,NEMSIP (where NEMSIP is the IP address of your NEMS server)
    • Clear the Password field for ease of deployment. NEMS sample scripts are setup to use NRPE without a password because I’m making the assumption that this is being deployed in a trusted LAN. If you do not blank the password here, you will have to edit all the scripts before NEMS will be able to communicate with this computer.
    • Enable all modules and change the NRPE mode to Legacy. NEMS uses Nagios 3.5.1 at present, and I suppose that’s technically “Legacy”. 🙂
    • Screen should look a little something like this:
      nsclient-setup
  3. Add your Windows host to NEMS. If you are using NEMS 1.1+ you can use the template “ourwinserver” in nconf. Just change the hostname and the IP address.

Please note: If you have a software firewall running on your Windows machine, setup an exception for your NEMS server IP to gain access through ports 5666 and 12489.

NEMS Migrator: Upgrade NEMS 1.0 or nagiospi to latest NEMS

Thanks for being an early-adopter! Whether you’re coming from NEMS 1.0 or its predecessor, nagiospi, I want to make it as easy as possible for you to get the latest and greatest, without having to reconfigure everything. It’s been exciting to see the NEMS project really catching on, and I endeavor to make it the best it can be. Your suggestions along the way have helped me focus on some great features for as NEMS continues to evolve.

NEMS 1.1+ has a nifty backup and export tool called NEMS Migrator. While it comes pre-packaged in 1.1+, I designed it specifically to run on legacy builds as well (NEMS 1.0 or nagiospi), giving you the opportunity to export your old configuration, deploy the latest version of NEMS, and then restore the configuration to NEMS. Easy peasy!

Here’s what you need to do:

Note: These instructions are for NEMS 1.0 or nagiospi only. Do not do this on NEMS 1.1+ as the tool is already built-in.

  1. SSH into your NEMS/nagiospi server.
  2. Become root: sudo su
  3. Update repository data. Type: apt-get update
  4. Install Git. Type: apt-get install git
  5. Install NEMS-Migrator in /tmp. Type:
    cd /tmp && git clone https://github.com/Cat5TV/nems-migrator
  6. Create the backup config on your NEMS/nagiospi system. Type:
    • If on NEMS 1.0: cd /tmp/nems-migrator && ./backup.sh
    • If on nagiospi: cd /tmp/nems-migrator && ./nagiospi2nems.sh
  7. Download the backup to your computer by opening it in your web browser. In your favorite web browser, simply add /backup/ to the end of your NEMS/nagiospi server address. Eg., http://10.0.0.5/backup/
  8. Now that you have your backup.nems file, follow the instructions here to restore your configuration to a new version of NEMS.

NEMS Migrator: Restore

The NEMS Migrator tool allows you to export/backup your NEMS configuration (backup.nems) as well as import a previous backup (through the Restore option).

The NEMS Migrator’s backup and restore options are great for keeping a safe backup without having to shutdown your NEMS server. Just download the file once in a while, or back it up automatically with your daily backup script.

NEMS Migrator is also helpful when upgrading from previous versions of NEMS, saving you having to reconfigure your NEMS deployment just to get the latest features.

Important Note
I am a firm believer in redundancy, and protecting your data. What I’d like you to do is, export your migration file, then install NEMS on a new MicroSD card. Then boot from that and restore your NEMS Migrator backup. Once you’ve confirmed everything worked well, you can deprecate the old one safely. However, if something went wrong, you can contact me to fix it for you, and continue running from the old SD card in the interim.

How to Restore a NEMS Migrator Backup
Requires NEMS 1.2+

  1. Place your backup.nems file on a USB flash drive. You can access this directly from your web browser at http://NEMSIP/backup/ where NEMSIP is the IP address of the NEMS server you wish to backup.
  2. Deploy the version of NEMS you wish to restore the backup to. Please heed my Important Note above.
  3. Boot the new NEMS deployment and mount the USB flash drive.
  4. Determine the location of backup.nems in relationship to your mountpoint. For example, if you mounted the USB flash drive on /mnt/flash you may determine the location to be /mnt/flash/backup.nems
  5. Armed with that information, run the following command (use the full path to your backup.nems file):
  6. Follow the prompts on screen to restore your configuration to the new NEMS deployment. If it fails for any reason, you can safely shut down and replace the SD card with your original deployment.

If you have any problems (or praise) please comment below.

Installing the Nagios NRPE Client Agent on Debian / Ubuntu

The Nagios Remote Plugin Executor (NRPE) allows your Nagios Enterprise Monitoring Server to communicate with the Linux machines on your server to determine things like free disk space, CPU load, and detect possible issues that a simple ping can’t determine.

There are countless instructions online to download tar.gz files and install manually, or use a PPA to install via apt-get, but you’ll be surprised to note the needed packages are in fact already in your Debian (and by proxy, Ubuntu) repositories.

To install the needed NRPE client on Debian / Ubuntu / other Debian-based Linux operating systems:

Don’t forget that you need to be root (Debian) or use sudo (Ubuntu).

Next, we just have to tell NRPE that it’s allowed to communicate with our Nagios server. On the client system, open the file /etc/nagios/nrpe.cfg

Find the line that reads: allowed_hosts=127.0.0.1

Now there are a few ways we can allow our server. First (and most obvious) is to add its IP address like this:

Where 192.168.0.5 is our Nagios/NEMS server.

Alternatively we can tell NRPE that it’s allowed to communicate with any local system:

Now, save the file and restart NRPE as follows:

And there we have it! Your Nagios/NEMS server should now be able to see your Linux machine.

Looking for a lightweight, affordable, easy-to-deploy enterprise monitoring server? Check out Nagios Enterprise Monitoring Server for Raspberry Pi 3!

Category5 TV Network license changed.

For the past 9 years, all Category5 TV Network programming has been licensed under Creative Commons Attribution 2.5 Canada.

In an effort to ensure both our own protection from the commercial reuse of our freely available content as well as to protect our viewers from companies adding protection such as DRM to our content, we are now moving all Category5 TV Network programming, retroactively, to the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0).

Effective immediately under our new license, any commercial reuse of our material (eg., broadcasting on a commercial television channel, using our videos to generate revenue online, etc) must be approved in writing by myself.

Category5 TV remains entirely free for its viewers, no matter where they live in this big ol’ world of ours.

Enjoy the shows!

Robbie