During support sessions I often request either SSH access or a copy of the user’s backup.nems file. SSH access, it should be obvious, should not be shared with just anyone. Also, you should never, ever, ever, open SSH to the world on your NEMS server if you have not yet initialized it. This is because there are botnets that look for Raspberry Pi computers which use the default “raspberry” password, and then compromize them. Continue reading
A step-by-step guide to configuring Email Notifications on your NEMS Server. Continue reading
The following is the technical information outline of NEMS (Nagios Enterprise Monitoring Server). Continue reading
One of the worst things that can happen to your NEMS deployment is having your SD card fail. So keeping a current backup of your NEMS configuration is a smart idea.
Your NEMS Migrator snapshots are always accessible at http://NEMSIP/backup/ and will automatically generate and send a backup.nems file, which contains all the NEMS configuration settings, logs, data, etc. to allow an easy recovery by restoring to a new NEMS deployment.
Knowing this, it’s easy to add a NEMS backup to your daily backup script.
From your Linux server (where your backups run), simply add this to your backup task:
wget -O "/backup/backup.nems" http://NEMSIP/backup/ --user=YOURUSER --password=YOURPASSWORD
… where /backup/backup.nems is where you want it to output the download, NEMSIP is the actual internal IP address of your NEMS server and YOURUSER and YOURPASSWORD are those you set during nems-init. From there, I recommend you have your backup script run an rdiff-backup of your /backup folder (in this example) to allow for versioning.
Please Note: As of NEMS 1.2 NSClient++ is optional for monitoring of Windows computers (thanks to the addition of WMIC). If you’d like to use it, please follow the directions below.
- Grab the latest Windows client at https://www.nsclient.org/download/
- Install the client with the following settings:
- Select to install the “Generic mode” NSClient++.
- Choose “Complete installation” and if asked, choose to save config to ini file.
- Under “Allowed Hosts” it should read 127.0.0.1,NEMSIP (where NEMSIP is the IP address of your NEMS server)
- Clear the Password field for ease of deployment. NEMS sample scripts are setup to use NRPE without a password because I’m making the assumption that this is being deployed in a trusted LAN. If you do not blank the password here, you will have to edit all the scripts before NEMS will be able to communicate with this computer.
- Enable all modules and change the NRPE mode to Legacy. NEMS uses Nagios 3.5.1 at present, and I suppose that’s technically “Legacy”. 🙂
- Screen should look a little something like this:
- Add your Windows host to NEMS. If you are using NEMS 1.1+ you can use the template “ourwinserver” in nconf. Just change the hostname and the IP address.
Please note: If you have a software firewall running on your Windows machine, setup an exception for your NEMS server IP to gain access through ports 5666 and 12489.
Thanks for being an early-adopter! Whether you’re coming from NEMS 1.0 or its predecessor, nagiospi, I want to make it as easy as possible for you to get the latest and greatest, without having to reconfigure everything. It’s been exciting to see the NEMS project really catching on, and I endeavor to make it the best it can be. Your suggestions along the way have helped me focus on some great features for as NEMS continues to evolve.
NEMS 1.1+ has a nifty backup and export tool called NEMS Migrator. While it comes pre-packaged in 1.1+, I designed it specifically to run on legacy builds as well (NEMS 1.0 or nagiospi), giving you the opportunity to export your old configuration, deploy the latest version of NEMS, and then restore the configuration to NEMS. Easy peasy!
Here’s what you need to do:
Note: These instructions are for NEMS 1.0 or nagiospi only. Do not do this on NEMS 1.1+ as the tool is already built-in.
- SSH into your NEMS/nagiospi server.
- Become root: sudo su
- Update repository data. Type: apt-get update
- Install Git. Type: apt-get install git
- Install NEMS-Migrator in /tmp. Type:
cd /tmp && git clone https://github.com/Cat5TV/nems-migrator
- Create the backup config on your NEMS/nagiospi system. Type:
- If on NEMS 1.0: cd /tmp/nems-migrator && ./backup.sh
- If on nagiospi: cd /tmp/nems-migrator && ./nagiospi2nems.sh
- Download the backup to your computer by opening it in your web browser. In your favorite web browser, simply add /backup/ to the end of your NEMS/nagiospi server address. Eg., http://10.0.0.5/backup/
- Now that you have your backup.nems file, follow the instructions here to restore your configuration to a new version of NEMS.
The NEMS Migrator tool allows you to export/backup your NEMS configuration (backup.nems) as well as import a previous backup (through the Restore option).
The NEMS Migrator’s backup and restore options are great for keeping a safe backup without having to shutdown your NEMS server. Just download the file once in a while, or back it up automatically with your daily backup script.
NEMS Migrator is also helpful when upgrading from previous versions of NEMS, saving you having to reconfigure your NEMS deployment just to get the latest features.
I am a firm believer in redundancy, and protecting your data. What I’d like you to do is, export your migration file, then install NEMS on a new MicroSD card. Then boot from that and restore your NEMS Migrator backup. Once you’ve confirmed everything worked well, you can deprecate the old one safely. However, if something went wrong, you can contact me to fix it for you, and continue running from the old SD card in the interim.
How to Restore a NEMS Migrator Backup
Requires NEMS 1.2+
- Place your backup.nems file on a USB flash drive. You can access this directly from your web browser at http://NEMSIP/backup/ where NEMSIP is the IP address of the NEMS server you wish to backup.
- Deploy the version of NEMS you wish to restore the backup to. Please heed my Important Note above.
- Boot the new NEMS deployment and mount the USB flash drive.
- Determine the location of backup.nems in relationship to your mountpoint. For example, if you mounted the USB flash drive on /mnt/flash you may determine the location to be /mnt/flash/backup.nems
- Armed with that information, run the following command (use the full path to your backup.nems file):
1sudo nems-restore /mnt/flash/backup.nems
- Follow the prompts on screen to restore your configuration to the new NEMS deployment. If it fails for any reason, you can safely shut down and replace the SD card with your original deployment.
If you have any problems (or praise) please comment below.
The Nagios Remote Plugin Executor (NRPE) allows your Nagios Enterprise Monitoring Server to communicate with the Linux machines on your server to determine things like free disk space, CPU load, and detect possible issues that a simple ping can’t determine.
There are countless instructions online to download tar.gz files and install manually, or use a PPA to install via apt-get, but you’ll be surprised to note the needed packages are in fact already in your Debian (and by proxy, Ubuntu) repositories.
To install the needed NRPE client on Debian / Ubuntu / other Debian-based Linux operating systems:
apt-get install nagios-nrpe-server nagios-plugins
Don’t forget that you need to be root (Debian) or use sudo (Ubuntu).
Next, we just have to tell NRPE that it’s allowed to communicate with our Nagios server. On the client system, open the file /etc/nagios/nrpe.cfg
Find the line that reads: allowed_hosts=127.0.0.1
Now there are a few ways we can allow our server. First (and most obvious) is to add its IP address like this:
Where 192.168.0.5 is our Nagios/NEMS server.
Alternatively we can tell NRPE that it’s allowed to communicate with any local system:
Now, save the file and restart NRPE as follows:
service nagios-nrpe-server restart
And there we have it! Your Nagios/NEMS server should now be able to see your Linux machine.
Looking for a lightweight, affordable, easy-to-deploy enterprise monitoring server? Check out Nagios Enterprise Monitoring Server for Raspberry Pi 3!
NEMS – Nagios Enterprise Monitoring Server for Raspberry Pi
NEMS is a modern pre-configured, customized and ready-to-deploy Nagios Core image designed to run on the Raspberry Pi 3 micro computer. At its core it is a lightweight Debian Stretch deployment optimized for performance, reliability and ease of use.
NEMS is free to download, deploy, and use. Its development however is supported by its community of users. Please consider contributing if you can.
Please Note: NEMS 1.2.x is my most ambitious NEMS release yet. Please consider throwing a little gift in my Tip Jar if you find NEMS saves you time or money. Thanks!
- What Is Nagios Core?
- Why NEMS?
- About NEMS
- NEMS’ Out Of The Box Experience
- System Requirements
- Buy The Needed Hardware
- Upgrade to NEMS 1.2.1
- Upgrade to NEMS 1.2.2
NEMS 1.1 Featured on Category5 Technology TV
Nagios® Core™ is an Open Source system and network monitoring application. It watches hosts and services that you specify, alerting you when things go bad and when they get better.
Some of the many features of Nagios Core include:
- Monitoring of network services (SMTP, POP3, HTTP, NNTP, PING, etc.)
- Monitoring of host resources (processor load, disk usage, etc.)
- Simple plugin design that allows users to easily develop their own service checks
- Parallelized service checks
- Ability to define network host hierarchy using “parent” hosts, allowing detection of and distinction between hosts that are down and those that are unreachable
- Contact notifications when service or host problems occur and get resolved (via email, pager, or user-defined method)
- Ability to define event handlers to be run during service or host events for proactive problem resolution
- Automatic log file rotation
- Support for implementing redundant monitoring hosts
- Optional web interface for viewing current network status, notification and problem history, log file, etc.
Nagios doesn’t need a big fancy supercomputer to offer exceptional enterprise monitoring of network assets and resources, so our temptation is to re-purpose older servers to perform this reasonably lightweight task.
If you ask me, that’s not only overkill, but the attempt to save money by reusing older hardware will actually cost more due to higher electricity usage vs. a tiny Raspberry Pi 3 Microcomputer… which ironically may in fact have more modern system specifications than that old beast of a server you’ve been using.
It can be a wee bit daunting to setup a Raspberry Pi Nagios server from scratch, and there were no projects I found which were actively maintained at a level adequate for professional use. So I decided to start a new project–called NEMS: Nagios Enterprise Monitoring Server (for Raspberry Pi).
This project uses freely available applications such as Debian Linux, Nagios Core and a variety of other goodies, and I too release a fully ready-to-use image for you to use within yours or your customer’s network environments.
The goal with NEMS is to provide a free, full-featured, up to date drop in Nagios Enterprise Monitoring Server for Raspberry Pi 3. It has all the bells and whistles while being optimized for solid stability and fast performance.
The original NEMS release was based (with big thanks to Ryan Siegel) on NagiosPi, with many of the settings reflective of the NagiosPi Wheezy distro (which is now obsolete).
Here is what Ryan Siegel (the creator of NagiosPi) has to say about NEMS: “I’d love to upgrade NagiosPi, but i don’t have ability to make a GUI that can beat that of NEMS. I strongly feel that it has always been a necessary addition to NagiosPi and NEMS was able to deliver what is essentially an updated and improved version of NagiosPi. No reason not to start using NEMS for the time being. Nice work Robbie!” [source]
NEMS has since become a popular drop-in Nagios distro for Raspberry Pi, with a major release twice per year.
If you like NEMS, please donate: donate.category5.tv
The Out-Of-The-Box NEMS Experience:
- Raspberry Pi 3 Micro Computer
- 8GB+ Micro SD Card (Recommend 16GB+)
- Network Connection (will obtain IP address via DHCP: create your reservation in the DHCP server for ease of access)
- Internet Connection
- Purchase and assemble your Raspberry Pi 3.
- Download the most current version of NEMS below.
- Unzip and “Burn” the image to your Micro SD card.
- Boot your NEMS server.
- Login to NEMS console by either connecting a monitor and keyboard, or using ssh (Username: pi / Password: raspberry)
- Type: sudo nems-init
- After rebooting, reconnect and restore your backup.nems NEMS-Migrator file if you have one.
- Edit /etc/nagios3/resource.cfg and add your SMTP information.
- Login to NagVis and change the password (default is: admin/admin)
- Reboot your Pi and enjoy NEMS when it comes online (see below for further details).
Here’s our link to buy the device you’ll need, complete with the Micro SD card, a power adapter, a good solid case, and more: shop.category5.tv
Please buy it through that link, or let me know if you need a customized link to a different model. We get a small percentage of the sale, and it helps to make it possible to offer this as a free download.
NEMS 1.2.1 Image File – 1.27 GB ZIP File (4.68 GB Uncompressed)
Released May 22, 2017. Fixes the problems with NEMS 1.2 release.
Unzip and “burn” to 8 GB+ Raspberry Pi 3 Micro SD Card (I recommend you use at least a 16 GB card so there’s plenty of room for logs, configs, data and so-on).
BitTorrent seed coming soon.
- Upgrade from NEMS 1.1+
- Connect to your existing NEMS dashboard from your computer and press Migrator->Backup. That will give you your backup.nems file.
- Deploy NEMS 1.2 on a new card (please use a new card so you can always revert back to your existing NEMS 1.1 if you have a problem).
- Boot your Pi into NEMS 1.2 and initialize it as normal (instructions are provided via your web browser when you connect)
- Once you get to the dashboard simply click Migrator->Restore to open the documentation which walks you through restoring your settings from your backup.nems file.
- Upgrade from NEMS 1.0 or nagiospi
See legacy upgrade documentation here: http://www.baldnerd.com/nems-migrator-legacy-upgrade/
Upgrade to NEMS 1.2.2
NEMS 1.2.2 is the first ever rolling upgrade of NEMS. This means you do not need to reinstall NEMS to get the latest features. You must have NEMS 1.2.1 already installed to upgrade to NEMS 1.2.2.
- Upgrade from NEMS 1.2.1
- Type: sudo nems-upgrade
Here is a list of the changes I recorded during development.
– NEMS now requires you to run nems-init when you first deploy. This tool takes care of some of those “first boot” prerequisites like setting passwords and expanding your filesystem.
– Underlying OS upgraded to Raspbian Stretch.
– Kernel upgraded to 4.9.28.
– PHP upgraded to 7.0.19.
– Reworking of nConf to make compatible with modern software (ie. PHP7.0, mySQLi).
– Maintenance and info scripts moved to /home/pi/nems-scripts.
– NEMS MOTD upon login now shows local IP address. Also improved how it determines some of the info (see info.sh in nems-scripts) and fixes a few bugs. Also set it up to rollover to wlan0 if no response on eth0, in case the user is on wifi.
– Temporary files and Monitorix image cache moved to RAM.
– Added RPi-Monitor as per Hesh’s comment. Reworked the Memory and CPU Frequency modules to correct the accuracy.
– Added nagios-api (JSON on Port 8090) as per Timothy Seibert’s request. [License]
– Added Webmin as per Hesh’s comment. Login as pi user with the password you created when initializing NEMS with the nems-init program.
– Added support for agentless Windows checks using WMI (big thanks to Ryan Siegel).
– Changed Apache log rotation to weekly (was previously daily).
– Upgraded nagvis to 1.9b16.
– Fixed sendemail paths in nConf to ease out-of-the-box email notifications (as they should just work now). Thanks so much to Jim for pointing this typo out!
– Enabled CPU governing (package cpufrequtils). On NEMS 1.0-1.1, NEMS was locked to 600MHz, but now it will automatically go up to 1200 MHz as needed.
– force resolver to generate new DNS resolv.conf at first boot (to ensure the detected DNS servers will be used rather than our development DNS servers which may not work for you).
– NEMS Migrator upgraded to allow direct migration from nagiospi to NEMS.
– Added Monitorix 3.9.0.
– Removed MySQL, replaced with MariaDB 10.1.22.
– Improve quality of Monitorix graphs used on NEMS Dashboard slideshow.
– Minor improvements to Monitorix page based on priority of service and image quality.
– Removed some old (obsolete) kernel modules, InnoDB logs and other bloat to reduce size of stock image.
– Added /var/www/nconf/temp to tmpfs. This way if someone breaks their nCONF (eg., pressing “Back” while generating config) they can just reboot to fix it 😀
– nems-init and nems-migrator restore significantly reworked to correct initialization bugs from NEMS 1.2. Now, both initialization of a new NEMS deployment and an import from an old one should work without a hitch.
– nConf and NEMS-Migrator backups now require your password (as set with nems-init).
– NEMS-Migrator no longer replaces the MySQL database with backup. Instead, it now clears the database completely, reconciles your backup with the current set of available commands and services and then imports everything together into the fresh database and activates the hosts. This way, if you restore your NEMS 1.1 settings to NEMS 1.2.1, you don’t miss out on all the WMIC features (which your 1.1 backup would overwrite), for example.
– Documentation updated to reflect changes in commands and versioning.
– I built a quick but lovely interface for Monitorix to make it mobile responsive and a little more dynamic in its functionality.
– Distribution now available via BitTorrent (thanks to our partnership with The Category5 TV Network
– number of online users count on MOTD fixed.
– undefined constant in apache error log every 5 minutes leading to a bloated error log.
– added missing icons in check_mk.
– NEMS Migrator mail settings fixed when importing backup.nems from NEMS 1.0/nagiospi.
– Fixed MySQL Initialization Bug – was causing NEMS to lose configuration and no longer work.
– Wifi (wlan0) restored after it broken in 1.2 (due to Debian Stretch upgrade and incompatible firmwares for the Pi).
– Fixed nems-init user creation. In NEMS 1.2 it was not adding the new user to the “admin” group correctly in NEMS nCONF, so upon config generation, user would lose access to Nagios Core and other features requiring admin user.
– many miscellaneous bug fixes.
– May 25, 2017 – Monitorix cronjob now detects which network connection is being used (ie., eth0/wlan0) and begins monitoring the correct one if it changes. Requires NEMS 1.2.1 or higher.
– May 26, 2017 – NEMS Migrator updated to fix bug in host presets. Was causing these two errors: “[ERROR] Failed to get host-alive check for host ‘NEMS’. Make sure the host is linked with a host-preset. Aborting.” and “Error: Cannot open main configuration file ‘/var/www/html/nconf/temp/test/Default_collector.cfg’ for reading!” – Thanks to Rick for giving me access to his affected system so I could fix this. Requires NEMS 1.2.1 or higher.
– May 27, 2017 – NEMS web interface has been updated to git repository so I can fix issues with it on the fly without forcing users to reinstall. Requires NEMS 1.2.2 or higher.
– May 30, 2017 – Monitorix graphs clear when rebooting the Pi. This is by design. However, they were meant to regenerate upon boot. Hesh found a bug that was causing the graphs not to generate since the service was not yet loaded. I rewrote the back-end to wait for a network connection, and for the service to respond, before moving on. Requires NEMS 1.2.1 or higher.
– June 1, 2017 – MOTD was displaying the disk usage as what was being used in the home folder. I realized this was absolutely pointless information, so rewrote this section to instead tell you the current % usage of your entire SD card (/dev/root). Requires NEMS 1.2.2 or higher.
– June 6, 2017 – NEMS-Init bug fixed: new user is not authorized_for_system_information, authorized_for_all_hosts, authorized_for_configuration_information, authorized_for_system_commands … and so-on. Wrote this into nems-init. Had caused users to not have access to all screens in Nagios Core. Also added the config to the NEMS Migrator backup. I also added the patch to fixes.sh so users don’t have to re-initialize to get the fix. It will retroactively fix the file, automatically. Requires NEMS 1.2.1 or higher.
– June 7, 2017 – Check_MK Multisite reports “user not found” on some screens. Similar issue to what was happening with Nagios Core: nems-init user was not being migrated correctly to Check_MK as pointed out by Rick. This has been fixed and a retroactive patch added to correct existing deployments. Requires NEMS 1.2.1 or higher.
– June 8, 2017 – Updated nems-migrator to support NEMS 1.2.2. Built nems-upgrade, which will roll NEMS 1.2.1 up to NEMS 1.2.2 without needing to reinstall. You can do this (backup first please) by typing: sudo nems-upgrade
– June 9, 2017 – Email notification service changed to automatically detect TLS. Changes will take effect immediately for new deployments, but if yours is already deployed (before June 9, 2017) please add the following to your service definitions in NEMS-nConf for notify-host-by-email and notify-service-by-email: -o tls=auto – add it immediately after /usr/bin/sendemail so it looks like this: /usr/bin/sendemail -o tls=auto -s $USER7$ ….
– June 21, 2017 – Created nems-benchmark and nems-mailtest.
– June 23, 2017 – Disabled swapfile.
Configuring Your NEMS Server:
Browse to: http://NEMS.local/nconf (or http://IPADDRESS/nconf)
Make changes to the config as needed for your environment (eg., add a host, service, edit a setting)
I’ve added a few sample configurations in there (v1.1+) to help you get started.
Click: “Generate Nagios Config”
If there are no errors, you will have the option to Deploy the new Nagios Config.
Using Your NEMS Server:
Browse to: http://NEMS.local (or http://IPADDRESS)
The navigation should be fairly straightforward. To add hosts or configure your environment, use the built in nConf feature. To access reports, choose one of the reporting options under Reporting. To access info specific to your Pi (eg., memory usage) you’ll find those under System.
When you initialize NEMS, you will provide a password for the NEMS web interfaces. This username/password will be what you use to access most NEMS features (eg., nCONF, Nagios Core, Check_MK) however, Nagvis still uses admin:admin. Please be sure to change that when you first login to Nagvis. To login to Webmin, use the username pi and the password you entered for the pi user during nems-init.
- Nagios Core help “?” links give 404 according to https://forum.category5.tv/thread-63.html – resolve and roll out a fix in NEMS 1.2.3.
- Webmin administration of Network settings don’t stick, as per kevinds.
- PNP4Nagios is installed, but not configured. You may encounter errors, which will be addressed in a future release.
- If you encounter an issue, please report it in the comments below so I can work on a fix (or please post your fix to help others and possibly to have it integrated into the next build).
This list details features which are planned for future releases of NEMS. NEMS is released on a 6 month schedule, with a major release every May and November. Thanks to the NEMS Migrator, upgrading is a cinch.
– Further improve the mobile responsive layout for tablets and smartphones (It’s really rudimentary at the moment due to the nature of Nagios’ interface… frames? C’mon now.)
– Upgrade Nagios to Nagios Core 4. (?) – Unlikely since most of the cool things about NEMS rely on Nagios Core 3.5.1 … at least for now.
– Further streamline the implementation of Check_MK. Deploy check-mk-server.
– Expand the documentation to include setting up SMTP and mail alerts, including a new video demonstration on Category5 Technology TV.
– Create an OVA of NEMS for deployment on existing virtual infrastructures (just for the challenge/fun of it).
– Build a graphical interface for nems-init.
– Build a graphical interface for NEMS-Migrator’s “Restore” feature.
– Make it so nems-benchmark runs a nightly test and saves to a log file. Include this backup in backup.nems for support purposes.
– Create symlinks to nems-benchmark and nems-mailtest.
– I’m open to suggestions! Please post your comments at http://baldnerd.com/nems
Things I Will Not Do:
This is a list of features I will not implement.
– Raspberry Pi Zero, 1 & 2 support. Buy a Raspberry Pi 3. The higher performance of a Pi 3 is a major asset for a project like NEMS, so I will not be focusing any of my efforts on supporting legacy hardware.
– GUI/Desktop. NEMS is a Linux server. If you require a GUI/Desktop Environment, you’re doing it wrong. I have built a very nice, easy to use web interface, and have worked hard to provide good documentation to ease the complication of configuration for novice users. NEMS itself should not even have a screen connected to it. Just power and Ethernet. Everything happens either through your browser, or for advanced users, an SSH connection.
Who Creates NEMS:
Robbie Ferguson is the host of Category5 Technology TV. He’s the kind of guy who when he figures stuff out, he likes to share it with others. That’s part of what makes his show so popular, but also what makes NEMS possible.
Support What We Do:
This project is a part of something much bigger than itself, and we’re all volunteers. Please see our Patreon page for information about our network.
– Please support us by simply purchasing your Raspberry Pi at https://cat5.tv/pi
– We have some support links on the NEMS menu, such as buying from Amazon using our partner link. Please use these every time you use those stores. A small percentage of your purchase will go toward our projects.
– Your donations are VERY MUCH appreciated – https://donate.category5.tv – Please consider how many hours (and hours) of work this project has saved you, and how much you’ll save on hardware and even electrical costs as you consider contributing
– Our network also has a Patreon page – Please consider becoming a patron – https://patreon.com/Category5