NTP on Debian reporting 95 years in the future – Part 2: The Time Traveler

If you haven’t read part 1 yet, make sure you start there.

This issue has really intrigued me.

Setting the date manually fails:

root@nems:/var/log# date
Mon Aug 28 06:39:22 EDT 2113
root@nems:/var/log# date -s "2018-07-05"
date: cannot set date: Invalid argument
Thu Jul  5 00:00:00 EDT 2018

Invalid argument? Maybe it wants me to set the time too?

root@nems:/var/log# date -s "2018-07-05 20:13:01"
date: cannot set date: Invalid argument
Thu Jul  5 20:13:01 EDT 2018

Nope, that made no difference.

Well, what does my hardware clock say (since the Pine A64+ has one)?

root@nems:/var/log# hwclock
2018-07-05 20:13:10.747892-0400

Oh yay, that looks better! Let’s use that! Obviously the system knows the date and time…

root@nems:/var/log# date -s "$(hwclock)"
date: cannot set date: Invalid argument
Thu Jul  5 20:14:20 EDT 2018

Oh, COME ON!

Maybe I’ll try the long-form command…

root@nems:/var/log# date --set="$(hwclock)"
date: cannot set date: Invalid argument
Thu Jul  5 20:15:49 EDT 2018

Nope. Same result. Ach!

This is looking a lot like an old kernel bug I recall from the late 2000’s. Better check what kernel I’m running…

root@nems:/var/log# uname -a
Linux nems 4.14.26 #1 SMP Sun Mar 11 16:34:42 UTC 2018 aarch64 GNU/Linux

If I had hair…

Just to be sure, let’s reconfigure my timezone config:

root@nems:/var/log# dpkg-reconfigure tzdata

Current default time zone: 'America/New_York'
Local time is now:      Mon Aug 28 06:47:54 EDT 2113.
Universal Time is now:  Mon Aug 28 10:47:54 UTC 2113.

root@nems:/var/log# hwclock
2018-07-05 20:19:51.275648-0400

Okay, so let me get this straight… it’s August 28, 2113. But it’s July 5, 2018 according to the RTC.

Think, Robbie, think.

I did a quick grep through the /var/log folder for anything talking about ntp, and interestingly, I find this at the top of dpkg.log:

2018-07-04 08:59:23 startup archives unpack
2018-07-04 08:59:24 install ntpstat:arm64 <none> 0.0.0.1-1+b1
2018-07-04 08:59:24 status half-installed ntpstat:arm64 0.0.0.1-1+b1
2018-07-04 08:59:24 status unpacked ntpstat:arm64 0.0.0.1-1+b1
2018-07-04 08:59:24 status unpacked ntpstat:arm64 0.0.0.1-1+b1
2018-07-04 08:59:24 startup packages configure
2018-07-04 08:59:24 configure ntpstat:arm64 0.0.0.1-1+b1 <none>
2018-07-04 08:59:24 status unpacked ntpstat:arm64 0.0.0.1-1+b1
2018-07-04 08:59:24 status half-configured ntpstat:arm64 0.0.0.1-1+b1
2018-07-04 08:59:24 status installed ntpstat:arm64 0.0.0.1-1+b1
2113-08-27 19:31:23 startup archives unpack
2113-08-27 19:31:23 install ntpdate:arm64 <none> 1:4.2.8p10+dfsg-3+deb9u2
2113-08-27 19:31:23 status half-installed ntpdate:arm64 1:4.2.8p10+dfsg-3+deb9u2
2113-08-27 19:31:24 status unpacked ntpdate:arm64 1:4.2.8p10+dfsg-3+deb9u2
2113-08-27 19:31:24 status unpacked ntpdate:arm64 1:4.2.8p10+dfsg-3+deb9u2
2113-08-27 19:31:24 startup packages configure

So on first boot, the system had the date correct: July 4, 2018. The time is off by a couple hours however (it was perhaps 6am when I flashed and fired up the system and then left for work).

What’s interesting here is that the typical ntp startup tasks unpack, install and run, but then after the package is installed (a presumably automated process since I didn’t do it!) the date suddenly changes to August 27, 2113.

Being a Raspberry Pi user all my SBC life, I’m honestly impressed that the Pine A64+ has a built-in RTC… but nowhere in the specs do I see that it also includes a corresponding flux capacitor, so I must presume the jump through time is more likely a glitch in the matrix.

I’m afraid to reboot.

What’s Next? Read Part 3:

NTP on Debian reporting 95 years in the future – Part 3: Community

Bad SD cards in a NEMS Server Raspberry Pi

With the coming release of NEMS 1.3, the retroactive stats system was implemented to allow me a way to see how many systems have been upgraded, and if any are having trouble.

Now, behind the scenes, I also developed the anonymous stats system as a way for me to gauge system load on various platforms.

Why would I do that? Because NEMS is branching out onto other SBC’s, starting with the ODROID XU4.

The fact remains, I often get the question “How many hosts can I add to NEMS” and the truth is, I have no idea! I’ve received great feedback from users who are monitoring 100+ hosts on a Raspberry Pi NEMS server, but otherwise I don’t really know much about what’s actually going on in the real world of NEMS.

So the stats system shows me, anonymously, how NEMS is being used, and how it is performing.

So, that system has been running for a few days, and today I began seeing some alerts: some users have bad SD cards!

This is causing their NEMS server to run very, very slowly.

If your NEMS server is running slowly, you may be one of these! Remember, the stats are anonymous, so I can’t contact you.

I’ve considered making the stats system able to send back a response to your NEMS server… that way, even though it’s anonymous and I don’t know who you are, I can have the system programmatically warn you that your card is failing.

It’s really quite cool for me to see how NEMS is being used. So far more than 100 NEMS servers have reported in with their current statistics, and many are running more than 100 nagios monitors with a 0.01 load average.

So back to my comment about branching out to other SBCs and knowing how many hosts we can handle on NEMS: the fact is, now that I will have real world statistical data, I will be able to put together proper information about how many services can be running on NEMS on various platforms! It’ll start with the Pi 3 obviously, but I’ll be able to see for example, how a NEMS server with 500 hosts has a higher load than one with 100, so we can therefore recommend if you have 500 hosts or more, you should install on an ODROID XU4 (for example). I’ll also be creating new graphs that show comparisons of various platforms, performance graphs over time, averages based on user deployments, and so-on. Really exciting stuff!

Thanks for using NEMS! If you have any questions, I’m here for you!

Robbie

Why support may ask for your backup.nems file, and why you should never share it with others

During support sessions I often request either SSH access or a copy of the user’s backup.nems file. SSH access, it should be obvious, should not be shared with just anyone. Also, you should never, ever, ever, open SSH to the world on your NEMS server if you have not yet initialized it. This is because there are botnets that look for Raspberry Pi computers which use the default “raspberry” password, and then compromize them. Continue reading

Setting up NRPE on Windows for NEMS

Please Note: As of NEMS 1.2 NSClient++ is optional for monitoring of Windows computers (thanks to the addition of WMIC). If you’d like to use it, please follow the directions below.

  1. Grab the latest Windows client at https://www.nsclient.org/download/
  2. Install the client with the following settings:
    • Select to install the “Generic mode” NSClient++.
    • Choose “Complete installation” and if asked, choose to save config to ini file.
    • Under “Allowed Hosts” it should read 127.0.0.1,NEMSIP (where NEMSIP is the IP address of your NEMS server)
    • Clear the Password field for ease of deployment. NEMS sample scripts are setup to use NRPE without a password because I’m making the assumption that this is being deployed in a trusted LAN. If you do not blank the password here, you will have to edit all the scripts before NEMS will be able to communicate with this computer.
    • Enable all modules and change the NRPE mode to Legacy. NEMS uses Nagios 3.5.1 at present, and I suppose that’s technically “Legacy”. 🙂
    • Screen should look a little something like this:
      nsclient-setup
  3. Add your Windows host to NEMS. If you are using NEMS 1.1+ you can use the template “ourwinserver” in nconf. Just change the hostname and the IP address.

Please note: If you have a software firewall running on your Windows machine, setup an exception for your NEMS server IP to gain access through ports 5666 and 12489.

NEMS Linux – Nagios Enterprise Monitoring Server for Raspberry Pi 3

NEMS Linux – Nagios Enterprise Monitoring Server for Raspberry Pi

Important Note: NEMS started as a small project here on my blog, but since has grown into a full-fledged distro! The blog therefore is here for historical purposes, but for the most current information, please visit the NEMS Linux web site: nemslinux.com


NEMS is a modern pre-configured, customized and ready-to-deploy Nagios Core image designed to run on the Raspberry Pi 3 micro computer. At its core it is a lightweight Debian Stretch deployment optimized for performance, reliability and ease of use.

NEMS is free to download, deploy, and use. Its development however is supported by its community of users. Please consider contributing if you can.

Please Note: NEMS is a very ambitious project, and I’m just one guy. Please consider throwing a little gift in my Tip Jar if you find NEMS saves you time or money. Thanks!

Support
[NEMS Documentation]
[NEMS Community Forum]
[NEMS User Comments]

Index

NEMS 1.1 Featured on Category5 Technology TV

 

If you like NEMS, please donate: donate.category5.tv

The Out-Of-The-Box NEMS Experience:

Buy The Needed Hardware

Raspberry Pi 3 Nagios ServerRaspberry Pi 3 are very affordable, and using our Micro SD image, you simply buy the device, “burn” the image to the Micro SD card, and boot it up.

Here’s our link to buy the device you’ll need, complete with the Micro SD card, a power adapter, a good solid case, and more: shop.category5.tv

Please buy it through that link, or let me know if you need a customized link to a different model. We get a small percentage of the sale, and it helps to make it possible to offer this as a free download.

Who Creates NEMS:
Robbie Ferguson is the host of Category5 Technology TV. He’s the kind of guy who when he figures stuff out, he likes to share it with others. That’s part of what makes his show so popular, but also what makes NEMS possible.

Support What I Do:
This project is a part of something much bigger than itself, and we’re all volunteers. Please see our Patreon page for information about our network.
– Please support us by simply purchasing your Raspberry Pi at https://cat5.tv/pi
– We have some support links on the NEMS menu, such as buying from Amazon using our partner link. Please use these every time you use those stores. A small percentage of your purchase will go toward our projects.
– Your donations are VERY MUCH appreciated – https://donate.category5.tv – Please consider how many hours (and hours) of work this project has saved you, and how much you’ll save on hardware and even electrical costs as you consider contributing
– Our network also has a Patreon page – Please consider becoming a patron – https://patreon.com/Category5