NTP on Debian reporting 95 years in the future – Part 5: Patching the Kernel

If you haven’t read part 1 yet, make sure you start there.

Okay — I can’t wait 24 hours — I’m an eager beaver. But 14 hours later, the system is still showing the correct time. Let’s play.

Okay – so I’m on the absolute most amazingly modern kernel ever. Great! So let’s fix that kernel.

We’ll check again to see if the timer workaround is implemented in our kernel:

So what does this tell us?

CONFIG_FSL_ERRATUM_A008585 shows whether the workaround for Freescale/NXP Erratum A-008585 is active. The workaround’s description is “This option enables a workaround for Freescale/NXP Erratum A-008585 (“ARM generic timer may contain an erroneous value”). The workaround will only be active if the fsl,erratum-a008585 property is found in the timer node.”

For those who are even nerdier than me, check out kernel.org’s log of the patch here: https://patchwork.kernel.org/patch/9487241/

You’re such a nerd!

Now I’m getting into the experimental. I’ve never changed a kernel config before. Funny, that. I guess if you’ve never needed to do something, you never learn to do it.

I have yet to find online instructions for doing this, and here’s what I’ll try.

Note for tl;dr – this didn’t work.

Alright, let’s change my default config as we prepare to re-compile the kernel. I’m unsure if changing this config will make the newly compiled kernel receive the change or not, but it’s worth trying.

Great, my config is now set to include this patch.

Okay, I’m obviously gonna need the kernel headers here…

Looks good, and the default setting is the same as my running kernel config:

Great! I think I’ve made a connection… this file has the same setting as my default kernel config.

So since I’ve already updated my own config, let’s compile with oldconfig – I’m assuming that means “grab the old config”… hmmm…. makes sense to me.

Alright, is it set now?

Weird… the setting is gone! Well, let’s see what happens? Maybe the kernel will default to “yes” if the setting … let’s try.

Oh – If it’s going to ask me to say yes to a crapload of defaults, I’ma abort that!

I’ll trust the kernel devs and my config 😛

Wow, that’s a lot of output.

Now, a quick reboot and re-connect, and then check the running kernel…

Darn.

Strangely, the default config (my config) shows that my setting remains there…

So maybe I just didn’t compile it correctly.

I wish someone in the threads would have posted how to actually add this. I mean, as a teacher, I try not to make assumptions. Saying to someone “set CONFIG_FSL_ERRATUM_A008585=y in your kernel” is a big assumption. Even I, the Bald Nerd, am not quite sure how to do that, yet I’m sure the people who make the statements are. Let’s instead post the actual commands or at least point the person in the right direction.

I’ll try to figure it out in the next couple days, and will post my step-by-step. I’ve posted a cry for help in the forum thread related to the Debian image I’m using. Hopefully they can point me in the right direction.

UPDATE: Ohmigosh, it just hit me… this is using make, yet I didn’t install!

Let’s try it, just for good measure:

Grr….

Yeah, there’s no install.sh there.

I’m sure it’s something simple I’m missing since I’m not experienced at this kind of thing.

More to come!

NTP on Debian reporting 95 years in the future – Part 4: Back To The Past

If you haven’t read part 1 yet, make sure you start there.

I had a great idea after posting part 3 last night… why not try to set the interface as static so DHCP isn’t impacted? That way, I won’t have to re-flash to an earlier image. So I stuck the SD card into my SD reader and low and behold, /etc/network/interfaces is accessible! So I commented out the DHCP line and set it instead to the static IP I have reserved for it, and low and behold, I’m up and running.

And to boot, get this:

That’s right… the date is correct after booting this way.

Strangely though, there are no updates in /var/log/dpkg.log – the last entry was in 2113.

Another interesting anomaly when checking the ntp service:

Hmm, yeah… it has been running for over 95 years. Yet the date is correct on my system now.

Hmm….

So, let’s just try…

And… wait for it….

Sweet.

Okay, so let’s leave the server running for 24 hours and see if this is a DHCP-related issue.

To be sure, I have indeed checked my router to make sure it’s not serving up bogus NTP timestamps, and it’s all good.

What’s Next? Read Part 5:

NTP on Debian reporting 95 years in the future – Part 5: Patching the Kernel

SBC Distros I’m Working On

Those who follow me closely in the SBC world know what I’m up to, generally, but I thought it’d be fun / useful to list some of my main distros I am working on.

NEMS Linux
This one is pretty darned obvious. My most popular distro to date. Learn more at https://nemslinux.com/

PlexPi
Plex Media Server for the Raspberry Pi 3/3B+. I may eventually port this to more powerful SBCs as well. I think it’d do amazingly on a RockPro64. https://plexpi.com/

Retropie-based Retro Gaming Distro for ODROID XU4 + others
Yet to be named build to port Retropie properly to the XU4 and other SBCs where it is either not yet [well] supported or onerous to configure.

Camera System
Yet to be named Raspberry Pi distro to turn any supported IP or USB camera into a sophisticated webcam ideal for weather stations, resorts, hobbyists. Not intended for surveillance, though it could be used for this too. Includes an impressive deduplication system I wrote which reduces the number of stored images by matching their image data: if there is no change in the scene, the image is not saved. Also allows stream relaying from an RTSP compatible IP camera or standard USB webcam to YouTube Live (video only), interval-based snapshots with automatic upload to web, and more. My hope is that I can turn even a Raspberry Pi Zero W into a very impressive webcam system.

Turnkey Servers
I plan to release some turnkey servers for SBC such as WordPress, LAMP, XMPP and so-on. Many of these will be in competition with existing turnkey builds, however I feel my Migrator off site backup system will help my builds stand out from the pack.

Which project has you most excited?

NTP on Debian reporting 95 years in the future – Part 3: Community

If you haven’t read part 1 yet, make sure you start there.

Hooray! I am not alone, and it does indeed appear to be a Pine64-specific issue. That means I’m not crazy. Or at least, this issue doesn’t prove my craziness.

  1. https://forum.armbian.com/topic/3458-a64-datetime-clock-issue/
  2. https://forum.armbian.com/topic/7423-pine64-massive-datetime-clock-problem/

Looking at martinayotte‘s suggestions, since he seems to have been through what I’m going through right now…

Okay, so just in case they fixed it already, let’s just try…

Hmm, it seems apt doesn’t like time travelers like me. So … [le gasp]…

Wait for it….

Like trying to play my 8-track cassettes in a Blu-Ray player, apparently some services don’t play nicely 95 years in the future. DHCP is showing locally on the TTY splash screen as 127.0.0.1, and the only way to recover will be to re-flash to a point before the NTP issue struck.

What’s next? Read Part 4:

NTP on Debian reporting 95 years in the future – Part 4: Back To The Past

NTP on Debian reporting 95 years in the future – Part 2: The Time Traveler

If you haven’t read part 1 yet, make sure you start there.

This issue has really intrigued me.

Setting the date manually fails:

Invalid argument? Maybe it wants me to set the time too?

Nope, that made no difference.

Well, what does my hardware clock say (since the Pine A64+ has one)?

Oh yay, that looks better! Let’s use that! Obviously the system knows the date and time…

Oh, COME ON!

Maybe I’ll try the long-form command…

Nope. Same result. Ach!

This is looking a lot like an old kernel bug I recall from the late 2000’s. Better check what kernel I’m running…

If I had hair…

Just to be sure, let’s reconfigure my timezone config:

Okay, so let me get this straight… it’s August 28, 2113. But it’s July 5, 2018 according to the RTC.

Think, Robbie, think.

I did a quick grep through the /var/log folder for anything talking about ntp, and interestingly, I find this at the top of dpkg.log:

So on first boot, the system had the date correct: July 4, 2018. The time is off by a couple hours however (it was perhaps 6am when I flashed and fired up the system and then left for work).

What’s interesting here is that the typical ntp startup tasks unpack, install and run, but then after the package is installed (a presumably automated process since I didn’t do it!) the date suddenly changes to August 27, 2113.

Being a Raspberry Pi user all my SBC life, I’m honestly impressed that the Pine A64+ has a built-in RTC… but nowhere in the specs do I see that it also includes a corresponding flux capacitor, so I must presume the jump through time is more likely a glitch in the matrix.

I’m afraid to reboot.

What’s Next? Read Part 3:

NTP on Debian reporting 95 years in the future – Part 3: Community

NTP on Debian reporting 95 years in the future – Part 1

Here’s one for the “I’d pull out my hair if I had some” files…

I have a wee SBC (a Pine A64+) I’m porting NEMS to, and everything was sweet for a day… working fine, all looks good. So I left it running.

Next day, while the system clock shows the correct date and time, the UTC and Local time are off by 95 years!

Now, I admit, it’s nice seeing a NEMS server that’s been up for that long 😛 but it’s very curious.

Logging into NEMS Linux, I find another oddity…Apparently my last login was in 1977.

Here is a picture of how that might have looked:

I’m pretty confident that my little Pine A64+ has more power and capacity than the supercomputer shown. Chances are good it also cost a bit less.

So it’s time to start digging… where did NTP get this ridiculous 1292443255.246538 second offset from, and why? And how to correct it?

What’s next? Read Part 2:

NTP on Debian reporting 95 years in the future – Part 2: The Time Traveler

Aquilo – Sorry

The moment I heard this track on 181.fm, I had to turn it up. I love its simplicity, and how the vocal is so cutting and powerful that you don’t even realize there are basically no instruments during a large portion of the song. I’ve never heard of Aquilo, but he kind of reminds me of when The Eden Project goes falsetto. I just love this track. Beautiful.

It’s turtle time! Turtlecoin, that is.

I’ve been mining TRTL for a little more than a week now, and it’s a blast.

Turtlecoin is fast becoming recognized for its amazing community.

One of the fun ways to obtain TRTL is by participating in a raindance.

A raindance happens when anyone donates 5,000 TRTL or more to the rainbot. At that time, it starts to “rain turtles” and anyone who “catches” them gets an equal portion of the rain. For example, if someone contributes 10,000 TRTL to the raindance and 100 people participate, each participant will receive 100 TRTL.

What makes it intense is that you only have 90 seconds to send the rainbot your wallet address and then click the icon that corresponds to the instructions you get back from rainbot. 90 seconds sounds like a long time, but when the rain is falling in a typhoon, it can get crazy.

Next Friday (a week from today) is going to be the most exciting raindance I’ve ever seen. Rather than donating to rainbot, everyone is meant to set their mining software to mine directly to the rainbot’s wallet. With enough participants (they’re expecting 160 or more) the rain could fall every 90 seconds!

I will be livecasting the Turtle Typhoon on YouTube, so make sure you subscribe to https://youtube.com/category5tv to get the notification when we go live!

Find out more about Turtlecoin at turtlecoin.lol. Make sure you get your wallet address setup in advance (so you can collect rain), and when you’re done, I’d welcome you to consider donating a portion of your TRTL to supporting Category5 TV.

Our wallet address is:

Here is the event announcement. You can check for updates by visiting their Discord and typing !tag typhoon

Turtle Typhoon – Mine the Storm!

You are Invited to the Raindance Party!

**From 8pmEST/1amGMT on Friday 9th March**

**THE RAIN PARTY WILL COMMENCE POWERED BY OUR MINING COMMUNITY**

If you would like to join the raindance here’s how to get started (rain will occur randomly before the set date also): http://github.com/turtlecoin/turtlecoin/wiki/Participating-in-Raindance

If you are mining TRTL and would like to donate to the rainbot during the Turtle Typhoon simply change your wallet_address in your xmr-stak config to the raindance donation address.

The raindance donation address is:

 

Automatically Deduplicating Data on Debian Linux

Deduplication is the process by which a filesystem (or application) stores data by first comparing data blocks within the data and then only storing one copy of matching data blocks. By doing this, the files require significantly less space on the storage medium.

A good example (just for the sake of understanding) would be WordPress. Let’s say you have a web server with 10 WordPress sites. The WordPress source code in this example is 30 MB on its own. Your server will be storing 300 MB (10x 30 MB). By storing this on a deduplicating filesystem, it’ll be the original 30 MB plus a little overhead for the deduplication data… so let’s say for the sake of ease, your server will be storing just 31 MB for exactly the same data.

These are small numbers. But I recently opened an off-site backup service for NEMS Linux, and I need to be able to store daily backups for its users. Guess what? From day-to-day, a significant portion of those backups are very, very similar. Config files don’t generally change much from day-to-day, most days. So why store them in such a way that they take up 30x the space? Deduplicating is going to save me a ton of storage space.

I’ve been reading up on some deduplication options. My first go-to was btrfs, but it looks like they’re not quite ready yet, with inline deduplication residing only out of tree. I feel like when that feature is implemented in stable, btrfs will be my go-to… but for now, I need to find an alternate solution.

Lessfs is another one I peeked at, but once I noticed their “official” web site was offline, and distribution is done through Sourceforge, I moved on pretty quickly as it seems pretty obvious that either it’s a dead project or at least not a well-supported one.

Then I got looking at OpenDedup’s SDFS, which is a volume-based deduping filesystem, which sounds ideal for my use case, for now. I won’t hold the fact that it is Java-based against it just now as the functionality sounds perfect. Plus SDFS appears well-supported and professional in its presentation, which gives me hope for its future.

I’m going to add some more memory to my little server to accommodate the RAM requirements. Make sure your system has adequate RAM… SDFS likes to eat memory for breakfast. “The SDFS Filesystem itself uses about 3GB of RAM for internal processing and caching. For hash table caching and chunk storaged kernel memory is used. It is advisable to have enough memory to store the entire hashtable so that SDFS does not have to scan swap space or the file system to lookup hashes. To calculate memory requirements keep in mind that each stored chunk takes up approximately 256 MB of RAM per 1 TB of unique storage.” [Admin Guide]

If you’re not using Debian, check out their Quickstart Guide.

Installation of SDFS and its dependencies on my Debian system (would also work for any other Debian-based system like Ubuntu, as long as you are root user):

Next up, we need to increase the limit of how many files can be opened at once… again, as the root user:

Next up, I need to create the volume itself, but I want to be specific about where it is stored. In this example I will call the volume “myvolume” and I will store it in a folder called raw_volume in my home folder… this way I know not to touch it (as it is raw):

Once created, if you’d like to see the status, type:

…and you can view/edit the configuration in the file /etc/sdfs/myvolume-volume-cfg.xml where myvolume is whatever you named yours with –volume-name above.

The reason I’m specifying to store in my home folder is because it will then be part of my backup set (without having to manually add it) and also because my home folder is on a different, bigger drive than the /opt folder, which is where SDFS would default to.

You’ll also notice in the above command I’ve set the capacity to 100GB. It won’t actually take this much space on my drive right now. That is the maximum I’m allowing the volume to become. You can change that to anything you like, to suit your need. On the disk itself (in /home/robbie/raw_volume by my example) the SDFS volume will actually only take up the amount of space of the deduplicated data. If you ever need to make the volume bigger, you can do so by typing the following with the volume unmounted: sdfscli –expandvolume 512GB

Also, since this is a local filesystem, I’ve specified to use a variable block size, which could reduce the amount of space and improve the deduplication.

Now I need to create the mountpoint and mount the SDFS volume so I can start writing data to it:

Now let’s prepare the mount.sdfs command:

Scroll to the end of the file and remove “-Xmx$MEMORY$MU”, and edit “-Xms$MEMORY$MU” to instead read “-Xms1M”.

So my final command looks like this:

Then, mount it to test:

Try writing some data to the the mountpoint. If all went well, all should work and automatically dedupe. As I write data to /home/robbie/backup, it is automatically deduplicated to save space!

Next up, adding it to fstab!

If all went well, unmount it and make it so it automounts.

Despite what some people are saying online, yes, you can indeed mount sdfs filesystems using fstab! It’s a fuse-based filesystem! #facepalm

Here’s how I added it to my fstab:

All is working great, but it’ll be most interesting to see what begins happening once I exceed ~1 GB storage and deduplication starts doing its thing.

The Results

To see the difference in usage, I like simply using this command:

This will output something along the lines of this:
47K -rw-r–r– 1 root root 1.6M Feb 2 10:02 test2.txt
1.5M -rw-r–r– 1 root root 1.6M Feb 2 09:55 test1.txt

You’ll notice I’ve colorized the filesizes. The first set (indicated in orange) represents the actual usage on disk thanks to deduplication. The second set (blue) is the actual filesize.

I even noted that copying multiple copies of the same file, the “extra” copies showed a use on disk size of 0B! Yes, the impact is so small it didn’t even register! Brilliant.

Shan Vincent de Paul – Walk On Water

I heard this track on CBC Radio on my way home from work and pulled over so I could take it all in and inevitably catch the title. The tasteful autotune in the first verse caught my ear and forced my right hand to turn the dial till I found 11… and the skippy beat at 0:39 sealed the deal. The moment I got home, this went on repeat.

And if you’re digging it like I’m digging it, here’s an interview on CBC Radio that you may enjoy…