Backup a Linux machine with LVM Snapshots and rdiff-backup

Here is the completed script I wrote on Episode 461. Make sure you check out the full episode for details on how to make this work for you.

And of course, here is the episode:

Convert Minecraft 1.8+ Skin to 1.6/1.7/Minetest Skin in PHP

RobbieF's Minetest SkinAs we build up #ThePixelShadow on Category5 TV, and introduce a creative Minetest server specifically for playing Minetest (the free Minecraft alternative), it became apparent that our users/viewers would like to be able to have their own custom skins.

We’re making it easy with a nice little interface to upload your own skins, but part of the process requires making a skin which is compatible with sdzen’s/PilzAdam’s player_textures mod … basically, these skins are Minecraft 1.6/1.7 skins… 64×32. Great Minecraft skin creator sites such as minecraftskins.com now generate Minecraft 1.8 skin files, which are 64×64.

The difference is essentially that the skins now support overlays (eg., removable headphones or glasses) and your left and right arms and legs can have different textures. Not the case with 1.6/1.7/Minetest… so we must convert the skin file to make it compatible.

Since we’re building a web interface to do this all automatically for you and place your player skin on our server automatically, I’m building the program in PHP. Since there are a lot of tutorials out there that simply instruct you to change your canvas size to 64×32 (which is wrong – you will lose your overlays!) I thought I would share my method with you in case it comes in handy.

And hey, it’s a fun exercise in PHP/GD anyways  🙂

If you find a good use for it in your project, please comment below. If you really love what I do, please consider supporting my Patreon profile, or throw a little something in the tip jar.

Hope to see you on #ThePixelShadow Minetest server soon, custom skin and all!

-Robbie

Make it so mountpoint can’t be written to if not mounted.

Have you ever accidentally saved files to a Linux mountpoint when the drive wasn’t mounted, and then couldn’t mount the drive thereafter? Or worse, had a backup run when the backup drive wasn’t mounted, only to fill your filesystem and crash the server?

These problems can be avoided by simply making your mountpoint immutable! What this means is, your mountpoint (the folder itself) cannot be written to. However, even as an immutable folder, it can be mounted to, and the filesystem of the mounted drive then controls the permissions of the folders therein.

It’s a simple Linux command. We’ll pretend our mountpoint is simply /mountpoint. Here’s all you have to do:

Brilliant! And oh, so simple.

Here’s a sample of what happens when I do this as root. Note that ‘mymountpoint’ is setup for me in my /etc/fstab file so it normally auto-mounts.

Enjoy that little tidbit!

As a side note, you might want to also get a notification if your drive isn’t mounted… so you could use the mountpoint command to send you an email if there’s a problem. Just add something like this to your backup script:

That simply checks if /mountpoint is a mounted mountpoint. If yes, it does nothing. If no, it will send you an email.

-Robbie

Convert numbers to words the easy way with PHP

Sometimes we want words rather than numbers, but it used to be a very onerous task to do this. Since PHP 5.3.0 however, the NumberFormatter Class was introduced, allowing us to do this conversion quickly, with a single line of code.

A good example of the need for a this would be a business web site that says “We’ve been in business for 18 years.” To keep the site current, they’re doing echo ‘We\’ve been in business for ‘ . (date(‘Y’) – 1997) . ‘ years.’; It would look much better to say “We’ve been in business for eighteen years. This bit of code will do that for you.

Search Goblin Number to Words in PHP Screenshot

With the new Number to Words in PHP system at Search Goblin (my little helper script site), you can enter any number and the script will be demonstrated for you, converting your number to plain text. The code is provided there so you can start using this technique on your own site.

Check it out: https://searchgoblin.com/php-numbertowords/

Convert video to several JPG images on Linux without ffmpeg.

I admit… I do love PHP in the command line. Does that make me a bad person? 😉

Here’s a tiny little script that I wrote to create many JPG screenshots of a video file. I use this each week to create a bunch of stills from our broadcast so I can use them as thumbnails and so-on. I didn’t want it to depend on ffmpeg since I don’t have that on any of my modern systems.

It requires just three packages: mplayer mediainfo php-5

Save it as whatever.php and run it like this: php whatever.php file.wmv

It will create a folder called file-Screenshots/ and will save one picture per 10 seconds for any video source. Just change “file.wmv” to the name of your video. Include the path if it’s not in the current folder.

Hope it helps you out.

-Robbie

Automated cache-buster on images in PHP

I have a particular site I manage where one particular image (a grid of sponsors) gets updated quite regularly.

Rather than edit my source code each time I upload a new image, I thought I’d let PHP do the work for me.

Now, every time I upload a new image, replacing silver.jpg, it will automatically update the image in the users’ cache.

Just a silly little time saver.

Note: I wouldn’t do this on every image on a site since it means an extra hit to the filesystem. That could mean a performance drop if a site is checking the filemtime of 100 images. In my case, it’s just a single image, so it’s okay.

Unify Theme ERROR! on CAPTCHA form.

For the life of me, I couldn’t figure out why the Sky Forms CAPTCHA was showing ERROR! on my Unify Theme Bootstrap 3 deployment.

Turns out this was just a rookie mistake… I wasn’t looking closely enough at the “how it works” and I was missing some code from the demo-contacts.php sample file that was crucial to the operation of the CAPTCHA. This code generates the CAPTCHA itself and stores it in SESSION data. Since it was missing, the CAPTCHA system’s image.php was turning out an ERROR!

That $_SESSION[‘captcha_id’] is what the image.php file is looking for. If it doesn’t find it, ERROR!

Refresh, and we’re good to go!

-Robbie

Calculate the size of a VARBINARY field required for AES_ENCRYPT

Here’s a simple Open Document Format spreadsheet tool to help you calculate 16 × (trunc(string_length / 16) + 1) without having to be a mathematical genius.

Download: AES_ENCRYPT Size Calculator

You can use an ODS compatible spreadsheet program such as LibreOffice or OpenOffice.

Why you may desire to use VARBINARY() instead of VARCHAR(), see this great post from Mahmud Ahsan: http://thinkdiff.net/mysql/encrypt-mysql-data-using-aes-techniques/

Preventing rsync from doubling–or even tripling–your S3 fees.

Using rsync to upload files to Amazon S3 over s3fs?  You might be paying double–or even triple–the S3 fees.

I was observing the file upload progress on the transcoder server this morning, curious how it was moving along, and I noticed something: the currently uploading file had an odd name.

My file, CAT5TV-265-Writing-Without-Distractions-With-Free-Software-HD.m4v was being uploaded as .CAT5TV-265-Writing-Without-Distractions-With-Free-Software-HD.m4v.f100q3.

I use rsync to upload the files to the S3 folder over S3FS on Debian, because it offers good bandwidth control.  I can restrict how much of our upstream bandwidth is dedicated to the upload and prevent it from slowing down our other services.

Noticing the filename this morning, and understanding the way rsync works, I know the random filename gets renamed the instant the upload is complete.

In a normal disk-to-disk operation, or when rsync’ing over something such as SSH, that’s fine, because a mv this that doesn’t use any resources, and certainly doesn’t cost anything: it’s a simple rename operation. So why did my antennae go up this morning? Because I also know how S3FS works.

A rename operation over S3FS means the file is first downloaded to a file in /tmp, renamed, and then re-uploaded.  So what rsync is effectively doing is:

  1. Uploading the file to S3 with a random filename, with bandwidth restrictions.
  2. Downloading the file to /tmp with no bandwidth restrictions.
  3. Renaming the /tmp file.
  4. Re-uploading the file to S3 with no bandwidth restrictions.
  5. Deleting the temp files.

Fortunately, this is 2013 and not 2002.  The developers of rsync realized at some point that direct uploading may be desired in some cases.  I don’t think they had S3FS in mind, but it certainly fits the bill.

The option is –inplace.

Here is what the manpage says about —inplace:

This option changes how rsync transfers a file when its data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the update data directly  to  the destination file.

It’s that simple!  Adding –inplace to your rsync command will cut your Amazon S3 transfer fees by as much as 2/3 for future rsync transactions!

I’m glad I caught this before the transcoders transferred all 314 episodes of Category5 Technology TV to S3.  I just saved us a boatload of cash.

Happy coding!

– Robbie

Running phpcs against many domains to test PHP5 Compatibility.

Running a shared hosting service (or otherwise having a ton of web sites hosted on the same server) can pose challenges when it comes to upgrading.  What’s going to happen if you upgrade something to do with the web server, and it breaks a bunch of sites?

That’s what I ran into this week.

For security reasons, we needed to knock PHP4 off our Apache server and force all users onto PHP5.

But a quick test showed us that this broke a number of older sites (especially sites running on old code for things like OS Commerce or Joomla).

I can’t possibly scan through billions of lines of client code to see if their site will work or break, nor can I click every link and test everything after upgrading them to PHP5.

So automation takes over, and we look at PHP_CodeSniffer with the PHPCompatibility standard installed.

Making it work was a bit of a pain in the first place, and you’ll need some know-how to get it to go.  There are inconsistencies in the documentation and even some incorrect instruction on getting it running.  However, a good place to start is http://techblog.wimgodden.be…..

Running the command on a specific folder (eg. phpcs –extensions=php –standard=PHP53Compat /home/myuser/domains/mydomain.com/public_html) works great.  But as soon as you decide to try to run it through many, many domains, it craps out.  Literally just hangs.  But usually not until it’s been running for a few hours, so what a waste of time.

So I wrote a quick script to help with this issue.  It (in its existing form – feel free to mash it up to suit your needs) first generates a list of all public_html and private_html folders recursive to your /home folder.  It then runs phpcs against everything it finds, but does it one site at a time (so no hanging).

I suggest you run phpcs against one domain first to ensure that you have phpcs and the PHPCompatibility standard installed and configured correctly.  Once you’ve successfully tested it, then use this script to automate the scanning process.

You can run the script from anywhere, but it must have a tmp and results folder within the current folder.

Eg.:
mkdir /scanphp
cd /scanphp
mkdir tmp
mkdir results

And then place the PHP file in /scanphp and run it like this:
php myfile.php (or whatever you ended up calling it)

Remember, this script is to be run through a terminal session, not in a browser.

See what we’re doing there?  Easy breezy, and solves the problem when having to run phpcs against a massive number of domains.

Let me know if it helped!

– Robbie