Studio D: Day 12

It’s nice to be refreshed! I took a week off and just baked in the sun, and spent time with my wife and kids. It was well needed, and now I feel like I could build 11 Studio D’s!

So, here’s the story… one of our viewers from the GTA contacted me about our security system needs, and on Day 12, he made the trip up the highway to visit Studio D and start laying the wiring needed to keep the studio protected from bad guys and robots.

My secret motivation also surrounded the fact that I didn’t want to be doing electrical work without someone spotting me.

This viewer’s name is Anthony, and on today’s Backstage Pass video, you’ll get the introductions.

As you know, the Teespring campaign ended, and my sweatshirt just arrived. It’s really awesome! So you’ll get to see that today too! And if you want one, well, head over to teespring.com/category5 and tell them you want them to re-open the campaign by clicking the “WAIT! I still want one” button. All proceeds go to getting us a brand new camera to make my shiny head even shinier, and give you the bestest show ever.

Check out today’s video!

Thanks for your continued support of our Studio D project! It’s happening, folks!!

-Robbie

Studio D: Day 11

The day after Paulo visited, I continued working on the wiring while a friend joined me to help clean the tiles.

The next few weeks will be very busy with all we need to do at Studio D, so I took some downtime last week to prevent burnout.

Thanks for your continued support! More to come later this week.

-Robbie

Mobile Site 4.0 … Coming Soon

A new mobile site rewrite for Category5 Technology TV will soon be gracing your smartphone and tablet!

I received a note from one of our viewers who recently found us on Roku, but wanted to take Category5 on the go with him…

Well, that’s kind of annoying for BlackBerry users, isn’t it?

I had a good close look at the mobile site back-end, and man-oh-man, it’s getting old.

I think as a developer, it’s hard not to want to go back and rewrite everything you’ve created about once per year. That’s how fast the technologies change it seems, and the way I was coding a year ago is much different from the way I code now.

The current (old) mobile site uses flat-file delivery powered by a database backend. Your mobile device must preload all textual resources, and then it loads the images and videos when you navigate around. But the architecture forced me to hold back some features at the time.

The episode list on our mobile site version 3.1.

The episode list on our mobile site version 3.1 contains the most recent 53 episodes.

For example, the mobile site (versions 1-3) have always been restricted to the past year’s worth of videos. That’s all that was practical since your device would have to preload all the text for each episode.  Loading all 7 seasons was not possible because it would just take way too long to load, and it’s already much too slow for my liking.

I also had to restrict the amount of images and overall weight of the mobile site since many assets were preloaded at launch. Preloading 53 videos is horrible for performance, and this is not a well optimized way to do things.

So, all that said, Tony’s message and my review of the code drove me to want to redesign the mobile site infrastructure and front-end from the ground up.

There are a few things that come into play, but the big one is that I recently developed the entire infrastructure to bring Category5 to the Roku platform. So I am able to tap into new, optimized database functions that didn’t exist even 3 months ago. Features such as separation of episodes by season, and having access to our entire library of videos, including our “Special Features” category which was introduced specifically for Roku, but now can be migrated to other platforms such as our mobile site.

Out the gate though, a rewrite means implementing greatly improved programming techniques such as AJAX data loading on the mobile platform… so I can rewrite the code so each episode’s text only loads after you click on it. I’ll also stop all these videos from preloading. All this means is that we can offer a million videos if we want, and the site will still load just as quickly as if there were only 5 videos. The ability to load data through AJAX has existed for a long time, but my skill-set has grown with regards to its implementation, so I am now capable of much more than I was a year ago.

The greatly improved episode list on version 4β of our mobile site.

The greatly improved episode list on version 4β of our mobile site features all available episodes (345 at the time of writing).

The biggest and most exciting thing to me about this change in programming style is that I can include all episodes in the mobile site.  Not just the past year’s worth. It also opens us up to eventually including other features, such as the aforementioned “Special Features”.

Do more with less. That’s what version 4 represents.

So keep your eyes on our mobile site. I’ll announce it when it’s launched, and hopefully it’ll even work on BlackBerry.  :)

-Robbie

Coming features on Category5.TV

Category5.TV:

  1. Next/Previous episode buttons when viewing the show notes page for any episode. – Done
  2. Photo gallery re-write with grid-style gallery layouts.
  3. RSS feeds to begin stating the age of an episode (eg., ’1 Week Ago’ or ’3 Years Ago’). – Done
  4. Add Google-indexable breadcrumbs to Episode pages. – Done
  5. Season 2 Episodes made available on-demand and in RSS feeds.
  6. Season 1 Episodes made available on-demand and in RSS feeds.
  7. Replace the main menu script with something more robust, to accommodate the growing list of menu items and introduce third-level navigation. – Done, sortof. Rather than replacing the menu, I recoded bits of the existing one to add new functionality. I think it works quite nicely.
  8. Replace the jQuery scroller at the top of the home page with something a little more current. – Done, sortof. I decided that for the moment, I still like the scroller we have, and it’s lightweight, which is a plus. So instead I just improved it slightly. Buttons have been changed to our orange color to make them stand out more, too. I have begun coding a new “animated” version to replace this one eventually, but for now I’ve bought some time.
  9. Make the giant header slideshow on season pages select a handful of random images from the season rather than the most recent ones.
  10. Related Episodes feature on show notes pages.

Now you can see what was said in the chat room in realtime while watching on-demand video from Category5 TV.

Beginning with Episode 231 (February 21, 2012) we have been keeping a log of the Category5 TV chat room during the live broadcast, complete with timestamps.

These logs have been available to view for a while now via the show notes page for each episode since that time, but I’ve always wanted to take it a step further.

Effective immediately, logged-in registered viewers may enable the chat logs as their source for closed captioning.

Video Captions from Chat Logs

This is a two-fold success.  First, it adds a pretty neat feature to the on-demand video since logged-in viewers can now see what was said at the time during the live show, overlayed on the video, but secondly, it allowed me to generate an effective closed captioning system on our web site. I hope to eventually offer English (and electronic translations to other languages) captions, and this provides the back-end technology to do this.

Enable CaptionsTo enable this feature on your account, login to Category5.TV and visit your profile page. Once their select “Chat Logs” as your source for closed captioning.

Make sure you’re logged in, as this is an exclusive feature to registered viewers.

Please offer your feedback below. What do you think of this feature?

-Robbie

RSS distribution changes and keeping costs manageable

Hi all!

Hope you’ve been enjoying the benefits of the new transcoders the past couple months, along with the absurdly fast distribution backend we added to the RSS feeds!

I’ve been feeling the waters now for a little over 2 months and am implementing some changes, effective immediately.

Our Goal: Every Episode for Free
One of the goals at Category5 TV is to always offer all our episodes free of charge to the viewers, and in a very accessible way (or many ways to accommodate many different types of users).

The Problem With Our Goal
Our new distribution mechanism is pricey. Category5 TV stores over a terabyte of video “in the cloud”, and we have a very active community of happy downloaders. A lot of those downloaders like to grab every episode, especially people who have just found the show and want to see all the oldies. For every person who downloads the whole series, we pay for that download, and it has become substantial. Too substantial to maintain.

So, I’ve been contemplating… how do we do it?  Do we start charging for downloads?  Obviously not!  That’s not in line with our goal to offer the video to you free of charge.  Do we restrict viewers to only being able to download say, the last 5 episodes?  Also a resounding “no!” as that would not be in line with our goal to give you the wealth of information that is available through Category5 TV.

And then it hit me.

It’s not the week-to-week downloads that cost us too much money in bandwidth fees… it’s the big downloads. The people who subscribe to the RSS feed (especially HD) and download all the available episodes. That’s about 90GB per user. And there are a lot of them.

So, my solution: effective immediately, the most recent 5 episodes of Category5 TV will be distributed through our fast, expensive CDN.  This ensures the viewers who are watching week after week get the fastest, most reliable delivery service, even if they miss a few weeks and need to catch up.

All episodes from more than 5 weeks ago will still be available, but will be hosted on our slower network (which costs us substantially less, but is quite slow for distribution, and a little less reliable). This means those wanting to download all episodes will get them much more slowly, but they’ll get them none the less. This will save us several thousand dollars per year in bandwidth fees, which I think you can appreciate.

Potential Issue & Solution
Many users have their RSS aggregator setup to automatically download new files as they hit the RSS feed. Every week as a new episode is added, the 6th episode in the feed gets moved to the other distribution servers. This means that aggregator may download both the new episode (1st in the list) plus the newly moved episode (6th in the list), even though it has probably already been downloaded. The new URL makes the aggregator think it’s a new file.

The solution? Only subscribe to the most recent files. Update your feed URL, adding /fast to the end of the URL.  For example, if you’re using the SD feed (http://rss.cat5.tv/sd), change your feed to http://rss.cat5.tv/sd/fast.  This will make sure you only receive files off the fast CDN, and your aggregator will completely ignore anything moved onto the other distribution platform. You’ll receive the new episode every week, and the last of the most recent 5 episodes will simply fall off your list.

Premium Accounts
I am toying with the idea of offering optional premium accounts for a reasonable fee, which would allow a user to download the whole RSS feed directly from our super-fast CDN, and am even giving some thought to providing the series on Blu-Ray if there is a desire from the community to have such a thing. What are your thoughts on premium accounts, keeping in mind that we will always offer Category5 freely to the community? But those who offset the high expense of distribution get some added perks.

It costs us about $0.25 for each download of an HD episode of Category5. Would you pay $1 per month to offset that cost? $2 a month to more than pay for usage? What are your thoughts?

Slowing down the transcoders and Season 4 is online!

As you may have noticed, I had to slow down the transcoders.

Yes, they operate super fast and can transcode all our back episodes in a very short time as previously reported.

However… they cost a lot of money! Le gasp!

As you can imagine, Category5 TV back episodes take up a lot of space on the web. Over a terabyte as a matter of fact. So transcoding, uploading and distributing all 300+ episodes all in one fell swoop was a bad idea from a cost perspective.

So I slowed things down to a more manageable price point so I can space the cost out over multiple months.

Okay, all that said, exciting news this weekend: Category5 TV SEASON 4 is now entirely transcoded and available for on-demand viewing at http://www.category5.tv/episodes-season-4.php!

It is my dream that one day all 7 seasons (and beyond) of Category5 TV will be available for on-demand viewing, and this brings us just a little bit closer to this goal.

Many of the episode content will be dated (as we’re talking episodes which are 3 or 4 years old), but there is still some excellent information in there, such as understanding how to safely dispose of a computer to avoid your private data being compromized, enhancing photographs with the GIMP, how to create panoramic photographs using a standard digital camera, creating mosaic images from hundreds or thousands of photographs, and even our popular web development series in its entirety.

Enjoy these “new” old episodes! Now, to begin tackling Season 3!

-Robbie

Preventing rsync from doubling–or even tripling–your S3 fees.

Using rsync to upload files to Amazon S3 over s3fs?  You might be paying double–or even triple–the S3 fees.

I was observing the file upload progress on the transcoder server this morning, curious how it was moving along, and I noticed something: the currently uploading file had an odd name.

My file, CAT5TV-265-Writing-Without-Distractions-With-Free-Software-HD.m4v was being uploaded as .CAT5TV-265-Writing-Without-Distractions-With-Free-Software-HD.m4v.f100q3.

I use rsync to upload the files to the S3 folder over S3FS on Debian, because it offers good bandwidth control.  I can restrict how much of our upstream bandwidth is dedicated to the upload and prevent it from slowing down our other services.

Noticing the filename this morning, and understanding the way rsync works, I know the random filename gets renamed the instant the upload is complete.

In a normal disk-to-disk operation, or when rsync’ing over something such as SSH, that’s fine, because a mv this that doesn’t use any resources, and certainly doesn’t cost anything: it’s a simple rename operation. So why did my antennae go up this morning? Because I also know how S3FS works.

A rename operation over S3FS means the file is first downloaded to a file in /tmp, renamed, and then re-uploaded.  So what rsync is effectively doing is:

  1. Uploading the file to S3 with a random filename, with bandwidth restrictions.
  2. Downloading the file to /tmp with no bandwidth restrictions.
  3. Renaming the /tmp file.
  4. Re-uploading the file to S3 with no bandwidth restrictions.
  5. Deleting the temp files.

Fortunately, this is 2013 and not 2002.  The developers of rsync realized at some point that direct uploading may be desired in some cases.  I don’t think they had S3FS in mind, but it certainly fits the bill.

The option is –inplace.

Here is what the manpage says about –inplace:

This option changes how rsync transfers a file when its data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the update data directly  to  the destination file.

It’s that simple!  Adding –inplace to your rsync command will cut your Amazon S3 transfer fees by as much as 2/3 for future rsync transactions!

I’m glad I caught this before the transcoders transferred all 314 episodes of Category5 Technology TV to S3.  I just saved us a boatload of cash.

Happy coding!

- Robbie

The new transcoder is proving itself.

Well, we’ve been on the new transcoders for one week now, and I’m excited to see the impact.

Last night was the first night where I was able to initiate an automated transcode of an episode shortly after we signed off the air.

There are still some things I need to work out.  For example, I could not initiate the conversion until I had imported the photos, because the transcoder uses the episode’s image for the ID3 cover art on MP3 transcodes.  So after I finished choosing and uploading the images for last night’s show, I fired the transcoder.

Here’s the log output:

It was less than 8 minutes after I initiated the transcoder that the MP3 RSS feeds received the new episode.  Just a little more than 48 minutes after initiating the transcoder, the Low Definition (LD) file completed.  The show went up on the web site almost immediately after that (the files first get sync’d to our CDN and then added to the database, automatically).

All files (MP3, LD, SD, HD and WEBM) were complete in just 2 hours 31 minutes 4 seconds, including all distribution, even cross-uploading to Blip.TV (also automated now).

From 17 hours to only 2.5 hours.  This thing is incredible.

And that means, on average, we’ll be able to transcode nearly 10 episodes per day — almost double the turnaround of our first week.  That means the job which was estimated to take 72 days on our main server alone has been cut to only a day or two longer than one month.  In just one month from now, all back episodes – six years worth of Category5 TV – will be transcoded.