Coding Horror: Re-Encoding Your DVDs

I bought my first DVD about 10 years ago. At the time, they were a technical marvel:

* 8.5 Gigabytes per side

* 720 x 480 MPEG-2 video at 30 frames per second

* Dolby Digital (AC-3) or Digital Theater System (DTS) digital multichannel sound

Today, those specs are rapidly becoming pedestrian in the face of high definition cable, broadcast, and Blu-Ray discs. A few of the video sharing websites offer something perilously close to DVD quality already.

I say the DVD is the new MP3. We’re going to start tossing these things around like candy.

Unlike audio CDs, DVDs are already compressed digital data. You could extract the files from the DVD as-is, and play them back to your heart’s content. No re-encoding required. But like The Six Million Dollar Man, we can rebuild them better than they were before. Video codecs have advanced tremendously since the heady days of MPEG-2. These new codecs take a lot more playback horsepower than MPEG-2, but offer comparable quality in about one-fourth the size. We can turn our digital DVDs into better digital DVDs through superior computer science.

[From Coding Horror: Re-Encoding Your DVDs]

The article is right on the money. I am going through that exact experience: I have tons of DVDs scattered all over the house, and (exercising my fair use rights) I am slowly converting them to h.264 for our two AppleTVs. Once you get used to picking your DVDs off a menu, there is no turning back, especially if you use MetaX to pull the DVD’s cover and main information off Amazon. It looks no different than browsing for movies through the iTunes Music store.

The main problems to doing this are logistics:

1. Using my particular settings for Handbrake, it takes about 1.5GB of disk space for each hour of DVD video. Thanks God 1TB external drives are affordable now.

2. Limitations of the Mac Book Pro Superdrive. It is a bit too slow to do a real time rip + encode. That means using Mac The Ripper first, to rip the movie, then Handbrake to convert it to h.264.

3. Some DVDs are using protection schemes that are not part of the DVD specification, so these may not be ripped consistently. Not the end of the world, all it means is I have to keep the DVD at hand instead of buried into a closet.

4. I have a hardware accelerator for h.264, but it doesn’t allow me to do AC3 passthroughs. This means if I want to use the encoder, I will lose 5.1. Not cool.

5. 3GB per movie is kind of large when you are trying to move it across a home network to a different machine. My piece of shit wireless G router would choke on it, same router with everything on 100MB ethernet seems to work OK. I expect this to stop being an issue once I upgrade to wireless N.

MetaX

It is obvious that my most common complaint about the iTunes and AppleTV platform is encoding issues. Still, there is a second aspect that most of us find annoying: proper tagging of media files.

iTunes simply can't deal with all of the tags that can be used in the movies that it can play:

1. You can't toggle video type.

2. You can't add a rating. You can add your own rating, but you can't mark a movie as PG-13, etc.

3. There is no way to mark a movie as multiple volumes. TV shows have seasons and episodes, movies don't have parts.

4. You can't change the artwork of movies with AC3.

I bet there is more, but those four are the most obvious. I was able to deal with #1 through an AppleScript, but until today I had no way to deal with the rest in a mechanical fashion. Today I learned of a donationware application, MetaX, that does most of that work. I just tested it with a half dozen movies and so far the results are beyond of what I expected.

Not only will it allow me to deal with these issues, but it automates them by both allowing a queue, and implementing automated lookups to both Amazon and the Internet Movie Database.

Kickass.

It even fills the description fields, ratings, etc. The only problem is that in the interest of paranoia, it rewrites your video file, then after completion it deletes the old one and adds the new one to iTunes. It makes for a slow queue, but at least if it fails you won't lose the original file.

Two weeks with the ElGato Turbo.264

It's been close to two weeks since I purchased my Turbo.264 and have managed to push about over 100 hours of video through it, equally split between TS_Video and transcodes from DivX. Almost every night for two weeks it has run nonstop doing TS_Video folders, and at least 2 hours every morning doing transcoding.

The good: it really works. Even at its busiest, it will run on real time (24fps or so for most video) while CPU utilization is very reasonable. The fastest I have seen it encode was 50fps for a WMV transcode for the iPhone (all my other encoding is for the AppleTV). Funny thing is that during the WMV transcode it could not recognize that I have a registered version of Flip4mac.

The bad: it is at the mercy of the software that you use. The application that ships with it slaughters DVDs. I have seen many instances of audio tracks our of sync. It botches the aspect ratio. It can't deal with most TV season DVDs. And no AC-3 passthrough.

Roxio Popcorn 3 is a little better. I haven't seen it screw up the aspect ratio, and again: no AC-3 passthrough. I don't understand why this is a problem since all it has to do is take the sound as is, it won't need to transcode it. It can handle DVD chapters and TV season DVDs a lot better. The in-progress status is not as good as the one with the application supplied with the Turbo.264.

After the first week or so I realized that I could still use the dongle for transcoding and leave the TS_Video jobs to run overnight with Handbrake. Handbrake on this laptop when it is idle can easily run over 30fps, which is not exactly terrible.

I still think it is a great buy, and if you don't care about the AC-3 passthrough then it is even better (for example, if you are watching AppleTV and you are using only the component cables). Just make sure you try Popcorn 3 before settling on the default application.

ElGato Turbo 264: First Impressions

There is a downside to having an AppleTV, iPhone or video capable iPod: converting videos to a compatible format is always a hassle. There are plenty of applications to handle this problem, no two of them are alike.

I have tons of DVDs, ripping them turned out to be quite the adventure. I did the first few dozen or so one by one, with Handbrake. I wouldn't know until I got an AC3 receiver that I ripped all of these DVDs wrong and will have to eventually re-rip them because the default Handbrake conversion for AppleTV converts the AC3 tracks to AAC. Ouch.

VisualHub is great, but it is slow. It doesn't have a pause button, which is really stupid: if you have a big batch of files to convert and you are using too much power, you can't pause the job, you have to kil l it, then remove from the queue whatever is good to go and then start again.

I didn't know I could use Handbrake to process files from a DVD already ripped. The time needed to rip a DVD with Mac The Ripper, then convert it for the AppleTV with Handbrake is usually less than if you let Handbrake do the whole job. More ouch.

Roxio Popcorn, for some really fucking stupid reason, won't allow you to pass-through the AC3 from a DVD, instead you are forced to convert to AAC. The interface *is* better, and it queues nicely, plus it has a proper pause feature. And it allows for hardware acceleration.

That's when I learned that ElGato has a hardware accelerator for h.264. I spent a week or so scouring the web for reviews and hate posts about it, and overall it seems to have a good reception. It is universally accepted that if your mac is not very new, the performance improvements are dramatic. But, what about recent fast macs?

After grinding through about 15 hours of video, I can tell that even in a Mac Book Pro Core 2 Duo 2.33GHz, 3GB RAM, Leopard, and while Time Machine is running, iTunes is streaming to my AppleTV over 100BaseT, and Parallels Desktop is running XP Pro over a Cisco VPN, it is running a hell of a lot faster than VisualHub.

The best part is that the resource utilization of the Turbo.264 application is very low, most of the time I can't tell it is running. The USB dongle with the hardware encoder doesn't even get warm. Right now I am seeing consistent 40fps or better encoding (the source material is 29fps, if you can't crank out at least that much, it means it will take you longer than one hour to encode one hour of video). In real life this means that I am cranking out avi/divx to h.264 conversions at about one hour of encoding for every two hours of video input. I imagine this is going to depend wildly on the source material.

So yes, even if the machine is fast, you can at your very worst expect it to use very little resources to crank out the same workload that would usually max out a multiple core mac.

AppleTV minor update

Apple pushed a minor update to AppleTV Take Two sometime yesterday. I haven't been able to find release notes, so here are the only obvious things:

1. A genre browser within the "My Movies" section. It is very slow when you move between genres.

2. The HDMI switching is still a huge pain. I thought it was because I have a really shitty cheap HDTV, but I am hearing complaints about it from people with really nice/expensive screens.

That's about it. Some people in forums are going all crazy about it, but it is mostly a halo effect. The one real constant is that the genre browser is slow. I don't think anything else in the AppleTV is affected.

Test Shows Comcast’s HD Squeeze In Virginia - 3/27/2008 3:12:00 PM - Multichannel News

Comcast has said it can pack three high-definition signals into space typically used for two—without viewers noticing a drop in quality. But at least one viewer has.

A member of AVS Forum, a community site for audio/video enthusiasts, has posted the results of his comparison of the compression rates for 10 HD channels offered by Comcast and Verizon Communications’ FiOS TV in Northern Virginia.

The user, “bfdtv,” said his test showed Comcast is delivering certain MPEG-2 HD channels at bit rates as much as 28% lower than Verizon, resulting in lower-quality pictures.

[From Test Shows Comcast’s HD Squeeze In Virginia - 3/27/2008 3:12:00 PM - Multichannel News]

I can say from first hand experience that most HD channels in Comcast for Reston, VA don't look as good as one would expect. The problem is that sometimes the content is not exactly HD, instead it is up-converted, and many times this is done poorly. A&E is sometimes shown simply stretched to fit the screen.

There are always a couple of channels that look really nice most of the time. One of the local PBS affiliates always looks fantastic, and the network channels look fine as long as they are running proper content, like for example Heroes, House, Bones, etc.

TiVo, YouTube to deliver videos to TVs - Internet- msnbc.com

SAN FRANCISCO - TiVo users will be able to watch YouTube clips on their televisions by year's end, TiVo Inc. said Wednesday.

That's the latest move by YouTube to reach beyond the Web's regular boundaries. Fans of the popular online video provider can view its videos on their cell phones and Apple TV, a box that streams movies from people's computers to their TV.

[From TiVo, YouTube to deliver videos to TVs - Internet- msnbc.com]

When I setup our AppleTV's I thought that the Youtube feature would just sit there unused. Next thing I knew, PJ was spending as much time browsing Youtube from his AppleTV than from his iMac (which is great, it means less bickering between PJ and Ivette about who gets to use the iMac). I also like it a lot, after spending just a half hour browsing videos on a 37" HDTV, you don't want to go back to your computer.

Of course, the feature needs work, but it is most an issue with searching and browsing. The videos look pretty damn nice.

The Faraday Cage

A real Faraday Cage is an enclosure that blocks electromagnetic radiation. Over the past few years I have been struggling with my condo because sometimes it feels like I am living inside of a Faraday Cage: my cell reception always sucks and I never get enough signal strength from my wireless access point.

Once we got the two AppleTVs, it got worse. Two AppleTVs streaming off two separate Macs, both on 802.11g is too much of a pain in the ass, and this is assuming the network is running normally. Mine wasn't, so performance for PJs AppleTV was always subpar when used in streaming mode. Because of that, his is setup to pull the actual content instead of streaming it.

Here's more or less what the network was like:

Red: 100MB ethernet.

Blue: 54MB wireless.

Green: Mac / AppleTV pair

I decided to hell with it, why bother with wireless when the condo is just 1000 square feet? I asked my friends around, and they all recommended the same: wire it yourself.

One of my coworkers lent me his crimping tool and his line testing gizmo, plus a bag of RJ45 connectors. I spent about $40 in cable, plus some really neat cable staples and a $10 5-port 100MB ethernet switch.

Last night was patch cord training, since I had not put together an ethernet cord since sometime in 1998. After two hours I had three completed patch cords that could actually pass the gizmo tests.

Today I wired my office, ran a line to PJ's room and made more patch cords. This is what the network looks like right now:





Red: VoIP line (off the Comcast Arris MTA)

Green: 100MB ethernet in my office

Blue: 100MB line to PJs room

Orange: 100MB ethernet in PJs room

There was virtually no benefit to the Mac Book Pro (which was never more than 10 feet away from the wireless access point), but my AppleTV is a little bit more responsive. The real benefit is that now there are no more networking issues with the stuff in PJs bedroom.

Mah Hero

There is this curious thing that happens when you are dealing with movies, music videos and TV shows from iTunes: the bulk update can't edit all of the video dependent fields, like video type, show name, season number, etc.

This is not that big a deal unless you love your AppleTV so much that you decide to rip all of your legally owned DVDs into a huge hard disk so you can watch them from your AppleTV. Try editing a network season of 24 shows, one episode at a time. Apple will probably catch up on this before iTunes 8 is out, after all they are fanatic about UI consistency, but for now we are screwed by having to do it by hand.

Hence, mah hero: Doug Adams.

Not only did Doug fix this with a simple Applescript, he put it on his website for free (he does take donations). The damn thing even has an installer! I find this amazing because adding an Applescript to iTunes involves grabbing a file and copying it to a folder in the library. Yet he went through the extra step of providing an installer script.

Photo Credit: Photo by cosmonautirussi, used under the terms of a Creative Commons license.

An open letter to HDMI Licensing, LLC (www.hdmi.org)

I am just one of many customers that feel that HDMI is a very promising technology. I personally own two HDTV screens with HDMI ports (one has two ports), two Microsoft Xbox 360s with HDMI ports, two AppleTVs and one upscaling DVD player with HDMI port. I like the standard because it allows me to move all of the required signals between a source device and a screen, which cuts down on clutter. The obvious benefits of these signals being digital are just icing on the cake.

The problem is consistency. Your own website says it the best: one cable, one standard. I should be able to plug one of my two Xbox 360s into any of my HDMI ports in either HDTV screen and it should work. I should be able to do the same with either AppleTV or the upscaling DVD player. And I should be able to mix and match between my three different HDMI cables, they should all work the same, because after all, HDMI is a standard.

The reality of the matter is that it does not work that way. For example, one of my HDTVs refuses to work with an Xbox 360 through HDMI ports, with different brands of cables, and with two different Xbox 360s. The upscaling DVD player will take up to 30 seconds to acknowledge that it is connected to an HDMI source. One of my two AppleTVs don't like when I turn off the TV, it takes up to a minute for it to auto detect the TV.

The funny thing is all of these devices are digital. We are used to that in the digital realm, things either work or they don't.

A good example is USB. There are less than 10 connector types for USB, and those are physical formats, electrically they are the same. If you find the right plugs, it doesn't matter what devices you are connecting, the USB layer will work. You never hear of people using the right USB format and not being able to connect two devices that are designed to work together. It just doesn't happen.

Another great example is Firewire, which has even less connector types. Firewire always works, period.

Why is HDMI failing at this? We are buying expensive HDTVs, digital cable boxes, PVRs, computers, A/V receivers. If it has any bearing to audio/video, it either has HDMI already, or somebody is trying to add it to it. We are paying a lot of money for these cables, in some places people are paying up to $50 per linear foot. That's a lot of money for a technology that is supposed to work 100% of the time and it doesn't.

Why?

When it works, it is fantastic. It is obvious HDMI is not a fluke. Then what is the problem? Shoddy vendor implementations? Sub-par cabling? Software?

We are going to be stuck with HDMI for a while, so please, figure out who is not getting along with the program. It is time to start enforcing the brand (think about it: when HDMI doesn't work, nobody says "fuck this piece of shit Xbox 360, " instead they say "fuck this piece of shit overpriced HDMI bullshit, I'm returning this $50 cable tomorrow") and the standard so whenever a customer buys HDMI, he/she gets HDMI and not a half-assed implementation.

Caution when updating your iTunes to 7.6.1

Be careful when updating iTunes to 7.6.1. If your library is not in the default location, it may reset it. For example, my library is in an external drive, and it reset it to Music:iTunes:iTunes Music. As I spent the day adding movies and TV shows to my library, they were stored in my laptop drive instead of my external drive. Fixing the problem only takes a few minutes, but then you have to remember that all of your videos will be clumped into the Movies group, so you must edit them one by one if you need to classify them as TV shows.

Photo by ElitePete, used under a Creative Commons License.

An Open Letter to Warner Brothers

Dear Warner Brothers,

As a recent owner of two AppleTVs (one for my child, one for us adults), I was extremely thrilled when my wife showed me that her "Harry Potter and the Order of the Phoenix (Two-Disc Special Edition)" included a 100% legal digital download. How nice of you guys to save me the trouble of having to download an illegal copy so I can watch the movie, or having to rip the DVD under my fair use rights.

The problem is, it doesn't work.

The instructions are quite simple: the box has a flier with a serial number. You are expected to pop-in the second DVD into your computer, select the digital download choice in the menu, then enter the serial number. And yes, the box has a sticker that clearly says that this feature is only Windows compatible.

Then how come I can't run it in Windows?

I followed the instructions perfectly, and what did I get? A pop-up menu that automatically changes pages and won't allow me to enter the serial number. Is it really that hard to make an auto play menu that runs properly in Windows? I am not even asking for it to work on Mac OS, but Jesus Christ, to what depths of suck do you have to fall into so you can't even deliver a god damn auto play menu that works properly in Windows? And yes, I am talking XP SP2, not some odd duck.

The sad thing is that when my wife handed me the box the very first thing I told her was "sweet, I don't mind paying for the DVD, I just don't want to waste the time to rip it."

Go Fuck Yourself, Warner Brothers.

iTunes 7.6.1 out: bug fixes, plus a weekly 99 cent rental special

iTunes 7.6.1 is out, I haven't been able to find the actual release notes, but so far it is reported as limited to fixes for AppleTV Take Two. On a side note, Apple is now offering a 99 cent movie rental every Thursday. You still have a month to start watching the movie and once started you have 24 hours to finish watching it.

Photo by heyjoewhereyougoingwiththatguninyourhand, used under a Creative Commons License.

More on AppleTV Take Two

After a weekend of basically beating the living crap out of our two AppleTVs (40GB), it is obvious to us that this thing is definitely the wave of the future. It is literally a cable box on crack.

It took me less than a day to get PJ convinced to stop watching videos in his Xbox 360 (streamed from a Mac running Connect360) and instead use the AppleTV. His AppleTV was running, almost nonstop, every minute he was awake over the weekend. And today, school holiday, the same deal. He woke up and immediately turned it on.

His only problem is that sometimes he gets lost in the menus, not because he can't read them, but because hitting the menu button multiple times doesn't keep moving you up the menu hierarchy, which is not very intuitive. Instead it alternates between two actions.

I am having a ball with my own, so far my only real frustration is that I can't use my second HDMI port in my 37" Olevia. It's not the end of the world, at 1080i and with nice component cables it looks almost identical to HDMI.

What probably became more of a challenge was to learn certain workarounds in iTunes. For example, you can't do a multiple edit for video files if you want to edit the video dependent tags, instead you must edit them one by one. I found some utilities to work around this, but I hated them all.

I think so far the most puzzling issue was that I was not seeing new artwork as I was assigning it to movie files. I found a post in the Apple support forums where it explained how to force it to see the new art. All you have to do is delete the file from the library, but leaving the file itself in the same place. When you add it again, it only takes a few seconds, and it will force the AppleTV to load the new art.

Another issue that had bugged me forever: space. This Mac Book Pro has a 200GB drive. After taking into account Parallels Desktop, iTunes and iPhoto, I was left with maybe 20GB or so to play with. I had forgotten that nothing forces you to run your iTunes library in the default location in your home folder. In fact, it can run from anywhere. Ivette bought me a 500GB external drive so we could back up whatever shows we bought for PJ, since we did not want to risk the iMac dying for good and having to pay for these shows twice. I decided to tell iTunes to switch my library home to a folder in the 500GB drive, and the damn thing just worked. First it spent a few minutes rewriting the library file, then it copied everything to its new location. The process did not take long, and now I have basically unlimited storage for my iTunes.