Edit: My solution that was here previously didn’t work properly. I’ve developed a Safari Extension to correct this, and am currently testing it. If it passes the tests, I’ll likely put it up here.

Sometime soon, I’ll get around to releasing my code for fetching old Escape Pod episodes that I hinted at earlier.

Background
First, some background on how drives work internally: A hard drive stores data on the platters in blocks, or sectors. The sector size is typically 512 bytes for a magnetic disk. With each sector, the disk must store a start code and an Error Correction Code (ECC). The start code is necessary so the disk can find a particular sector, and the ECC is necessary to correct any errors in the data read from the disk.
Western Digital decided to change from 512 byte sectors to 4096 byte sectors. This means that the new sector can store the same amount of data as 8 of the previous sectors. Since each sector needs a start code, this means there are now 7 fewer start codes for each new sector.
The ECC of the 4096 byte sector needs to be larger than the ECC of a 512 byte sector, but it doesn’t need to be 8 times as big. This is due to the fact that ECC is better at correcting errors with larger block sizes and thus a lower rate (or percentage in size) ECC can achieve the same Bit Error Rate (BER). Interesting side note: Hard drives are designed to store data so compactly and so close to the limits of the hardware that data is expected to have some corruption as it was stored, and rely on the ECC to correct the corruption. In fact, the manufactures intentionally design the drives so that it will have a few corruptions but few enough that the ECC can handle it.
With the reduction in the number of start codes, and decrease in ECC rate, larger sector sizes allow for greater density of data on the drive itself.

Writing Sectors
When a computer does a read, it reads an entire sector at a time. When it does a write, it, again, writes an entire sector at a time. There is no such mechanism to write only a part of a sector. Western Digital’s EARS drives have 4096-byte sectors, but the drive claims it has 512-byte sectors (since some opperating systems still in use cannot handle different sector sizes). This means that the drive does some work internally to handle this difference. Often, when writing data to a hard drive, multiple consecutive sectors are written. If a computer writes 8 512-byte sectors in a row, which all correspond to the same 4096-byte sector, then the drive can simply wait until all 8 sectors are written and write out all the data as a single 4096-byte sector. This is what the EARS drives do when sectors are written. If not all 8 sectors are written, then the drive has to do a partial sector write. This is typically not the case since many file system operate on 4096-byte blocks, and thus write 4096 bytes at a time.

Partial Sector Write
If not all 8 512-byte sectors in a 4096-byte sector are written, then the drive has to write part of a sector. So, the drive must find the data on the drive, and write the part of the sector that has changed, right?
Wrong. The problem is the ECC was written with the 4096 bytes that was there previously, so it must be updated. Due to the nature of how the ECC is computed, one cannot write a new ECC without first reading at least some of the data that was in the original sector. While it is possible to read the data to be overwritten, the old ECC, and compute the new ECC, this can propagate errors beyond the ECC’s ability to correct. So, the drive must read the 4096-byte sector, correct any data errors in the sector, substitute in the changed data, compute the new ECC, wait for that part of the drive to spin around again, and write the data to the drive. In comparison, if all 8 512-byte sectors were written, then the drive need only compute the new ECC and write the data.
So, if a single 512-byte sector is written, the drive must read 4096 bytes, wait for a rotation, and then write 4096 bytes. On a 7200 RPM drive, the waiting for a single rotation is 8.333 ms. If one assumes no smart scheduling, this results in a maximum throughput of a megabyte every 2 seconds, which is reminiscent of drives from the previous decade. Obviously, intelligent scheduling is utilized, but in the best case scenario, the drives performance will be dropped to half. From this, one can see that it is crucial to always write 4096 bytes at a time to get the best performance.

Alignment
As stated in the previous section, the best performance comes when 4096 bytes is written at a time, but this data must also be properly aligned. If 2048 bytes is written to one sector and 2048 to the next, then this involves 2 reads and 2 writes of 4096 bytes each. So all writes must be aligned on 8 sector boundaries (sector 0, 8, 16, 24, 32, etc) This is why Western Digital put in a jumper to help with the alignment since some formatting methods start on sector 63, which is not an 8 sector boundary. Those on Unix/Linux systems must pay special attention to this when formatting the drive.

Solutions
The solutions depend on the file system you are using. If you know your file system uses 4096-byte blocks, then you only need to ensure that the file system is properly aligned on 8-sector boundaries. So, when partitioning the drive, ensure that the partition offsets are divisible by 8.

ZFS
So, what does this have to do with ZFS? ZFS doesn’t have a pre-set block size. It uses variable sized blocks depending on the amount of data it is writing. If it’s writing 1000 bytes, then it will write the minimum number of sectors necessary to fit that data, which in the case of 512-byte sectors, is 2. This means that writing 1000 bytes requires reading and writing 4096 bytes or maybe 8192 depending on alignment. This means that properly aligning the partition will not solve the issue with ZFS. Here we need a different solution.

File System Agnostic Solution
The best solution is for the file system to think the drive has 4096-byte sectors. Since the drive will not claim this it needs to be done in software (really Western Digital, did you think this was not worthy of a jumper on the drive? It would have made things much easier). Since ZFS has performance issues on Linux anyway due to it’s kernel’s license, I won’t cover it at all. The primary case I’ve seen this issue raised is from those using FreeNAS on FreeBSD. Here, there are two solutions, depending on whether you want your drive encrypted or not. Both of these solutions give a new device, which the rest of the OS sees as a normal drive, but the drive has 4096-byte sectors, which are properly aligned. Warning, both of these methods destroy the data on the drive.
Not Encrypted
The drive can be presented as a drive with 4096-byte sectors by using gnop. Here is how one would create a raidz1 pool for da1-4:
for i in da1 da2 da3 da4; do gnop create -S 4096 $i; done
zpool create tank da1.nop da2.nop da3.nop da4.nop
The for loop must be run on each reboot otherwise ZFS will not see the da*.nop devices.
Encrytped
The encryption setup is similar:
for i in da1 da2 da3 da4; do geli init -s 4096 $i; done
for i in da1 da2 da3 da4; do geli attach $i; done
zpool create tank da1.eli da2.eli da3.eli da4.eli
Here the second for loop needs to be executed on each reboot. On FreeNAS, the encryption can be made simpler by setting up the drives in the GUI to use encryption, then after the encryption is setup, go into the shell, and execute the two for loops, then continue setting up the encrypted drive in the gui to use ZFS.

So, that’s how to handle ZFS on WD EARS drives. I’ve read reports that some, if not all, of what I described in the solutions will be in the next version of FreeNAS. Here’s hoping that it will.

BTW, the primary legacy Operating System that cannot properly handle the 4096-byte sector is Windows XP.

The problem is the following: A Raid-Z configuration uses n partitions, giving the user the capacity of n-1 of those partitions, with the nth being the redundant to survive a failure. If the n partitions are not the same size, with the smallest being x, only the first x bytes of each partition is used. One cannot remove a Raid-Z from an active pool without a backup/restore. One cannot add a disk to a Raid-Z without a backup/restore (maybe in the future). The only expansion that can be done is to replace the partitions with larger ones, and once the smallest partition is increased, the available space increases.

For the purposes of demonstrating this technique, I will use an example with 4 disks:

• 125G
• 250G
• 500G
• 750G

The biggest theoretical capacity of the array, while retaining single disk failure resiliency, is simply the total size of the array minus the largest disk. This means that the capacity of the above example is 875G. So, how does one achieve this capacity? I proposed the following structure using 3 raids:

If you add the capacity in the above, you can see it is 3*125G + 2*125G + 250G = 875G, which is the theoretical capacity.

Suppose I wish to replace the smallest disk, 125G, with a new disk, say 1T. Using the same layout, I should see something like this:

If you add the capacity in the above, you can see it is 3*250G + 2*250G + 250G = 1500G, which again is the theoretical capacity.

The question is, can I migrate from the first configuration to the last without a backup/restore process? The answer is YES, and data is moved/copied at most once.

For the purposes of demonstration, I’m going to show the expansion with 5 drives connected, but the expansion can be done by immediately replacing the smallest drive with the largest and relying on the redundancy to keep things intact in the process.

First, connect the 1T drive and partition it as 250G, 250G, 250G, leaving the rest free. Replace the first 125G partition on 750G disk with the first 250G partition on the 1T disk:

Continue the replacement:

Repeat the same process with the 3 partitions on the 500G disk:

At this point, the move of the blue raid is complete. Depending on the sizes of the disks, the blue raid may be bigger, at which point the pool immediately increases in size. In this example, it does not change size.

Repeat with the 2 partitions on the 250G disk:

At this point, the red raid is complete. In this example, the red raid has changed from 250G to 500G, and the job isn’t complete yet!

Repeat for the final time with partition on the 125G disk:

Now we are done, and can disconnect the 125G disk if not done already. In this example, the green raid has changed from 375G to 750G, yielding a total change of 875G to 1500G.

Note: There is one limitation with this structure. Ordering the disks in increasing size, disk(n) must be at least as big as 2*(disk(n-1))-disk(n-2). This essentially means that if a disk is xG bigger than it’s previous disk, then the next disk must be at least xG bigger than this disk. Since disk sizes tend to grow exponentially, this assumption shouldn’t be much of a problem since the requirement is at least linear growth.

The disk sizes in this example meet the requirement I listed above. Given the requirements of this expansion, the disk added in this example must be at least 1000G, since disk 4 is 250G bigger than disk 3, disk 5 must be at least 250G bigger than disk 4, or 1000G.

This technique also allows adding another disk, under the same conditions. Consider adding a 1T disk and creating another Raid-Z:

I didn’t structure the partitions in nice pretty rows to demonstrate the fact that not all of the data needs to be moved (in this case, 750G of data is not moved). In this example, the capacity increased from 875G to 1625G, again the theoretical capacity.

Anyway, I thought of this while considering using ZFS/Raid-Z in a FreeNAS setup. I haven’t tested any of this; it’s theoretical only. What do you think?

Kevin sent me the code he used for DVD playback inside NitoTV as a place to start. I stripped it down to a smaller piece, and started the framework. After I had it doing basic playback, I worked on overlays to provide feedback to the user. Now, hitting up and down changes the overlays between normal, chapter view, audio/subtitle selection, and zoom.

Chapter Overlay Sceenshot

Also, I added a pretty menu using a blurred image as the resume menu. This is more consistent with Apple’s own playback

Resume Overlay

In addition, I used my work on AC3 Passthrough in Perian to see if I could pull off the same in DVD Playback. The issue is the AppleTV claims to only have a device which can play uncompressed audio, not one that can send Dolby Digital to a decoder. So, I created an audio driver that claims to do exactly that and pipe the data through to the optical/HDMI port. It was a lot of fun getting that working, and then I added support for DTS.

Anyway, now Sapphire uses the common media player framework for DVD playback. In the future, we’ll add other playback mechanisms as well.

I started to suspect that this was my fault, since this one podcast, Escape Pod was a podcast I was listening by going through its archives. I had written a program which downloads the archived episodes, and modifies the ID3 tags so iTunes will properly recognize the file as a podcast and insert it into its database as if it had downloaded it itself. Getting the release date working correctly was one of the harder points, so I figured that I still didn’t have it quite correct and the new iPod firmware was being more sensitive to it. I got even more annoyed by the fact that iTunes, after upgrading my iPod’s firmware, decided to trash the old firmware versions preventing me from ever downgrading.

Then, I read online discussions where others were having this problem. Now I know that it is not my fault, but rather something that Apple screwed up. I also saw someone who noted that they had upgraded their iTunes, but not their iPod, and had the same issue. So, I trashed iTunes, pulled out my Snow Leopard disk, found the iTunes package on the disk, installed it, reverted my iTunes library to the backup prior to the upgrade, and voila, my podcasts are sorting correctly again.

Now if only I will learn to let others try first, and upgrade myself after reading their reports. This is a really sloppy bug on Apple’s part, and likely means that iTunes 9 was rush. I’m actually happier back on 8, especially since the window zoom still functions as a toggle between the mini player and the normal window, which was changed in 9.

Hopefully someone will benefit from this post, and not upgrade to iTunes 9 until this regression is resolved.

In my test case, I worked with the NASB translation of the Bible. The raw text of this translation, minus formatting and book/chapter/verse identifiers is 3.965MB. Since the iPhone already has zlib, using gzip compression is an obvious choice. When compressed with gzip, the file size becomes 1.189MB, a significant savings. Even though bzip2 is not readily available on the iPhone (at least not that I could find), I tested its compression which produced a file size of 0.8548MB. While these mechanisms provide a significantly smaller file, when one desires a certain portion of the file, one must first decompress the entire file up to that point. This is an expensive operation on a small device such as the iPhone.

One compromise is to compress each book individually. This yields a file size of 1.242MB. However, some books, such as Psalms, are still quite large, requiring a long decompression operation to read some of it’s chapters. Since the application displays a chapter at a time, a logical compression block would be a single chapter which yields a file size of 1.628MB. While this is a large increase in file size, a single chapter can be decompressed without requiring decompression of any other parts of the file.

I was not happy with these compression ratios and was determined to find a better way. I looked into Huffman encoding, which uses a variable length string of bits to represent a symbol. Its compression ratio does not change with the order of symbols, which typically means it won’t provide as high a compression ratio. However, if you are told which bit is the start of a string, you may begin decompression without examining any of the file which precedes it. In addition, the decompression scheme only requires walking a binary tree, which means it is also quite fast. So, if Huffman encoding can be made to provide a good compression ratio, it will work well for this problem.

When using Huffman encoded, the question becomes how to decompose the string into symbols. One typical decomposition is to make each character into a symbol. While this is an easy representation, it doesn’t provide a good compression ratio. Instead, I chose to make each word into a symbols. After adding in punctuation, the tree contained 16,246 words. This resulted in a dictionary size of 0.1364MB. When compressed, the entire Bible was represented by a Huffman encoding of 0.9947MB, meaning a total of 1.131MB for the compressed stream and dictionary.

After this experiment, I concluded that Huffman encoding of words is the best for a large quantity of text. It yielded a compression ratio that was better than gzip (though not as good as bzip2), but at the same time a scheme that could decompress individual parts of the file without having to read preceding parts. In this scheme, a single chapter can be decompressed very quickly, and still have a high compression ratio.

The whole ordeal was mostly due to the CARS program, which is often known by “Cash for Clunkers”. Anyway, I finally got that resolved and got a lot more for my old truck than I would have ever gotten in trade-in value. So now I have a VW Jetta, and so far I’m happy with it. I just need to get used to driving a car instead of a truck, but otherwise it drives well.

So, my options are:

• Only use the command line utility, or use Xcode, on a single repository checkout, but never both.
• Use a horribly out of date version of Subversion
• Hack Xcode’s plugin to work with newer versions of Subversion

I chose the last, as it should give me the best of both worlds. If you were to conduct a quick Google search, you would find a whole host of people posing a solution which involves moving around system libraries!!!!! This is a horrible idea, as it breaks other things such as Apache. Buried somewhere in the sea of bad ideas, I ran across a good one, written by Jean-Daniel Dupas and improved by Philippe Casgrain which only changes the path location of libraries used by Xcode’s Subversion plugin. This script worked well for Subversion 1.5, but recent versions of Subversion would cause Xcode to crash in libapr. The solution is to add libapr and libaprutil to the script. Here is the corrected script for your use.

#!/bin/sh

echo "Updating install path using svn libraries in \"$2\"..."
install_name_tool -change \
/usr/lib/libapr-1.0.dylib \
$2/lib/libapr-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libaprutil-1.0.dylib \
$2/lib/libaprutil-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libsvn_client-1.0.dylib \
$2/lib/libsvn_client-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libsvn_delta-1.0.dylib \
$2/lib/libsvn_delta-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libsvn_diff-1.0.dylib \
$2/lib/libsvn_diff-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libsvn_fs_fs-1.0.dylib \
$2/lib/libsvn_fs_fs-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libsvn_fs-1.0.dylib \
$2/lib/libsvn_fs-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libsvn_ra_local-1.0.dylib \
$2/lib/libsvn_ra_local-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libsvn_ra_svn-1.0.dylib \
$2/lib/libsvn_ra_svn-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libsvn_ra-1.0.dylib \
$2/lib/libsvn_ra-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libsvn_repos-1.0.dylib \
$2/lib/libsvn_repos-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libsvn_subr-1.0.dylib \
$2/lib/libsvn_subr-1.0.dylib "$1"
install_name_tool -change \
/usr/lib/libsvn_wc-1.0.dylib \
$2/lib/libsvn_wc-1.0.dylib "$1"
echo "Done!"


Now, how long till Xcode supports Mercurial?