Make a disk image

The first step for a data recovery project should be to make an image of the drive or partition: when a drives starts to lose data because of physical errors on the disk these errors tend to spread, things are getting worse and you will not get any data out of anymore at some point. And every tool will encounter the same problems when trying to read defective sectors on disk, and it will not be possible to repair these.

By the way, if your data is really valuable you shouldn’t even try to recover it yourself, that is, if the drive shows signs of some physical damage. You should disconnect it as soon as possible and hand it to a professional data recovery company – if it is worth a few hundred or thousand Euros.

The data I was working with, however, wasn’t business critical. So, after consulting $SEARCH_ENGINE, I did an image of the drive with dd_rescue – this works similar to a normal “dd” but handles I/O errors more gracefully. But here things started to get confusing: there are two programs for this purpose with an almost identical name:

• Kurt Garloff’s original dd_rescue tool uses the executable named “dd_rescue”, the Debian/Ubuntu package is named “ddrescue”.
• Antonio Diaz Diaz new and improved GNU ddrescue provides an executable named “ddrescue”, the Debian/Ubuntu package carries the name “gddrescue”.

The latter is the one to choose: you get much better progress information – copying hundreds of gigabytes takes quite some time, so you want to know what’s going on – and the capability to interrupt the process and continue where you left off with the help of a log file. After finding out about that the hard way I got my image with

sudo ddrescue -r3 /dev/sdb1 hdimage logfile

where “-r3″ means: “in case of an error, retry 3 times” and /dev/sdb1 is the name of the partition of the USB disk, obviously.

Unfortunately, the resulting image still couldn’t be mounted. “ddrescue” only reported a few bad sectors on the disk, but it was obviously enough to make file access impossible. Another idea that I wasn’t able to pursue: it might have been possible to repair the NTFS filesystem with a virtualized windows instance running in VirtualBox – but VirtualBox only takes complete disks as images, not single partitions. If I had done an image of the complete disk instead, including the partition table, this might have worked out. I didn’t feel like copying the 230 GB over into a new image with a partition table and also didn’t have enough free disk space to do it.

Recovery tools: file carvers

The next step was trying to recover as much of the data as possible. I had successfully used a “file carver” before to recover images from my digital camera’s memory card after the FAT filesystem became corrupted. A file carver is a program that scans a raw binary stream for the headers of known file types, like that of JPEG images or MP3 audio files, and tries to extract the contents, completely ignoring the file system. The advantage is that it doesn’t matter how broken your filesystem is – the program doesn’t have to know anything about the filesystem’s structure. It can also recover deleted files. The disadvantage is that you lose all information that is stored in the filesystem, the file name and directory structure. It’s also prone to errors for fragmented file systems, which also means that you’re less likely to succeed when recovering large files.

I tried two tools from this category, “foremost” and “photorec”. “foremost” is a simple command line tool, you call it like this:

foremost -i hdimage -o recovered -v

and it will sort the files it can find by file type into sub folders of “recovered”.

Photorec has a curses interface. It also takes hints about the structure of the image, like presence of a partition table or filesystem type. It is part of the “testdisk” package. The command line invocation shows that this tool was ported from DOS:

photorec /log /debug /d output-directory hdimage

Recovery tools: Sleuth Kit

Researching further, I stumbled upon the Sleuth Kit and Autopsy. These are forensic analysis tools and therefore are designed to recover data that someone deliberately tried to hide or destroy. The Sleuth Kit is a suite of command line tools which Autopsy is a web frontend for. Autopsy comes with its own web server. I started it with these commands:

mkdir my-autopsy-dir/
autopsy -d my-autopsy-dir/
firefox http://localhost:9999/autopsy

Getting around the web interface can be a bit confusing: you have to create a “case” first, then add a host to investigate and finally a hd image to look at. Anyway, the time it took me to get used to autopsy wasn’t wasted because I now was able to see the complete contents of the original NTFS filesystem! I was able to look at the data, browse the filesystem, download single files and compute MD5 sums. However, autopsy offers no feature for copying whole directory trees. This is because it is intended for forensic analysis rather than data recovery. So you, the computer forensics expert, are supposed to look at every single file and make notes about it which in turn are then recorded in the “case”.

I wasn’t really interested in a forensic analysis of the contents of my friend’s drive so I took a closer look at the command line tools. The relevant commands from the Sleuth Kit are “fls” for listing files in an image and “icat” for getting at the contents. You use “fls” like this:

fls -urp hdimage

where -u means that I’m not interested in deleted files, -r that I want a recursive listing and -p that I need to have the full path for every file. The output looks something like this:

d/d 180-144-8:  some-dir
d/d 5192-144-1: some-dir/some sub dir
r/r 5190-128-3: some-dir/some sub dir/some_file.exe
r/r 5188-128-3: some-dir/some sub dir/another_file.jpg

The funny numbers in the second column are the “inode” of the file, which you need to feed into “icat” to get the contents. So how do you recover a whole directory tree with these tools? What I should have done is using a script like this one:

#!/bin/sh
IMAGE=hdimage
fls -urp $IMAGE | 
while read type inode name; do
    case $type in
        d/d) mkdir "$name" ;;
        r/r) icat $IMAGE $(echo $inode | sed 's/://g') > "$name" ;;
    esac
done

But I was lazy and so I saved the file listing in a text file which I turned into a big shell script using Emacs’ rectangle functions, regular expressions and keyboard macros. This wasn’t working so well: there were some funny characters in the file names I forgot to escape, like single quotes and backticks. So, as always, it turned out to be more work doing it “the easy way”. However, in the end I was able to completely recover the data from the partition.

Analyzing the data

Since now I got all the data back, having already tried other methods of recovery before, this can serve as a nice real world benchmark of the usefulness of the file carving tools I used.

Just counting how many files these tools think they’ve found doesn’t help us much, we also need to know if the recovered files were really complete and undamaged. I did a quick check with the files the Sleuth Kit recovered, and all files I checked seemed to be ok: the photos were fine and the videos and mp3s played without any hiccups. So, let’s assume that the data I got from the Sleuth Kit is really genuine. To find out about the identity of the recovered files, I computed the MD5 hash for all of them with this little script:

for tool in foremost photorec sleuthkit; do
    find $tool -type f -print0 | xargs -0 md5sum | tee md5sums/${tool}.txt
done

And here’s a script I hacked together to do some analysis on these files:

#!/bin/bash
 
md5s_by_ext() {
    local ext=$1
    shift
    grep -hi "\.${ext}\$" "$@" | awk '{ print $1 }' 
}
unique_md5s_by_ext() {
    md5s_by_ext "$@" | sort | uniq
}
unique_md5s() {
    cat "$@" | awk '{ print $1 }' | sort | uniq
}
clean_wc() {
    wc -l | sed 's/ //g' 
}
 
common_files() {
    local ext="$1"
    echo -ne "${ext}\t"
    echo -ne $(unique_md5s_by_ext $ext sleuthkit | clean_wc) "\t"
    for tools in photorec foremost "photorec foremost"; do
        echo -ne $(unique_md5s_by_ext $ext $tools | clean_wc) "\t" \
            $(comm -12 <(unique_md5s             $tools) <(unique_md5s_by_ext $ext sleuthkit) | clean_wc)"\t"\
            $(comm -12 <(unique_md5s_by_ext $ext $tools) <(unique_md5s_by_ext $ext sleuthkit) | clean_wc)"\t"
    done
    echo
}
 
common_files_total() {
    echo -e "total\t"\
         $(unique_md5s sleuthkit         | clean_wc) "\t"\
         $(unique_md5s photorec          | clean_wc) "\t"\
         $(comm -12 <(unique_md5s photorec) <(unique_md5s sleuthkit) | clean_wc) "\t\t"\
         $(unique_md5s foremost          | clean_wc) "\t"\
         $(comm -12 <(unique_md5s foremost) <(unique_md5s sleuthkit) | clean_wc) "\t\t"\
         $(unique_md5s photorec foremost | clean_wc) "\t"\
         $(comm -12 <(unique_md5s foremost photorec) <(unique_md5s sleuthkit) | clean_wc)
}    
 
echo -e "\tsleuthkit\tphotorec\t\t\tforemost\t\t\tphotorec+foremost"
 
common_files_total
for i in jpg gif mp3 avi mpg zip rar exe cab dll txt htm rtf pdf doc xls; do
    common_files $i
done

And here are the results as a really ugly table:

This table needs a bit of an explanation:

• “found” means the number of files the tool extracted from the image
• “matching” means the number of files the tool found that are identical with files recovered with the sleuth kit
• “matching+ext” means that we’ve also got the extension right

Foremost recovered almost 90% of the images, Photorec following close behind 85%, and it found only 8 photos that foremost couldn’t identify. Looking at other data types, Photorec is clearly superior: it found 24 of the 25 RAR files present in the image, foremost only got 8 of them right. And only photorec was able to recover any mp3s: it found 89% of them, but we also got quite some false positives here. Neither of the tools was able to recover any movies – possibly because they were fragmented on disk.

Conclusion

So here comes the take home message:

• Use GNU ddrescue to make a hard drive image first.
• If your filesystem is not mountable, try the sleuth kit – you might get all your data back including the file names and directory structure.
• If the sleuth kit fails and you’re trying to recover some photos, “foremost” might help you.
• If the sleuth kit fails and you’re looking for something other than images, give “photorec” a shot. Anyway, it’s less likely in this case that you’ll get your data back.

While having your personal config files under version control is a good idea, making your home directory version controlled isn’t – it’s not easy to see what files are under version control, and some operations just take ages because they have to crawl through your all your personal files. So I created a git controlled directory, named “.vc” and did symbolic links from there to the config files locations. As I add more files to version control, this will get tedious to do by hand, so I wrote a little script to help me that I want to share with you: I just called it “setup” and put it into .vc as well:

#!/bin/sh
cd $(dirname $0)
VC_DIR=$(pwd)
find . \( -path \*/.git \) -prune -o -type f -not -name "$(basename $0)" | \
    while read path; do
        if [ "X$(basename "$path")" = "X.git" ]; then
            continue
        fi
        destination="$HOME/$path"
        if [ -e "$destination" ]; then
            if [ -L "$destination" ]; then
                echo "Symlink for $path exists, skipping."
                continue
            else
                echo "backing up existing $destination to ${destination}.vc_backup"
                mv "$destination" "${destination}.vc_backup"
            fi
        fi
        echo "doing symlink for $path"
        ln -s "$VC_DIR/$path" "$HOME/$path"
    done

The other option is to use flymake with some command line javascript syntax checker. Two possible syntax checkers are Mozilla’s stand alone SpiderMonkey interpreter “smjs” with the “-s” (strict) option and Douglas Crockford’s JsLint running under Rhino. There are reasons to use each of them:

• SpiderMonkey is the javascript engine currently used in Firefox. So, if it can not parse your javascript, neither can Firefox.
• JsLint does more than just syntax checking, it also checks for bad style and possible programming errors, like missing semicolons and braces, assignment in an if conditions. This can be good, but it’s also a hinderance when you’re working on code that was not written to make JsLint happy and you don’t want to refactor it. Furthermore, the style enforced by JsLint might differ from your programming style, as it says on the web page: “Warning! JSLint will hurt your feelings.”
  I, for instance, like to leave out the braces after an if statement if the executed block is itself just one statement – it takes up less space, and if I ever want to add to this block Emacs will indent the code correctly, so I won’t forget to add the braces afterwards. However, maybe JsLint is right, because you sometimes have to collaborate with people that don’t use Emacs – maybe I should do them the favor.

Setting up SpiderMonkey

On an Ubuntu system the stand alone SpiderMonkey interpreter is provided by the package “spidermonkey-bin”. SpiderMonkey has the problem that it is just the javascript engine, but does not include the DOM, and there is no global window object. So if your javascript code does more than just defining functions you need to do some mocking. This is fortunately very easy to do.

The code I wanted to check also installs event handlers with prototype’s Event.observe method, so I had to mock that, too:

// fake the global window object
var window = this;
var Event = { observe: function() {} };

You will most probably need to extend this for your code. Now you can do a syntax check from the command line like this:

smjs -s -e "(load '/path/to/mock.js')" file-to-check.js

Setting up JsLint

JsLint is itself written in javascript. To use it from the command line you need the javascript interpreter Rhino. It is, not surprisingly, provided on Ubuntu by the package “rhino”. JsLint for rhino is avaiable here. To check a javascript file with rhino, just call it like this:

rhino /path/to/jslint.js file-to-check.js

Integrating both into Emacs

The following code allows you to do live syntax checking from within Emacs with either SpiderMonkey or JsLint. You can switch between them using “C-c t”. The code uses a circular list structure to keep the list of available methods, just for the heck of it.

(setq flymake-js-method 'spidermonkey)
(defun flymake-js-toggle-method ()
  (interactive)
  (let ((methods '#1=(spidermonkey jslint . #1#)))
    (setq flymake-js-method (cadr (memq flymake-js-method methods)))
    (flymake-start-syntax-check)))
 
(defun flymake-js-init ()
  (let* ((temp-file (flymake-init-create-temp-buffer-copy
		     'flymake-create-temp-inplace))
         (local-file (file-relative-name
		      temp-file
		      (file-name-directory buffer-file-name))))
    (if (eq flymake-js-method 'spidermonkey)
        (list "smjs" (list "-s" "-e" (format "load('%s')" (expand-file-name "/path/to/mock.js")) local-file))
      (list "rhino" (list (expand-file-name "/path/to/jslint.js") local-file)))))
 
(eval-after-load "flymake"
  '(progn
     (add-to-list 'flymake-allowed-file-name-masks
                  '("\\.js\\(on\\)?$" flymake-js-init flymake-simple-cleanup flymake-get-real-file-name))
     (add-to-list 'flymake-err-line-patterns
                  '("^\\(.+\\)\:\\([0-9]+\\)\: \\(SyntaxError\:.+\\)\:$" 1 2 nil 3))
     (add-to-list 'flymake-err-line-patterns
                  '("^\\(.+\\)\:\\([0-9]+\\)\: \\(strict warning: trailing comma.+\\)\:$" 1 2 nil 3))
     (add-to-list 'flymake-err-line-patterns
                   '("^Lint at line \\([[:digit:]]+\\) character \\([[:digit:]]+\\): \\(.+\\)$" nil 1 2 3))))
 
(defun my-js-setup-flymake ()
  (flymake-mode 1)
  (local-set-key (kbd "C-c n") 'flymake-goto-next-error)
  (local-set-key (kbd "C-c p") 'flymake-goto-prev-error)
  (local-set-key (kbd "C-c t") 'flymake-js-toggle-method))
 
(add-hook 'javascript-mode-hook 'my-js-setup-flymake)

The result

And here is what it looks like with SpiderMonkey:

And here an example of some code and what opinion JsLint has about it: