Earlier this week Google announced that they were introducing GPUs to their cloud offering. As I was reading the article I was reminded of a Hak5 video I saw recently about a password cracking program called Hashcat. The video talks about how Hashcat can use GPUs (and even FPGAs) to accelerate hash cracking. Naturally I began to wonder what Hashcat could do if I handed it 8 of Google's shiny new GPUs.

Setting up the VM

I've been running some VMs on Google Cloud for awhile now so my account was almost ready to go. The one catch was that Google doesn't give you GPU quota by default. I submitted a quota increase request for 8 GPUs in the us-east1-d zone of their cloud and within a few hours the quota had been granted (although they made me put $30 in the account first).

From the there it was pretty much just a matter of following the instructions in their developer docs. The first step was to make sure the gcloud tool was up to date. The cloud web UI doesn't support GPUs yet so this all needed to be done through the CLI utility. GPUs are supported by version 144 and up. I had just updated so I was already at a good version:

[email protected] ~> gcloud version
Google Cloud SDK 145.0.0
alpha 2017.02.21
beta 2017.02.21

After that I checked to make sure I had GPU quota available and it showed up as expected;

[email protected] ~> gcloud beta compute regions describe us-east1
  - limit: 8.0
    metric: GPUS
    usage: 0.0

And now for the exciting part, spinning up a VM with 8 GPUs:

[email protected] ~> gcloud beta compute instances create gpu-test \
     --machine-type n1-standard-2 --zone us-east1-d \
     --accelerator type=nvidia-tesla-k80,count=8 \
     --image-family ubuntu-1604-lts --image-project ubuntu-os-cloud \
     --maintenance-policy TERMINATE --restart-on-failure
Created ...
gpu-test  us-east1-d  n1-standard-2  RUNNING

Yes folks, that's how easy it is to own 8 GPUs now. The last step was to add my SSH key to the VM and SSH into it:

[email protected] ~> gcloud compute instances add-metadata gpu-test \
      --zone us-east1-d \
      --metadata-from-file ssh-keys=~/.ssh/id_ed25519.pub
Updated ...

[email protected] ~> ssh -i ~/.ssh/id_25519 [email protected] 

Installing the GPU drivers

Now that I was the proud owner of a 2010 Bitcoin mining rig I need to install the necessary drivers to make it run. Google provides some simple instructions in their developer docs so I just followed those:

[email protected]$ curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb

[email protected]$ sudo dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb 

[email protected]$ sudo apt-get update

[email protected]$ sudo apt-get install cuda

After waiting 20 minutes for the VM to install every font known to mankind (seriously, why did it install fonts when all I wanted was CUDA) the rig was ready to go. I used nvidia-smi to make sure all the GPUs showed up:

[email protected]$ nvidia-smi
| NVIDIA-SMI 375.26          Driver Version: 375.26    |
| GPU  Name        Persistence-M| Bus-Id        Disp.A |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage |
|   0  Tesla K80           On   | 0000:00:04.0     Off |
| N/A   43C    P0    58W / 149W |      0MiB / 11439MiB |
|   1  Tesla K80           On   | 0000:00:05.0     Off |
| N/A   58C    P0    70W / 149W |      0MiB / 11439MiB |
|   2  Tesla K80           On   | 0000:00:06.0     Off |
| N/A   45C    P0    60W / 149W |      0MiB / 11439MiB |
|   3  Tesla K80           On   | 0000:00:07.0     Off |
| N/A   58C    P0    73W / 149W |      0MiB / 11439MiB |
|   4  Tesla K80           On   | 0000:00:08.0     Off |
| N/A   41C    P0    57W / 149W |      0MiB / 11439MiB |
|   5  Tesla K80           On   | 0000:00:09.0     Off |
| N/A   54C    P0    72W / 149W |      0MiB / 11439MiB |
|   6  Tesla K80           On   | 0000:00:0A.0     Off |
| N/A   36C    P0    55W / 149W |      0MiB / 11439MiB |
|   7  Tesla K80           On   | 0000:00:0B.0     Off |
| N/A   55C    P0    71W / 149W |      0MiB / 11439MiB |

And there they are in all their glory. Next up was to get a copy of Hashcat running.

Running Hashcat

I grabbed the prebuilt binaries from the Hashcat website:

[email protected]$ wget https://hashcat.net/files/hashcat-3.30.7z
‘hashcat-3.30.7z’ saved [2575000/2575000]

And yeah, they actually use 7zip.... I installed p7zip and extracted the archive:

[email protected]$ sudo apt-get install p7zip-full

[email protected]$ 7z x hashcat-3.30.7z
Everything is Ok

Is it just me or does anyone else think more programs should print "Everything is Ok" to the console every once and awhile? From there I could go into the unpacked directory and run the benchmark:

[email protected]$ cd hashcat-3.30/

[email protected]$ ./hashcat64.bin --benchmark | tee benchmark.txt
hashcat (v3.30) starting in benchmark mode...

The benchmark took about 1.5 hours to run and went through every hash Hashcat supports. I've made the full output available in this gist. There's a ton of data there, but what I'm really interested in is how long will it take to crack a password of a certain length and how much that will cost. To answer those questions I broke out a bit of IPython notebook (or Jupyter notebook now?) kung-fu.

Analyzing the results

I ended up getting a little carried away with this one. The notebook I wrote parses the benchmark results and creates some interactive plots with them. There are options for selecting the password length, password character set, and which hash algorithms to show. The chosen values are used to generate plots for time and cost of bruteforcing different length passwords. I sampled a few common ones including MD5 (cough...Yahoo), SHA1 (cough...LinkedIn), SHA256, SHA512, bcrypt, and scrypt.

The differences between algorithms is easy to see:

Graph of password length vs time to crack for various hash algorithms

An 8 character password containing upper case letters, lower case letters, numbers and symbols would only take a few dozen hours to crack if it was hashed with MD5. SHA1 isn't much better and SHA256 and SHA512 are only marginally better. Real key derivation algorithms like scrypt and bcrypt separate themselves from the field though: scrypt would take about 1 million hours to brute force and bcrypt would take about 100 million.

In theory, if you wanted things to be faster you could spin up 100s or 1000s of these depending on how many GPUs Google will give you. The one thing that would remain constant would be the cost, so here's what it works out to:

Graph of password length vs cost to crack for various hash algorithms

Again looking at 8 character passwords, MD5 is hundreds of dollars and the SHA family is only thousands. The benefit of key derivation functions is clear here. Each takes many millions of dollars to pull off.

I guess the takeaway is: use the longest password you can. Even better use a passphrase. If major websites like Yahoo and LinkedIn are using outdated hashing algorithms for our passwords they may as well not be hashing them at all. With cloud computing anyone with a few hours and $100 can crack them. And if you're a site owner, for the love of God switch to bcrypt or scrypt.

I put the notebook on Jupyter's notebook viewer. If you want to check out the data for different hashing algorithms and password combinations you can download it and run it for yourself.