So what is a Tensor processing unit (TPU) and why will it be the future of Machine Learning?

05.02.2019 3,411 4

Martin Pramatarov

So, for sure you know that a CPU is. GPU is probably something that you know too. But have you ever heard of a TPU? TPU stands for a tensor processing unit. Google produces these processors. They are ASIC – application-specific integrated circuits. ASIC devices got popular with the cryptocurrency boom. The TPU has a different purpose – to accelerate machine learning workloads.

Advantages of the TPU

The TPU provides accelerated machine learning. Google’s team has created it for running deep learning models which are computationally demanding. Google has already been using it in some of its products – Translate, Photos, Assistant, Search and Gmail.

The TPU was designed to run machine learning models with AI on the Google Cloud platform. It can provide superb performance:

• Cloud TPU v2 – 180 teraflops and 64 GB HBM;
• Cloud TPU v3 Beta – 420 teraflops and 128 GB HBM;
• Cloud TPU v2 Pod Alpha – Very impressive 11.5 petaflops and 4TB HBM;

The machine learning model requires training which can be very time-consuming and costly. The TPU helps in this department, reducing both time and costs.

When to use a TPU?

This ASIC device is made for very specific workloads.

• Models without custom TensorFlow operations inside the main training loop;
• Models dominated by matrix computations;
• Models with a long training period – weeks or months;
• Large models with massive batch sizes;

When not to use a TPU?

Of course, there are cases when it is still more practical to use a CPU or a GPU. TPUs are not suitable for linear algebra programs that have various branching or are element-wise algebra. Not very good for high-precision arithmetic.

Use CPUs for:

• Quick prototyping;
• Simple models with a short training period;
• Small models with small batch sizes;
• Custom TensorFlow operations that are written in C++;

Use GPUs for:

• Models not written in TensorFlow;
• Models with a large number of custom TensorFlow operations that partially need to run on CPUs;
• Models where the problem is that the source does not exist or is too onerous to change;
• Models with TensorFlow ops not accessible on Cloud TPU;
• Medium and large models with big batch sizes;

Should you try out a TPU for your Machine Learning needs?

Short answer – yes! According to Google, it can deliver between 15 and 30 times greater performance than current CPUs and GPUs. And what’s even more impressive – 30 to 80 times greater performance per watt. Another excellent reason is that there’s a free trial. See if it is convenient for you before you decide to purchase it.