What Are Tensors?

Tensors have been around ever since William Hamilton coined the term 200 years ago to describe a mathematical object representing a set of numbers with some transformation properties.

But Albert Einstein truly brought them into the spotlight by developing and formulating the whole theory of general relativity entirely in the language of tensors. Even though he wasn’t a big fan of tensors, Einstein popularized tensor calculus more than anyone else.

Definition of a Tensor

A tensor is a mathematical object that generalizes scalars, vectors, and matrices into higher-dimensional spaces. It’s an array of numbers and functions encompassing physical quantities, geometric transformations, and various mathematical entities. In a way, tensors are containers that present data in n-dimensions. They are typically grids of numbers called N-way arrays.

The word tensor, however, is still somewhat obscure. You won’t hear it in high school. Your math teacher may have never heard of it. But state-of-the-art machine learning frameworks are doubling down on tensors, with the most prominent example being Google’s TensorFlow.

The mathematical concept of a tensor could be broadly explained by describing a scalar as the lowest dimensionality and is always 1x1.

It can be considered a vector of length 1 or a 1x1 matrix. It’s followed by a vector, where each vector element is a scalar. The dimensions of a vector are nothing but Mx1 or 1xM matrices.

Then we have matrices, which are a collection of vectors. The dimensions of a matrix are MxN. In other words, a matrix is a collection of n vectors of dimensions m x 1. Alternatively, a matrix can be an array of m vectors of dimensions n x 1. Since scalars make up vectors, you can also think of a matrix as a collection of scalars.

Ranks

A rank is a fundamental concept that refers to a tensor's number of indices or dimensions. It determines the types of operations a tensor can perform and thus plays a vital role in manipulating multi-dimensional data.

In this sense, scalars, vectors, and matrices are all tensors of ranks 0, 1, and 2, respectively. An object we haven’t seen is a tensor of rank 3. Its dimensions could be signified by k,m, and n, making it a KxMxN object. Such an object can be thought of as a collection of matrices.

How Do You Code a Tensor?

Let’s look at that in the context of Python. In terms of programming, a tensor is no different than a NumPy ndarray. You can store tensors in ndarrays—a common way to deal with the issue. Generally, it’s a two-step process.

Step 1: Create a tensor out of the two following matrices.

The first matrix, m1, will be a matrix with two vectors: [5, 12, 6] and [-3, 0, 14].
The second matrix, m2, contains the following: [9, 8, 7] and [1, 3, -5].

Step 2: Then, you can create an array, T, with two elements: m1 and m2. After printing T, you’ll realize that it contains both matrices:

It’s a 2x2x3 object containing two matrices, 2x3 each.

If you want to create the same tensor manually, you can write the following line of code:

With its many elements and confusing brackets, tensors are challenging to create by hand. You’ll mainly load, transform, and preprocess the data to obtain tensors. But it’s always good to have a theoretical background.

Why Are Tensors Useful in TensorFlow?

In machine learning, we often explain a single object with several dimensions. For instance, a photo is described by pixels—each with intensity, position, and (color) depth.

If we’re talking about a 3D movie experience, each eye could perceive a pixel differently. That’s where tensors come in handy and are incredibly scalable—no matter how many additional attributes we add to describe an object, we can always include an extra dimension to our tensor.

Today, we have various frameworks and programming languages. For instance, R is a famous vector-oriented programming language, meaning that the lowest unit is not an integer or a float but a vector.

In the same way, TensorFlow works with tensors, which not only optimizes CPU usage but also allows us to employ GPUs to make calculations. In 2016, Google developed Tensor Processing Units (TPUs)—processors that consider a tensor a building block for a calculation, not 0s and 1s, as does a CPU. This makes calculations exponentially faster.

If you wish to grow your machine and deep learning knowledge, tensors are a great addition to your toolkit. You can learn more in our Deep Learning with TensorFlow 2 and Convolutional Neural Networks with TensorFlow in Python courses.

Next Steps

Enroll in our Data Scientist Career Track and enhance your domain knowledge. Start with the fundamentals and dive right into our Statistics, Mathematics, and Probability courses. You can learn at your own pace in a step-by-step manner, sharpening your skills in SQL, Python, and TensorFlow. Sign up today and begin your data scientist career journey!

FAQs

What is a tensor in simple terms?

A tensor is a mathematical object that generalizes scalars, vectors, and matrices. Just like a multidimensional array, it can work with complex data. Tensors are widely popular in many fields (mathematics, physics, machine learning, computer science) to manipulate data with multiple dimensions. What’s unique about tensors is that they allow us to analyze and model relationships between datasets while still considering their multidimensional nature.

What is an example of a tensor?

A grayscale image is a tensor built on a grid of pixels, each containing a single value determining the brightness at that pixel location. The grid of values represents a tensor with two dimensions: height and width. You can use the coordinates of the pixels to index the tensor, where an individual element will identify the intensity value of a specific pixel.

Are tensors just matrices?

No. A tensor is an umbrella term that can hold and manage different dimensions. A matrix represents a type of tensor with two dimensions. So, while all matrices are tensors, not all tensors are matrices.

What are tensors in deep learning?

Tensors are the primary data structures you can employ to manipulate data within deep learning models. They are multidimensional arrays that store numerical data. The arrays can represent such different data types as text, audio, images, and numerical values. Because tensors enable parallel processing and promote efficient computation, they are vital for training and inference in deep neural networks.

Is a tensor a 3D matrix?

A tensor is more than just a 3D matrix. Tensors go beyond the functions of matrices with their algebraic properties because they’re more than just multidimensional arrays. A tensor may possess any number of dimensions, including three layers, like a 3D matrix. But that’s not all tensors can handle. They have broad practical applications in various fields because of their powerful ability to manipulate complex data structures.