Machine Learning
0 Introduction
1 Tensors
1 Construction
2 Transform
3 Operations
4 Products
2 Calculus In Vector Spaces
1 Vector Space
2 Vector Calculus
3 Model Fitting
Line Fitting (Video)
Binary Classification (Video)
4 Tensorflow Api
1 Basics Of Tensorflow
2 Line Fitting (Video)
3 Polynomial Line Fitting (Video)
5 Keras Api
1 Basics Of Keras
2 Binary Classifier In Keras
6 Machine Learning
1 Overview
2 Training Testing
3 Overfitting
7 Convolutional Networks
1 Filtering
2 Convet
8 Sequence Learning
1 Text Encoding And Rnn
2 Keras Imdb
3 Lstm
{:check ["true"]}

Index

3 Tensor Operations

Tensor Operations

In [1]:
import numpy as np

Element-wise operations

Element wise functions

NumPy comes with a vast collection of functions. They are to operate on individual elements.

These functions are implemented at a very low level in C or Fortran, and are much more efficient than an equivalenet implementation using element-by-element enumeration.

In [2]:
# Let x be a simple matrix

x = np.arange(12).reshape((3,4))
x
Out[2]:
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
In [3]:
# We can operator on x element-by-element.

np.power(x, 2)
Out[3]:
array([[  0,   1,   4,   9],
       [ 16,  25,  36,  49],
       [ 64,  81, 100, 121]])
In [5]:
np.sin(x/11*2*np.pi).round(decimals=4)
Out[5]:
array([[ 0.    ,  0.5406,  0.9096,  0.9898],
       [ 0.7557,  0.2817, -0.2817, -0.7557],
       [-0.9898, -0.9096, -0.5406, -0.    ]])

Binary Operations and Broadcasting

Binary operations requires compatible shapes

Suppose we are computing element-wise addition of two tensors.

x + y

This works if x.shape == y.shape.

In [5]:
x = np.random.randint(100, size=(3, 6))
x
Out[5]:
array([[49, 32, 49, 45, 51, 35],
       [62, 99, 84, 60, 94, 55],
       [56, 23, 63, 17,  0, 69]])
In [6]:
y = np.random.randint(100, size=(6, 3))
y
Out[6]:
array([[74, 11, 41],
       [13, 78, 20],
       [60, 58, 37],
       [61, 55, 38],
       [96, 17, 58],
       [74,  2, 72]])

At this point, we know that:

  • x.shape == (6,3)
  • y.shape == (3,6)

So, x + y will fail.

In [7]:
try:
    x + y
except Exception as e:
    print(e)

operands could not be broadcast together with shapes (3,6) (6,3)
In [8]:
y = np.random.randint(100, size=(3, 6))
y
Out[8]:
array([[99, 29, 74, 82, 33, 97],
       [95,  8, 82, 66, 29, 62],
       [86, 83,  4, 85, 82, 38]])

At this point, we know that:

  • x.shape == (6,3)
  • y.shape == (6,3)

So, x + y will succeed.

In [9]:
x + y
Out[9]:
array([[148,  61, 123, 127,  84, 132],
       [157, 107, 166, 126, 123, 117],
       [142, 106,  67, 102,  82, 107]])

Binary operators allow axes mismatch

Suppose

  • x has shape $(n_1, n_2, n_3, n_4)$, and
  • y has shape $(n_1, 1, n_3, 1)$.

NumPy allows binary element-wise operations between them using a mechanism called broadcasting. The tensor y is transformed structurally by repeating axis-2 and axis-4 to match the shape of x.

It's functionally equivalent to using numpy.repeat to restructure y to match the shape of x.

In [10]:
x = np.random.randint(100, size=(3, 4, 2, 3))
x.shape
Out[10]:
(3, 4, 2, 3)
In [11]:
y = np.random.randint(100, size=(3, 1, 2, 1))
y
Out[11]:
array([[[[19],
         [38]]],


       [[[99],
         [20]]],


       [[[31],
         [46]]]])
In [12]:
y_ = np.repeat(y, 4, axis=1)
y_.shape
Out[12]:
(3, 4, 2, 1)
In [13]:
y_ = y
y_ = np.repeat(y_, 4, axis=1)
y_ = np.repeat(y_, 3, axis=3)
y_.shape
Out[13]:
(3, 4, 2, 3)
In [14]:
(x + y_).shape
Out[14]:
(3, 4, 2, 3)

We can just use broadcasting

In [15]:
(x + y).shape
Out[15]:
(3, 4, 2, 3)

Binary operators allows different number of axes

Suppose:

  • x has shape $(n_1, n_2, n_3, n_4)$, and
  • y has shape $(n_3, n_4)$.

NumPy allows binary element-wise operation by a series of transformations:

  1. Pad y.shape with 1s to match the rank of x.

    $$\mathrm{y.shape}\to(1, 1, n_3, n_4)$$

  2. Use broadcasing to match the shape of x:

    $$\mathrm{y.shape}\to(n_1, n_2, n_3, n_4)$$

In [16]:
x.shape
Out[16]:
(3, 4, 2, 3)
In [17]:
y = np.random.randint(100, size=(2, 3))
y
Out[17]:
array([[ 0, 40, 50],
       [43, 57, 47]])
In [19]:
(x + y).shape
Out[19]:
(3, 4, 2, 3)

Version 01.30.2021. - Copyright 2021, Ken Pu