{:title ["Binary Classification (video)"], :check ["true"]}
Index
Binary Classification
Classifier
Binary Classification¶
In [1]:
import numpy as np
import matplotlib.pyplot as pl
Let's first load the data.
In [5]:
data = np.loadtxt('./linear_classifier_data.csv',
skiprows=1,
delimiter=',')
data.shape
Out[5]:
In [10]:
data[0:10,:]
Out[10]:
In [13]:
# Plot the two classes
x_data = data[:, :2]
y_data = data[:, -1]
I0 = y_data == 0
I1 = y_data == 1
pl.plot(x_data[I0,0], x_data[I0, 1], '.', color='red')
pl.plot(x_data[I1,0], x_data[I1, 1], '.', color='blue');
Define the necessary functions:
sigmoid function:
sigmoidthe model:
f
In [14]:
def sigmoid(x):
return 1/(1 + np.exp(-x))
In [15]:
def f(x, w, b):
return sigmoid(x @ w + b)
We can use f to perform classification.
Assume some (bad choice) of $(w, b)$.
Let $z = f(x, w, b)$, and classify the input $x$ according to the rules:
- if $z < 0.5$, then $y = 0$.
- if $z \geq 0.5$, then $y = 1$.
In [16]:
# Purely randomly arbitrary choices of w, b
w = np.array([1, 1])
b = 3.0
z = f(x_data, w, b)
In [17]:
# convert z to y_pred
n = 2000
y_pred = np.zeros(n)
y_pred[z >= 0.5] = 1
In [20]:
np.sum(y_data != y_pred) / n
Out[20]:
Visualization
Plotting the classification of this randomly obtained classifier.
In [21]:
I0 = z < 0.5
I1 = z >= 0.5
pl.plot(x_data[I0, 0], x_data[I0, 1], '.', color='red')
pl.plot(x_data[I1, 0], x_data[I1, 1], '.', color='blue');
Loss function¶
The loss function is the binary cross entropy between
the output $z$ and the true outputs y_data.
In [22]:
def loss(w, b):
z = f(x_data, w, b)
L = -(y_data * np.log(z) + (1-y_data)*np.log(1-z))
return np.sum(L)
In [23]:
loss(w, b)
Out[23]:
The gradient of the loss function:
- $\partial L/\partial w_i$ for $i=1,2$.
- $\partial L/\partial b$
In [24]:
def loss_grad(w, b):
z = f(x_data, w, b)
grad_w = x_data.T @ (z - y_data)
grad_b = np.sum(z-y_data)
return grad_w, grad_b
Gradient Descent For Training¶
Implement the training loop
In [29]:
def train(w0, b0, alpha, epochs):
w = np.array(w0, dtype=float)
b = b0
for i in range(epochs):
w_grad, b_grad = loss_grad(w, b)
w -= alpha * w_grad
b -= alpha * b_grad
L = loss(w, b)
if (i % (epochs//10)) == 0:
print(L)
return w, b
Train the model using x_data and y_data.
In [33]:
w, b = train([1, 1], 3, 1e-5, 10000)
Use the parameters from the training.
In [34]:
z = f(x_data, w, b)
I0 = z < 0.5
I1 = z >= 0.5
pl.plot(x_data[I0, 0], x_data[I0, 1], '.', color='red')
pl.plot(x_data[I1, 0], x_data[I1, 1], '.', color='blue');
Visualization of the separation boundary¶
Let's look at the separation boundary used by the classifier.
Steps to take:
- Generate a 2D meshgrid: $100\times 100$ points.
- For each of the $10,000$ points, we will compute the
zvalue based on the model.
- Plot the contour of
zw.r.t. to the meshgrid.
In [35]:
# Generate the 2D meshgrid
xs = np.linspace(-10, 10, 100)
ys = np.linspace(-10, 10, 100)
xx, yy = np.meshgrid(xs, ys)
In [36]:
# Compute the z-values at each grid point.
z = np.zeros((100, 100))
for i in range(100):
for j in range(100):
x = xx[i][j]
y = yy[i][j]
z[i, j] = f(np.array([x, y]), w, b)
In [42]:
# Plot z as a contour plot
pl.figure(figsize=(6,6))
pl.contourf(xx, yy, z, levels=[0., 0.5, 1.])
z = f(x_data, w, b)
I0 = z < 0.5
I1 = z >= 0.5
pl.plot(x_data[I0, 0], x_data[I0, 1], '.', color='red')
pl.plot(x_data[I1, 0], x_data[I1, 1], '.', color='blue');
In [ ]: