Previewing Machine Learning#

Under the course code CSCI 4050U, I will be teaching an introductory course on Machine Learning. This is the fastest evolving field in Computer Science, and so I feel extra motivated to keep the topics up to date. Even though the course is at an introductory level, every time I offer the course, I still find myself redesigning about 50% of the content so that it feels fresh and modern. This year will be no exception.

Math used by neural networks#

Machine learning is really about mathematical modeling using a rich set of functions and their compositions. To truly appreciate the principles behind the success of machine learning and neural networks, studenst will be presented a short survey of the mathematics used by machine learning algorithms. To that end, we will present the following topics:

  1. Linear algebra in Euclidean vector space, particularly the geometric interpretation of matrix algebra.

  2. Calculus, functions, fields and their gradients.

  3. Optimization without and with constraints.

  4. Probability and statistics

Basics of neural networks#

We will build neural networks using compositions of simple functions. We show that using the developed mathematical tools, we can perform supervised learning very quickly, and achieve surprisingly good performance for tasks such as classification.

  • Linear networks

  • Nonlinear activation

  • Backpropagation for training deep networks

  • Multi-layer perceptron

On performance of machine learning systems#

Before building more complex neural networks, we will talk about performance considerations of machine learning systems. It’s really important for a designer to have a good understanding on the best practices in assessing a machine learning system. We will be discussing:

  1. Training versus testing

  2. Precision, recall and potential bias

  3. Regulation and overfitting

  4. Cross validation

  5. Model complexity

Neural network architectures#

We will introduce a range of neural network architectures that have been shown to be effective for their respective domains.

  • Convolutional networks for computer vision

  • Residual networks

  • Recurrent networks for sequence learning

  • Embedding layers

Novel networks#

Neural network based learning has shown the ability to gain unexpectedly deep semantic understanding in order to perform classification tasks. Often, we train networks to perform classification only for the purpose of the more powerful semantic understanding that comes as a side effect of the main task.

  • Word embedding vectors and pretrained NLP models

  • Image style transfer using pretrained computer vision models

  • Generative networks: autoencoder

  • Generative networks: adversarial networks