Sebastian Höfer's Blog

Yet Another Machine Learning 101

Thu, 12 May 2016 00:00:00 +0000

This post is a somewhat short recap of machine learning in general. I wrote it as lecture notes for a tutorial that I gave in one of the robotics courses at TU Berlin.

It will probably be difficult to understand it in detail if you are a beginner. For beginners, I recommend reading my blog on intuitive machine intelligence.

The post is also available as an ipython notebook which allows you to play around with the python examples.

General setting

In general, we distinguish three general paradigms in ML

Supervised Learning
Unsupervised Learning
Reinforcement Learning

Additionally, there is a variety of other paradigms, which are not going to be covered here, such as semi-supervised learning and learning from side information which will not be covered here.

In all types of machine learning the goal is to learn a function to predict some output from some input data:

\[f: X \rightarrow Y\]

The different paradigms differ in

the goal, i.e. what kind of data $X$ and $Y$ are
the training data available to learn $f$

We will now review the different paradigms and give examples, however, mostly focusing on supervised learning.

Supervised Learning

Supervised learning is by far the most popular paradigm. In this setting, we are given examples from $X$ and $Y$, i.e. $N$ pairs ${ x^{(i)}, y^{(i)} }_{i=1\ldots N }$. The data $x^{(i)}$ are called the input data and $y^{(i)}$ the labels. Together, they are called the training data.

Any ML method usually consists of (at least) three ingredients.

A representation of $f$, i.e. whether it is a linear function, polynomial, neural network, or non-parametric, etc.
An appropriate loss function $\mathcal{L}$,
which is minimized using an optimization method.

The loss function for supervised learning looks as follows:

\[f = \textrm{argmin}_{f}\ \mathcal{L} ( f, \{ x^{(i)}, y^{(i)} \}_{i=1\ldots N })\]

A loss function can be thought of as an assessment of how good $f$ fits the training data. If its value is high, it means that $f(x^{(i)})$, i.e. our prediction of the label for $x^{(i)}$ given current function $f$, is very different from the known true value $y^{(i)}$. If $\mathcal{L} = 0$, it means the $f$ perfectly fits the data.

An optimization method can be thought of as a method that is given data and a loss function and automatically tries to find the best function minimizing the loss. There is a wide variety of optimization methods, and they mostly differ by the properties of $f$ they exploit (e.g. if $f$ is differentiable, we can compute its derivative, set it to zero, and go step by step in the direction of steepest descent; this is called gradient descent optimization, see below). We only look at some very basic optimization methods, but it is of course the success in learning a function greatly depends on the optimization used.

The representation of $f$ also drastically influences its learnability. The easiest and best-understood representations are linear functions, but also neural networks (which are some sort of nonlinear functions) are very common nowadays.

Usually, as a practitioner, you don’t have to worry about making choices these three things. An ML method usually determines the loss, optimization and function representation, and is tuned in such a way that all of them work together nicely.

Overfitting

We said that the goal of supervised learning is to learn a function $f$ from training data, using the loss $\mathcal{L}$. However, it is important that a perfect loss, $\mathcal{L}=0$, for your training does not necessarily solve the problem – the actual task of supervised learning is to learn an $f$ that generalizes to unseen examples $x^{(j)}$. And actually, fitting the training data is very easy: just memorize all of them by heart! This will trivially give us $\mathcal{L}=0$ for our training data. But as we will see, that might mean that for unseen data we will actually make blatantly wrong predictions.

To repeat, the core problem of any machine learning method is to learn an $f$ that generalizes to unseen data. If $f$ only works well on the training data, but not on unseen data, we say $f$ overfits the data.

To ensure that our function does not overfit, we divide the data into three sets:

Training data
The data used for actually learning/fitting the function
Validation data
Additional data, not used for learning the function, but only used to compute the loss for our current estimate of the function. It is also used to tune the hyperparameters of your method, or decide which ML method to use.
Test data
Another additional data set, neither used for training nor for validation. You can think of it as being locked in some secret drawer, and it is only released and used to evaluate $f$ once you are sure that you have properly learned $f$.

Example: Linear regression

Let us consider a simple example for supervised learning: linear regression.

Assume we want to predict the price of apartments in Berlin, given their size of square meters of the apartment. Here is some fictitious data:

$X$ (square meters)	$Y$ (monthly rent in Euro)
40	500
65	620
80	855
81	910
100	1100
120	1250

Let’s plot the data

In [1]:

%pylab inline
set_printoptions(suppress=True)

X = array([40, 65, 80, 81, 100, 120]).reshape((-1, 1))
Y = array([500, 620, 855, 910, 1100, 1250])

scatter(X, Y);

Populating the interactive namespace from numpy and matplotlib

The idea is linear regression is to fit a linear function, i.e. a line of the form $f(x) = w_1 x + w_0$ to the data. To learn such a line we define as loss the mean-squared error which penalizes large deviations of $f(x)$ from the known $y$:

\[\mathcal{L}_\textrm{MSE} = \frac{1}{2N} \sum_{i=0}^N ( f(x^{(i)}) - y^{(i)})^2\\ = \frac{1}{2N} \sum_{i=0}^N ( w_1 x^{(i)} + w_0 - y^{(i)})^2\]

Maybe you are a bit confused about the fraction $\frac{1}{2N}$. The reason we use it, because if we compute the derivative $\mathcal{L}_\textrm{MSE}$, the square within the sum cancels out this fraction; and the $\frac{1}{N}$ normalizes for the number of training examples.

Note that usually $x$ is multi-dimensional, i.e. a vector $\mathbf{x}$. Then $w_1$ becomes a weight vector $\mathbf{w}$. We can do another trick to get rid of $w_0$ (called the bias or intercept term) by appending a $1$ to $\mathbf{x}$, and extending $\mathbf{w}$ by one dimension. This facilitates notation a bit, as it allows us to write the loss using vector notation:

\[\mathcal{L}_\textrm{MSE} = \frac{1}{2N} \sum_{i=0}^N ( \mathbf{w}^T \mathbf{x}^{(i)} - y^{(i)})^2\]

Gradient descent

How to optimize it? One way is to compute the partial derivative wrt. each $j$-th element of the weight (the so-called gradient), and change the updates:

\[\nabla \mathcal{L}_\textrm{MSE} = \frac{\partial \mathcal{L}_\textrm{MSE}}{\partial {w}_j} = \frac{1}{2N} \sum_{i=0}^N \frac{\partial}{\partial {w}_j} ( \mathbf{w}^T \mathbf{x}^{(i)} - y^{(i)})^2 \\ = \frac{1}{2N} \sum_{i=0}^N 2 w_j^{(i)} (w_j x_j^{(i)} - y^{(j)})\\ = \frac{1}{N} \sum_{i=0}^N w_j (f(x_j^{(i)}) - y^{(j)})\]

The gradient descent algorithm (for linear regression with MSE) then goes as follows:

Initialize the weights $\mathbf{w}$ randomly
Update each $j$-th entry in the weight vector by following the negative gradient:
$w_j^\textrm{new} = w_j - \alpha \frac{1}{N} \sum_{i=0}^N w_j (f(x_j^{(i)}) - y^{(j)})$

Here, $\alpha$ denotes the step size. It has to small enough, otherwise we might overshoot and miss the minimum.

If we image the loss function to be a “bowl” and every $\mathbf{w}$ to be a point in this bowl, the gradient points towards the bottom of this bowl, and thus the minimum of the loss. The update rule then takes small steps towards this minimum. This is a cool visualization which is also accompanied by a Matlab script:

Gradient descent is a very important technique, very popular especially for training neural networks (see below).

Ordinary least squares

In the linear regression case, however, there is a more direct solution. If we consider $\mathbf{w}$ to be a matrix rather than a vector, we can write the loss in the following way:

\[\mathcal{L}_\textrm{MSE} = \frac{1}{2} (\mathbf{X}\mathbf{w} - \mathbf{y})^T (\mathbf{X}\mathbf{w} - \mathbf{y})\]

where $\mathbf{X}$ contains in each $i$-th row on training example $\mathbf{x}^{(i)}$, and $\mathbf{y}$ in each $i$-th row a label $y^{(i)}$.

Now we can compute the derivative of $\mathcal{L}_\textrm{MSE}$, set it to 0, and solve for $\mathbf{w}$. As you can check yourself, the derivative is given by: $\frac{d \mathcal{L}_\textrm{MSE}}{d \mathbf{w}} = \mathbf{X}\mathbf{w} - \mathbf{y} \\ \mathbf{X}\mathbf{w} - \mathbf{y} = 0\\ \mathbf{X}\mathbf{w} = \mathbf{y}$

To now solve it for $\mathbf{w}$, we need to invert $\mathbf{X}$ – which is usually not possible because it is not square in the general case. But we can apply a trick, namely use the pseudo-inverse:

\[\mathbf{X}\mathbf{w} = \mathbf{y}\\ \mathbf{X}^T\mathbf{X}\mathbf{w} = \mathbf{X}^T\mathbf{y}\\ \mathbf{w} = (\mathbf{X}^T\mathbf{X})^{-1} \mathbf{X}^T\mathbf{y}\\\]

where the pseudo-inverse is given by $(\mathbf{X}^T\mathbf{X})^{-1} \mathbf{X}^T$.

Computational example

Let’s now implement this in python.

In [2]:

# transpose training data and append 1
Xhat = np.hstack([X, np.ones((X.shape[0],1.))])
w = np.linalg.inv(Xhat.T.dot(Xhat)).dot(Xhat.T).dot(Y)
w

array([ 10.04581152,  58.78926702])

In [3]:

scatter(X, Y)
plot(X, Xhat.dot(w));

Luckily, there are libraries that do all that for us. One of the most popular ML libraries in python is scikit learn. We can easily verify that it computes the same function:

(we see that sklearn automatically adds a bias, stored in the variable “intercept_”)

In [4]:

from sklearn.linear_model import LinearRegression
lr = LinearRegression()
lr.fit(X, Y)

X_rng = np.linspace(40, 120, 100).reshape((-1,1))

scatter(X, Y)
plot(X_rng, lr.predict(X_rng));
lr.coef_, lr.intercept_

(array([ 10.04581152]), 58.789267015706741)

We see that the weights and our prediction (the blue line) are identical. And you see that it fits the data Ok, but not perfectly. Still, it looks like a reasonable guess. Most importantly, it also makes a prediction for inputs $x$ for which we did not have any training data.

Overfitting

What if we don’t want to fit a line, but some more complex model, e.g. a polynomial? This is easily done by augmenting our input (also called feature expansion) by different powers of the input:

$f(\tilde{\mathbf{x}}) = [\mathbf{x}, \mathbf{x}^2, \mathbf{x}^3, … ]$

Let’s try that out:

In [5]:

def polynomial_feature_expansion(X):
    return np.hstack([X, X**2, X**3, X**4, X**5,])

X_rng = np.linspace(20, 140, 100).reshape((-1,1))

lr = LinearRegression()
lr.fit(polynomial_feature_expansion(X), Y)

scatter(X, Y)
plot(X_rng, lr.predict(polynomial_feature_expansion(X_rng)));
ylim(200, 2000)
lr.coef_, lr.intercept_

(array([ 14413.72962393,   -394.63014472,      5.20499616,     -0.03319069,
             0.00008213]), -201202.688457913)

We see that the training data is fitted almost perfectly; but the function hallucinates weird values inbetween and before/after the training data! Also, the weights are actually very high. This is a classical example of overfitting: we used a function that is too “powerful”, as it has many more parameters than the linear model. There are different ways to remedy this problem:

More training data!
Restricting the function to a simpler one (e.g. less parameters)
(The danger can be underfitting, i.e. choosing a too simple model)
Model selection, e.g. using cross-validation
Incorporating prior knowledge about the problem, e.g. by feature engineering
Regularization

At this point we will only talk about regularization which a special way to force a powerful function not to overfit. The idea is to put additional terms into the loss function. A popular regularization is L2 which puts a L2-norm penalty on the weights, i.e. $||\mathbf{w}||^2$ into the loss. The optimizer then has to make sure not only to fulfill the initial loss, e.g. the mean-squared error, but also the regularization.

A linear regression with L2 regularization is called ridge regression. The loss then becomes: $\mathcal{L}_\textrm{Ridge} = \frac{1}{2} (\mathbf{X}\mathbf{w} - \mathbf{y})^T (\mathbf{X}\mathbf{w} - \mathbf{y}) + \alpha ||\mathbf{w}||^2$

where $\alpha$ weights the influence of the regularizer.

Ridge regression is also implemented in scikit learn:

In [6]:

from sklearn.linear_model import Ridge

lr = Ridge(alpha=20.)
lr.fit(polynomial_feature_expansion(X), Y)

scatter(X, Y)
plot(X_rng, lr.predict(polynomial_feature_expansion(X_rng)));
ylim(200, 2000)
lr.coef_, lr.intercept_

(array([-0.05208327, -2.0080791 ,  0.05283245, -0.00047523,  0.00000145]),
 1401.9265643035942)

We see, that the weights are lower and the values inbetween are much smoother; but still for values $>120$ and $<40$ the linear model reflects our intuition about the real $f$ better. Therefore, we might choose to increase the regularization, collect more data or choose a different function parametrization, e.g. a polynomial with lower degree.

Regression vs. Classification

Before we talk about more sophisticated supervised learning methods, we should clarify the terms regression and classification. The only difference between these two concepts is whether $y$ is discrete or continuous. In the previous example we used regression, i.e. we treated the price as a continuous variable. In classification, we are usually given a discrete, finite set of classes, and we are only interested in predicting the class of a new input. The only thing that changes because of this is the loss. We won’t bother about these loss functions now, but in case you are interested, common choices are the categorical cross-entropy loss or the hinge loss.

But watch out, the terminology is not always fully consistent: training a linear model with a categorical cross-entropy loss is called logistic regression – although it is actually a classification method!

(Deep) Neural Networks

Currently, they are probably the most popular approach in supervised learning. The idea is to compose the function $f$ of small slightly nonlinear functions (neurons) and connect them. This small nonlinear functions are called neurons, and together they form a neural network that can be visualized as follows:

The picture (image taken from wikipedia) depicts a network with an input layer ($=\mathbf{x}$), an output (should be equal to our labels $y$) and a hidden layer. This hidden layer can learn some representation of $\mathbf{x}$ that is suitable for predicting $y$. For the record, this network structure is also sometimes called multi-layer perceptron.

What does a (non-input and non-output) neuron look like? In fact, a neuron basically multiplies a linear weight vector with its input (sounds exactly like linear regression, right?) and then applies some nonlinearity on the output of this operation (that’s different from linear regression). Let’s make this formal; a neuron $h_i$ (in the hidden layer), given input $\mathbf{z}$ (in the network above $\mathbf{z} = \mathbf{x}$), computes its output as follows:

\[h_i(\mathbf{z}) = \sigma(\mathbf{w}_{h_i}^T\mathbf{z})\]

where $\sigma$ denotes some nonlinear activation function, often something like the sigmoid-function:

$\sigma(t) = \frac{1}{1 + e^{-t}}$,

although the hyperbolic tangent and rectifiers are more en vogue in state-of-the art neural networks.

A hidden layer $f_h$ composed of $H$ neurons then computes its output as follows:

\[f_h(\mathbf{z}) = \sigma(\mathbf{W}_{h} \mathbf{z}),\]

where ${\mathbf{W} _ h}$ is a $\mathrm{dim}(\mathbf{z}) \times k$ matrix composed of the stacked (transposed) weight vectors $\mathbf{w}_{h_i}^T, i=1 \ldots H$, and the activation function $\sigma$ is applied separately to each output dimension of $\mathbf{W}_H$.

So what are deep networks? The idea is to stack multiple hidden layers – the more hidden layers there are, the “deeper” the network is. Mathematically, it is just a repeated application of multiplying a linear weight with the output of the previous layer, then computing the activation, passing it to the next layer, and so on. The advantage of these networks is, that they can learn more sophisticated representations of the input data, and thus solve more challenging learning problems.

Finally, we have to say how to train neural networks. We can use the same loss functions as for linear regression (or classification, of course), and apply gradient descent. However, if we have multiple layers, we need to apply some tricks to adapt gradient descent. The first trick is backpropagation; it basically says that to train multiple layers, we apply gradient descent to each layer separately, and pass the loss backwards through the network layer by layer until we reach the beginning. For this to work well, we must apply also some additional tricks, e.g. setting the initial values of all weights appropriately and so on. But in principle, training a neural network is not much different from performing a linear or logistic regression.

Of course there is a lot more to know abouot deep learning, and if you want a more in-depth treatment of this topic, check out this recent book.

Unsupervised Learning

Unsupervised differs from supervised learning that we are only given ${x^{(i)}}_{i=1 \ldots N}$, an no labels. This obviously means that the loss functions we’ve seen so far will not work. Instead the loss functions impose some “statistical” constraints on $y$. A good example is Principal component analysis (PCA): here we want to learn a low-dimensional variant of $x$ – however, which still contains roughly the same “information” as the original $x$. The question is how to quantify “information”. This very complicated and highly depends on the task; but PCA takes a pragmatic stance by defining information as high variance (in a statistical sense). Therefore, the loss for PCA is roughly equivalent to:

\[{\mathcal{L}}_\textrm{PCA} \approx -\textrm{Var}(f(\mathbf{x}))\]

where $f$ is indeed a linear function as used above. The $\approx$ sign reflects that formally the loss is a bit different - we need some additional constraints to make this problem well-defined, i.e. not find infinitely large weights. I will not go into details here, but you should understand that one can formulate learning objectives without any supervised signal, just by formulating some desired properties of the result of $f$ in the loss function.

Note that PCA is somewhat the “regression” variant of unsupervised learning. There are also methods that map data into discrete representations, e.g. in clustering. The most popular and yet simplest method is probably k-means.

Also note that unsupervised learning has somewhat different applications than supervised learning. Often it is used for pre-processing the input data, in order to then feed its output to a supervised learning method. Another important application is exploratory data analysis, i.e. studying and finding patterns in your data.

Reinforcement Learning

In reinforcement learning our outputs $Y$ are actions that an agent should take, and our input X is the state. Therefore, we also rename $X$ and $Y$: the state is denoted by $\mathbf{s}$, the actions by $\mathbf{a}$, and the function we want to learn is called policy $\pi$ (instead of $f$):

\[\pi(\mathbf{s}) = \mathbf{a}\]

A crucial difference to supervised learning is that we do not know the correct actions $\mathbf{a}$. Rather, we only get a reward signal $r(\mathbf{s}, \mathbf{a})$ for every action we take (in a certain state). The reward is a real number that is high if the action was good, and low if it was bad.

Obviously, this problem is much harder as learning becomes indirect – you need to figure out the right action only from getting a reward. The biggest problem is that rewards are usually sparse and delayed; that means the agent might receive a positive reward only after it has executed a set of actions. However, it might be that actually the first action in this sequence was the most important one.

There is a wide variety of different techniques, such as policy search, value-based methods and model-based reinforcement learning to tackle this problem. We will look at these techniques at this point, but it is important that you at least understand the general setting of reinforcement learning, and its difference to supervised learning (and unsupervised learning).

The Curse of Dimensionality

Sun, 03 Jan 2016 13:45:00 +0000

In the last post we have looked at one of the big problems of machine learning: when we want to learn functions from data, we have to fight overfitting. In this post we will look at another archenemy of learning: dimensionality.

Parameters

Let’s briefly recap our stock price prediction example. In an earlier post we used random search to find the parameters of a line that explains the training data examples well. The algorithm learned two numbers, namely the parameters p₁ and p₂:

Stock price = f(Revenue) = p₁ * Revenue + p₂,

where visually, p₁ changes the slope of the line and p₂ the shift along the y-axis of a line.

The random search learned the numbers p₁=0.00015 and p₂=58:

This line is not very far away from the parameters of the “true” function p₁^true=0.00013 and p₂^true=70. Interestingly, for the learned function the slope p₁ is a bit higher, but this is “compensated” by a lower shift p₂. Intuitively, this makes sense: the higher the slope, the more we need to shift the line downwards in order to approximately hit the training examples. We will see in a second why this observation is important.

Adding dimensions

Now, I would like to reason about the influence of the number of parameters of a function on the difficulty of learning that function. You can think about the parameters of the line, but in fact I will formulate it in a general way.

For the sake of the argument let’s assume that we do not consider all possible numbers as possible parameter values, but that we restrict ourselves a fixed list of numbers (in mathematics that’s called discretization), and we make this list finite. To make it really simple, we will only use the numbers from 1 to 10.

Assume that we have a function that has only one parameter (for example, only the slope of a line), we immediately see that there are 10 possible values that the parameter can take. We visualize each value of the parameter by a blue box:

What happens if we add a second parameter that can take 10 values? Well, you might think that results in 10 values for parameter one and another 10 for parameter two, which makes 20. But unfortunately that’s wrong: we need to consider all possible combinations of the parameter values! The reason is that the parameters are usually dependent as we have seen in the line example: when changing parameter one (here the slope), we can still improve how well the line fits the data by changing parameter two (shift). Let’s visualize all possible combinations:

(1, 1)	(1, 2)	(1, 3)	(1, 4)	(1, 5)	(1, 6)	(1, 7)	(1, 8)	(1, 9)	(1, 10)
(2, 1)	(2, 2)	(2, 3)	(2, 4)	(2, 5)	(2, 6)	(2, 7)	(2, 8)	(2, 9)	(2, 10)
(3, 1)	(3, 2)	(3, 3)	(3, 4)	(3, 5)	(3, 6)	(3, 7)	(3, 8)	(3, 9)	(3, 10)
(4, 1)	(4, 2)	(4, 3)	(4, 4)	(4, 5)	(4, 6)	(4, 7)	(4, 8)	(4, 9)	(4, 10)
(5, 1)	(5, 2)	(5, 3)	(5, 4)	(5, 5)	(5, 6)	(5, 7)	(5, 8)	(5, 9)	(5, 10)
(6, 1)	(6, 2)	(6, 3)	(6, 4)	(6, 5)	(6, 6)	(6, 7)	(6, 8)	(6, 9)	(6, 10)
(7, 1)	(7, 2)	(7, 3)	(7, 4)	(7, 5)	(7, 6)	(7, 7)	(7, 8)	(7, 9)	(7, 10)
(8, 1)	(8, 2)	(8, 3)	(8, 4)	(8, 5)	(8, 6)	(8, 7)	(8, 8)	(8, 9)	(8, 10)
(9, 1)	(9, 2)	(9, 3)	(9, 4)	(9, 5)	(9, 6)	(9, 7)	(9, 8)	(9, 9)	(9, 10)
(10, 1)	(10, 2)	(10, 3)	(10, 4)	(10, 5)	(10, 6)	(10, 7)	(10, 8)	(10, 9)	(10, 10)

We see that we have 10 times 10, i.e. 100 possible parameter pairs. What happens if we add a third parameter? We get a cube with 10 times 10 times 10 equals 1000 parameters! And with 10 parameters each taking 10 values, we get 10.000.000.000 which are 10 billion different combinations of parameter values!

You probably see the formula behind our reasoning: If we have n values a parameter can take, and m parameters, we end up with n^m possible parameter value assignments. We say that the number of parameter values grows exponentially.

Now how big of a problem is it? Well, it is very big indeed, which is why this problem is called the curse of dimensionality. The problem is that data are usually high-dimensional, and that each parameter usually has significantly more than 10 possible values.

For example, the tiny pictures we played around with in an earlier post had 27x35, that is 945 pixels. If we were to learn a function that has a parameter for every pixel and again every parameter can only take 1 out of 10 values we would still end up with 10⁹⁴⁵ parameter values - this is a number consisting of a 1 with 945 trailing zeros, and it is several orders of magnitudes higher than the number of particles in the entire universe! We will never be able to try out even a tiny fraction of all possible parameter values.

So we see that the curse of dimensionality forces us to find smarter ways of finding the parameters of functions. We will save this for later articles.

Hughes effect

Irrespective of the way how smart we go about searching for parameters, high dimensionality has another very problematic (and somewhat unintuitive) implication, namely for classification. This problem is also known as the Hughes effect. I will only sketch the idea very briefly but you find a more elaborate explanation with nice illustrative figures in this blog article.

The problem is stated as follows: recall that in classification, we aim to find a function that discriminates between two or more categories of input data. We presented a way of doing so, namely by finding hyperplanes in space that separate the categories. So how much easier or simpler does it get to find a hyperplane if the dimensionality of the data is higher?

Here comes the problem: the more dimensions the data has, the easier it is to find a hyperplane separating categories in the training data; but at the same time, the harder it gets to also perform well on the unseen data (for example, test data). The reason is that because we have more dimensions that we can choose from to lay the hyperplane through, we are much more prone to overfitting. Thus, having higher-dimensional data has the same effect as allowing our function to have more “wrinkles”. And the less training data we have, the less sure we are that we have identified the dimensions that really matter for discriminating between the categories.

Summary

In this post, we got to know another nemesis of data: high dimensionality. When looking at learning from the perspective of searching for the right parameters, each additional dimensionality means that we must search an exponential number of more parameter values. And in classification, every additional dimension makes it harder to find the right hyperplane to discriminate between the categories.

But another curse is already on its way; and it has to do with (no) free lunch.

TL;DR:

Curse of dimensionality refers to the dimensionality of the data and parameters to be learned
Searching for the right parameter values becomes exponentially harder with every dimension
Similarly, in classification every dimension makes us more prone to overfitting by choosing the wrong dimension as discriminatory

Learning Functions from Data: A Primer

Tue, 29 Dec 2015 23:00:00 +0000

In the introductory articles we have learned that data is a bunch of numbers encoding some information, and that data can be multi-dimensional which makes them live in vector spaces. We have also looked at the core competence of machine intelligence: applying functions to data. In this and the following posts we will look at the most powerful tool of machine intelligence: learning functions from data.

The roadmap is as follows. In this article, we will understand why learning functions from data is in principle rather straightforward. Indeed, at the end of this article we will have a developed a very simple learning method.

The next few posts will then bring us back down to earth and explain some fundamental problems that learning from data has, and discuss solutions to these problems. This will endow you with a powerful intuition of how learning from data works and what its limitations are.

Learning Functions from Data

In order to understand how learning from data works, let’s use as our running example the stock price prediction problem introduced earlier: given the annual revenue of a company, we want to predict the company’s stock price. We have learned that such a prediction is represented by a function. We discussed two representations of functions; first, tabular functions:

input: Annual Revenue in January (Euro)	output: Price (Euro)
40.000	122
50.000	135
60.000	148
80.000	174
100.000	200

These functions had the drawback that we could not extrapolate from them, that is we do not know from this table what the stock price would be if, for example, the company’s revenue was 120.000 Euros per year.

Therefore, we were interested in finding a different function representation, and I suggested to use linear functions, namely this one:

Stock price = f(Revenue) = 0.00013 * Revenue + 70

I have told you that since the function is a linear we can visualize it with a line:

More importantly, the regularity enabled us to predict the stock price given any annual revenue. Intuitively, when the the revenue gets higher, the stock price moderately increases.

How to extrapolate from data?

By drawing the line through the bunch of dots, I actually did the task that the computer is supposed to do: I looked at the data, and I found out that there is a linear relationship between stock price and annual revenue. But what I really would like to do is to give the machine the data from the table above and let it figure out on its own how to make stock price predictions into the future! How do we do that?

Automating exactly this process is the core of machine learning. In machine learning, you are given a set of training examples - which correspond to the table given above (and are visualized by the blue dots). These training examples are a bunch of data that need to be in the form (input, output) where output is the true, known output for the given input. The computer should then spit out an appropriate function, namely the blue line. Luckily, the stock price training data are exactly in the form (input, output). So how to learn from them?

I will now show you the simplest learning procedure I could come with. And it is indeed very simple! Intuitively, the computer will do the following: we tell the compute to use some line which should relate input and output, and the computer will then fiddle around with this line until it comes as close as possible to as many training examples as possible. That’s it!

Problem solved?

Defining the learning problem

Well, almost. If we would ask our computer to do that, it would reply with some of the following questions:

What function should I find? How is it represented?
What does “as close as possible to as many training examples as possible” mean?
What does “fiddle around” mean?

So let’s give our eager computer some answers:

Start off with a single line. A line is represented by two basic parameters: how much it is shifted up or down (the 70 in the example above), and its slope (the 0.00013).
Calculate the difference of every training example to the value predicted by the function the computer guesses. When we sum all of these differences up we obtain a single number which we call the training error. The training error allows us to calculate how well any line fits the training examples.
Do random guessing; we randomly guess a line (more precisely, its parameters) and see how well it fulfils the criterion defined in 2.

That sounds like a plan, doesn’t it?

Let’s first get an intuition what the training error looks like for some randomly guessed function:

Here, the blue dots are the training examples, the red curve is the random guess, the dotted lines indicate the discrepancy between the guessed line and the training data, and the numbers next to the lines indicate how big the discrepancy is. By summing over all these differences, we obtain the red number at the bottom. This number is the training error - we want to get it as low as possible in order to find a highly predictive function.

So how does random guessing perform? The following video shows what this looks like for 100 random guesses. The green line is the one with the lowest training error so far. We see that it takes some guesses, but eventually we get a close match of the green line and the blue dots, the guess and the training data.

Congratulations: you have witnessed your first machine learning algorithm [1]!

Caveats

You might be a bit disappointed now. Random guessing? That’s machine intelligence? That doesn’t seem intelligent at all! Fair enough, that is the simplest thing you can do. And in practise, it won’t work very well. Yet, some of the most powerful algorithms do apply random guessing (which is officially called random search) but combine it with some “tricks” to make the guessing smarter. So random search is an important building block for creating learning algorithms, although it’s not very powerful when used on its own.

At this point, we can already think about why random search does not work so well on its own. Here are a couple of thoughts.

First of all, you don’t now how big you should go. Should you shift the line only between -1 to +1, or from -1000 to +1000? Should you try out both 1.1 and 1.10001? That means, in which granularity should you vary the parameters? In fact, in the animation above, I have set the bounds of the parameters very tightly around the real (but in the usual case unknown) parameters. Otherwise, the search would have taken much longer.

Second, we don’t learn from our mistakes. If I have guessed line 1 that got a training error of 1000. Next, I guessed line 2 with a training error of 800. Intuitively, I should try to find out what might line 2 better than line 1, and progress into that direction. Instead, random search will just look for a new line which might be much worse than both line 1 and line 2.

Can you think of more issues? [2]

Terminology

Before we close, a note on terminology. Some paragraphs ago, we had to specify three things for our learning algorithm to work. I want to quickly state how these things are called in machine intelligence.

Model, function representation (sometimes also hypothesis space): What kind of functions do I look for?
(In the example: single straight lines)
Objective or loss function [3]: Criterion for evaluating whether the learned function is good. Often this is the training error, a measure of how well the learned function accounts for the training data.
(In the example: distance between samples and function)
Learning algorithm: In which way to come up with functions (or parameters) in order to get a good value for the loss function.
(In the example: random search)

Moreover, if you learn from training data of the form (input, output), the approach is called supervised learning. The picture is that the pupil comes up with different inputs, and a supervisor gives the pupil a hint by telling her the right output, and the pupil tries to figure out the regularity (aka function) from these hints.

Finally, there are two general types of supervised learning, which depend on the type of function you want to learn. If you learn to map an input to a continuous number (as done in the stock price example), the learning task is called a regression task; in the last post we called this fitting a line through the data. In contrast, if you map the input to a category (as in the image classification example), it is called a classification task. We solve this class by finding a (hyper)plane or line that separates the data.

In the next post, we will critically assess what we have done, and find out that there is a fundamental issue in learning from data which we have neglected so far. It is coarsely related to the question: why did we actually use a straight line, and not something with the shape of a curve or a snake?

TL;DR:

Machine intelligence learns functions from data
Data are given as training examples, that is input-output pairs
This type of learning is called supervised learning
To allow a compute to learn, we have to define model, objective and a learning algorithm
Random guessing is a learning algorithm that works for very simple problems

Footnotes:

An algorithm is the machine intelligence people name for “recipe”. It is a list of steps that you have to execute in a certain order; it has to be detailed enough so that the machine has no doubt about what it has to do.
Think, for example, what happens if your function has more parameters, let’s say twice as many. How much harder does the learning problem then get?
Note the term loss function. The criterion itself is again formulated as a function - but this is not the function want to learn! It is an auxiliary function that takes as input three arguments: a set of (1) inputs and (2) outputs of all the training data and (3) the corresponding outputs of the learned function. It outputs a single number, assessing the quality of the learned function for the training data. In contrast, the learned function takes only one input, namely the one it requires to predict the output.

Functions as Data Translators

Mon, 28 Dec 2015 18:10:00 +0000

If you have read the previous posts carefully you should now be familiar with high-dimensional data. In this last introductory article we are now going to look at what machine intelligence people mean when they think about manipulating data: they apply functions to data.

We will learn what functions are and look at different examples. Once we know what they are, we can in the next post see how to learn them - which is essentially what learning machines do.

Mathematical Functions

Mathematicians use the term function rather differently from the common sense definition. In everyday language we talk about the function of things in the sense “what is the purpose of the thing” or “how does this thing work”. A function in mathematical sense is rather different. It can be best thought of as a translation from one type of data to another type. Or in other words, you give some input (data) to the function, and get some output (data):

(This figure and the first example are taken from the highly recommended Wikipedia page on functions)

Let me give you a couple of examples of these things called functions.

Object description functions

Let’s assume a couple of primitive shapes, such as triangles, squares, etc. all of different color. We can now define a function that, given a shape, outputs the color of the function:

This visualization shows how to map input data X (the shape) to some output, the shape’s color Y. All the arrows between X and Y are the defining elements of the function.

Note that for each object in X there is only one arrow pointing away from it. This is indeed a requirement because we want a function to be a unique mapping: for each input we get exactly one output. The inverse is not required, though: the square and the triangle have the same color, so the red color in Y has two arrows pointing to it. You also notice that some colors, namely blue and purple have no arrows pointing to it. All of these things are very common in functions.

So far we have only talked about functions conceptually but we have not stated how a function can automatically compute the output from the input. Let us therefore look at a simpler example to shed some light on this.

Stock price prediction function

Let us now re-introduce our simple example from the introductory post. We want to assess the stock price of some company. For the sake of simplicity, we will only use information about the annual revenue of the company (in Euro) and try to predict the stock price (also in Euro). We can describe this relationship in a big table:

Year	Annual Revenue in January (Euro)	Price (Euro)
2010	40.000	122
2011	50.000	135
2012	60.000	148
2013	80.000	174
2014	100.000	200

This table can be considered a function in at least three ways: either we use the year as input, we use the annual revenue as input, or we use the stock price as input. We choose the annual revenue as the input and the stock price as the output because we believe that the annual revenue is more predictive for the stock price than the year - if the company was founded 10 years earlier, we would still expect the relationship of annual revenue and stock price to be similar (unless there was something like a global crisis, but we ignore that for now).

So the input of our function is annual revenue and the output stock price. And “computing” this function is very simple: given a number for the annual revenue, we look up the row in the table containing this number and return the corresponding stock price.

Such tabular functions are common but they have a severe drawback: we cannot extrapolate - from this table alone we do not know how to get the value for the stock price for an annual revenue of 90.000 Euros or 200.000 Euros! However, the example data I have given exhibits a regularity (surprise, surprise!), namely that the stock price values in the right column are exactly 0.00013 times the annual revenue plus 70 (check this for yourself). This gives us a much more concise way of describing this function:

Stock price = 0.00013 * Revenue + 70

You see that the function gets as input the revenue, and gives as output the stock price. To make this even clearer, we usually denote the function by the symbol f and write:

f(Revenue) = 0.00013 * Revenue + 70

The parentheses behind the f contain the inputs of the function, sometimes called arguments.

Let us visualize this function by plotting a graph which has on one axis (the x-axis) the annual revenue and on the other (the y-axis) the stock price:

We see two interesting things here: first, we can now predict the stock price value from the annual revenue, no matter which revenue. Secondly, the relationship between revenue and stock price in our example turns out to be a line. The fact that such relationships / functions can be drawn by a line results in them being called linear functions. Linear functions are amongst the most important types of relationships in mathematics - and actually they are one of the few that mathematics can really deal with [2]. Therefore, the majority of methods in machine intelligence are based on linear functions as the one given here.

In this example, we have talked about functions that takes a single number as input, and outputs another single number. Let’s now look how functions can be applied to vectors.

Vectorial functions

In the previous post we have dealt with the question of representing high-dimensional images as vectors. Of course, we can also define functions on these vectors. Recall that a vector is merely a list of numbers of fixed size, e.g. a 3-dimensional vector looking like this:

0.909	1.000	0.860

So let’s define a linear function on 3-dimensional vectors:

f_a(Image) = f(Image₁, Image₂, Image₃) = 2*Image₁ + 5*Image₂ - 1* Image₃

What does this function do? With the little subscript we denote the individual dimensions of the input image. The function therefore computes the sum of the individual dimensions of the input vector, each dimension multiplied with some number. These numbers (here 2, 5 and -1, which I have chosen arbitrarily in this example) are called the parameters of a function. The result for this function applied to the example vector above is:

f_a(Image) = 2*0.909 + 5*1.0 - 1*0.86 = 5.958

At first sight this function does not really seem to make much sense. Why should we sum up pixel values of an image? For example, it allows us draw some conclusions on whether the image is rather dark (low value of f_a) or light (high value); and the different parameters of f_a allow us to emphasize certain regions of the image more than others. We will see later that this is actually very useful for recognizing things in images.

You might have noticed that the previous function mapped a vectorial input to a single number. But we can also define functions that map vectors to vectors:

f_b(Image)</b> = [ 2*Image₁ + 4*Image₂, 3*Image₂ + 1*Image₃ ]

This function maps the 3-dimensional input vector to a 2-dimensional output vector by summing over subparts of the input (the brackets indicate that the output is a vector, and the comma separates the two output dimensions). These types of functions will be very useful as they allow us to transform data into different representations, for example lower-dimensional ones.

Image classification function

I will now introduce a last type of functions which I call decision functions or classification functions. In a nutshell, these functions look like that:

f_c(Input) = 1 If Input > 10
f_c(Input) = 0 otherwise

Classification functions map input data to 2 (or more) categories which we simply enumerate from 0 to the number of categories (minus one).

If we put together classification functions with our knowledge about vectorial functions, we can reconsider the example in the previous post we have seen that we can draw (hyper)planes to separate two categories of objects, namely blobfish and Sebastians:

We would now assign the category Sebastian to 0 and blobfish to 1, and make the if-else-part of f_c such that it takes into account whether an input sample lies on one or the other side of the line. I will not write that out explicitly, but in fact the left-of-or-right-of-line can also be cast as a multiplication of the input with a bunch of numbers. So classification functions are exactly what we want if we want to solve classification tasks - surprised?

Relationship between functions

Before closing this article, I would like to point out the relationship between the stock price and the image classification function.

For image classification, the trick for visualizing the hyperplane that separated Sebastians from blobfish was to shrink the 27x35 images to a 3x1 image which allowed us to treat images as points (vectors) in 3D space. What does this look like if we shrink the images even further, namely to a 2x1 image? We can then visualize the images as vectors in 2D. Similar as before we can now ask how to separate the shrinked Sebastians and blobfish. The answer is that we have to find a “2D hyperplane” - which turns out to be just a line!

Interestingly, this looks very much like the stock price prediction above. The main difference is that we do not draw the line through the data but in such a way to separate the data. Still, in both applications we use a linear function.

Maybe it’s now a bit clearer how these linear functions translate into higher-dimensional spaces: the plane is a generalization of the line to 3D, and the hyperplane a generalization of the line/plane to higher dimensions! So in the future if we think about discriminating hyperplanes, we will usually visualize this as a line or a plane separating two sets of 2D or 3D points. And as before the mathematical field of linear algebra describes how to deal with these lines, planes and hyperplanes.

Summary

For now, you should have gotten a feeling for what functions do: translating input to output data. Functions can take different inputs and return different outputs, such as numbers and vectors. (In fact, they can even take other functions as inputs as well! But we won’t bother with these insane cases now.)

In the next post, we will get to the real meat: how to learn functions automatically.

TL;DR:

Functions in the mathematical sense translate an input to an output
The simplest and best understood functions are linear functions
Functions map between numbers, or more general, vectors.
Functions can map to single numbers, but also to vector
If functions map onto a limited set of categories/classes, we call them classification functions

Footnotes:

Of course this is largely oversimplified. The actual price would also depend on the size of the company at the time, economic health of the country, stock market crashes, etc. But for the sake of the example let’s assume none of these factors plays a role.
To cite the mathematician Ian Stewart: “Classical mathematics concentrated on linear equations for a sound pragmatic reason: it could not solve anything else.”

Stop Teaching Math in School?

Mon, 28 Dec 2015 13:43:00 +0000

Today, I watched the TEDx talk by John Bennett who advocates that we should stop teaching math to middle and high school students. I find it great if people make controversial claims and defend them argumentatively. And indeed, there seems to be a problem with how math is taught in school, given the infamous “math anxiety” that Bennett is alluding to. Yet, I strongly disagree with Bennett’s conclusion.

In his talk, he tells us that at some point he realized that 99% of the American population don’t need math. Therefore, you shouldn’t teach math to them. He also rejects the argument that you need math in school at least to get a high SAT score - which is important for getting a good job. He aims to disprove this argument by citing Alfie Kohn with the following quote:

The SAT is a measure of resources more than of reasoning. Year after year, College Board’s own statics depict a virtually linear correlation between SAT scores and family income.”

Whether or not you buy the argument, my question is: how are you going to understand this argument and this quote, if you don’t know what a linear correlation is - which you only get with a basic education in higher math? Even worse, how are you going to understand other claims based on correlations unless you understand the relationship between correlation and causation? Are you going to understand that any argument based on a single correlation is prone to be false?

There is a deeper point that I want to make here. In my opinion, Bennett misconceives education as preparation for professional life. Of course, if you are not becoming an engineer, you might never see an integral sign again in your life. However, education is a preparation for being a member of the society - you should be able to critically read the newspaper, question claims and lines of argumentation, which are nowadays often backed up by some sort of statistics. And sound argumentation, sharp thinking and - obviously - statistics are at the very core of math. All of this is required because you have the right to vote for your leaders - if you are well-informed and able to make up your own opinion on valid logical ground, you are less likely to fall for populists and propaganda.

I do agree that my math curriculum did not do a great job at teaching me all of the this - but it should. We most definitely should change the way math is taught, and there are various ideas of doing so (I have some thoughts about what is going wrong in university courses, and some people have ideas on how to fix them). However, dropping maths from the curriculum does not sound like a viable solution to me. The number of people supporting clowns like Donald Trump shows that we have an education problem. Let’s try to make education better, and let’s make teaching better math part of this endeavour.

The Curse of Overfitting

Fri, 07 Aug 2015 15:00:00 +0000

In the last post we obtained an understanding of how to learn functions from data, and we developed our first learning method. In this post, we will start building an understanding of why learning from data is actually pretty hard. The first problem we are facing is the curse of overfitting. Let’s see what that is.

Recap again the stock price prediction problem: given the annual revenue of a company, we want to predict the company’s stock price. I have given you set of example data and we have found the following linear regularity in the data:

Alhough I told you that these linear functions are really cool and mathematics can cope with them quite well you have probably already thought of many cases where linear functions do not suffice and we need different ones. And indeed, even in the stock price problem, there seems to be no reason for disregarding different functions, for example this one (red curve):

It is not as nice as the line, but it perfectly passes through all of the data points. And from an economic perspective this function looks much better as it predicts that if annual revenues exceed 1000000 Euro the stock price will go up much more quickly! So better go and buy some shares now!

Occam’s Razor

Let me give you a reason in defense of the linear function: the function’s simplicity. Although both functions perfectly fit the data (both the blue line and the red curve perfectly coincide with the blue dots) the line has much less wrinkles than the curve - namely zero rather than six! For the line we do not have to choose whether the wrinkles go up or down, how high/low they should go etc.

In fact, this type of reasoning is so common that it has already been brought up over 700 years ago by William of Ockham. Therefore, this principle is called Occam’s razor (although Ockham wasn’t the first to bring it up, nor was he known for being particularly passionate about shaving). It states that “among competing hypotheses that predict equally well, the one with the fewest assumptions should be selected”.

Overfitting

Although this explanation of Occam’s Razor sounds reasonable, there is an even better way of arguing why we should prefer the line in this stock price example. Occam’s razor rejects the wrinkled function because we have to make too many choices about the direction and the size of the wrinkles. In fact, there are infinitely many wrinkled functions going through all of the points, for instance this one (green line):

The problem with that is that the chance of picking the wrong wrinkled function is much higher than picking the wrong straight line - because (at least in our example) there is only one line that goes exactly through all of the points. I cannot rotate or shift the line up without it losing contact with one or more of the points. Therefore, we should prefer the line. So in a way, we have given a justification of Occam’s razor by realizing that using a more complex function for predictions forces us to take more arbitrary decisions, which makes us more prone to making mistakes.

Random fluctuations

In the case where the data lies exactly on a line the answer was clear that the line is the best choice. However, in reality, data is not perfect but almost always undergoes random fluctuations and noise. So let us make the stock prediction data a bit more realistic and then look at how well line and wrinkled function explain the data:

We see that no line will be able to pass directly through all the data points, but we have easily found a wrinkled function doing so. Still the line shown here seems to capture the main trend quite well. Which of the two functions do you prefer for making predictions about future stock prices?

Since I have told you that the fluctuations are random, you will probably prefer the line - even though it does not fit the data exactly. Although the wrinkled function explains the individual data points well, it does not explain the more moderate trend of the revenue-stock price relationship well; it focuses too much on explaining the random fluctations in the data. Data scientists call this general effect overfitting, and we say that the wrinkled function overfits the data.

Battling Overfitting

Out there in the real world, discovering whether your function overfits or not is really hard. Probably, it is the problem of machine learning. It is so prominent that two machine learning professors have even written a song about it. Seriously:

Luckily, there are a few things you can do to counteract overfitting apart from singing about it. I want to give you a few examples.

First, you can retain some preference for simpler explanations, by making a similar argument as before: even for the randomly fluctuating data there are still only a few lines that are not too far away from any of the points. But since there are infinitely many wrinkled functions passing directly through all of the points, the chance of picking the wrong wrinkled function is very high. So using a simpler function is better.

Secondly, you can hold back some of your data and not use it for finding the function - you only use it after having computed the function to test how your function copes with that unseen data. If your function really sucks at explaining the data you have held out (called test data) you should consider using a different type of functions. In fact, using test data is a standard procedure in machine learning and everyone has to do it because fitting well your training data does not tell you at all whether you learned the right thing.

Third, you can use more data. In fact, this is why everyone is so excited about big data which heavily relies on lots of data being available. It does work well, but we will see in the next post, that there are some limitations about how much data you can get.

Forth, you can get more knowledge about which shape of the functions you actually expect. If for example you know that the stock price varies in cycles, that is goes up, slightly down, then up again etc., you can try to find a function that reflects this knowledge.

Although all of these things are good ideas and they definitely help to counteract overfitting, in the end you still have to keep your fingers crossed and hope your function predicts the right thing. There is no guarantee that your function doesn’t overfit although all of the things suggested make it less likely.

I hope that by looking at this very simple example, and by understanding how overfitting can occur here, you have already come to think more critically about decision making - both in machines and in humans.

Underfitting

Last but not least, let me say that of course also the opposite thing can happen: underfitting. If we choose a function that is too simple to explain a relationship we will not get a good prediction, either. I leave it as an exercise for you to think about a good example of underfitting.

That’s basically it - you have now understood the biggest problem of learning from data. And in case you want to brag in front of your friends, here is some more terminology: the topic of this post, that a function learned from data always has to balance under- and overfitting is also called the bias-variance trade-off. Variance relates to how much all possible different wrinkled functions vary - the more wrinkles you allow, the higher is the variance of your learning method and hence the more prone the learner is to overfitting. Bias refers to underfitting, stating that a too simple function adds a systematic bias to the prediction which cannot be overcome by the learner, not even when given more data. The bias-variance trade-off is a smart way of stating that if don’t consider the simplest yet not too simple hypothesis, your hypothesized function is more likely to be wrong – too simple and wrong or too complicated and wrong.

This post elucidated the first big problem of machine learning: overfitting. Next, we will look at the curse of dimensionality.

TL;DR:

Finding the right predictive function for data is hard
Even if the function perfectly fits the data it might be totally wrong: an effect called overfitting
Occam’s razor states that one should prefer simpler solutions
A justification for this is: when using simpler functions we are less likely to pick the wrong one
Underfitting occurs when choosing too simple functions
Singing about data science is fun

The Power of Machine Intelligence

Tue, 28 Jul 2015 15:00:00 +0000

Thinking machines have revolutionalized our everyday life, maybe even more than industrialization or the invention of the car. But although everyone of has an intuition about how a car works thinking machines remain obscure and magical to most of us.

I believe that everyone should have at least a “car intuition” about thinking machines. Unfortunately, common introductions to artificial intelligence and machine learning require quite some knowledge about programming and math and scare laymen off. Luckily, none of these skills is required to grasp the core ideas behind thinking machines. And these core ideas are what I would like to tell you about in the forthcoming series of articles in this introduction to machine intelligence.

To whet your appetite I would like to present four prototypical applications of machine intelligence. We will get back to these applications later and get a more detailed understanding of how machine intelligence approaches each of these problems.

Image Understanding

For decades, researchers have been trying to make machines see the world we do. In 1966, one of the most famous artificial intelligence researchers, Marvin Minsky, hired an undergrad to solve the following problem as a summer project: connect a TV camera to a computer and make the computer describe what it sees. Now, 50 years later, we still haven’t fully solved the problem, but at least we have made some progress. In the image classification task computers are now able to outperform humans in certain regards. In this task, the computer has to decide for any given image what kind of object is depicted on that image. As manually programming the computer to do so has failed, machine intelligence tries to solve the problem by learning from data. To learn successfully huge amounts of labeled data (images together with descriptions of what object category is depicted on each image) are required which is why the community created a database of over 15 million images containing 21841 object types. By learning from these images, using a method called deep neural networks, machine intelligence is now able to predict the object category correctly in over 95% of all cases - which is above human performance. These neural networks will one of the topics covered later.

Top 5 predictions by an image classification approach. Credit: Microsoft Research

Note however that the general problem of seeing machines hasn’t been solved yet since the image classification task is limited to pictures where only one object is present. Describing entire scenes is much more complicated, but recently progress has been made there, too.

Algorithmic Trading: Stock Price Prediction

Machines are slowly taking over the stock markets using algorithmic trading. One important thing these machine traders have to do is predicting stock prices. How would machine intelligence approach this problem? In fact, the task is very similar to the image classification problem, only that, instead of an object type, the computer should predict a continuous valued number: the stock price. But what data should the stock price predictor use for learning? In the image classification task this was obvious: the image itself. Here, it is not clear what influences the stock price. It could be last months’s revenue of the company, the size of the company, the stock price from last month, last year, and many more. A data scientist will approach this problem by collecting as many of these pieces of information, so-called features, and try to weigh the features in the best way to predict the stock price. We will look later in more detail how the computer can automatically weigh features to make predictions, and take a look at why the task of “weighing in the best way” is much more difficult then it sounds.

Spam Classification

Everyone knows the problem of spam. It tastes weird and, even worse, it fills your mailbox with juicy offers for penis enlargement or earning money from fake Nigerian royalty. But thanks to machine intelligence, nowadays most spam automatically finds its way into your junk folder. This application of machine intelligence is called spam classification and - in its simplest form - works as follows: your mail program receives an email and extracts features (as in the example before) from the email, such as what words the email contains, whether you know the author of the email, and many more. It then weighs each of these features and makes a binary decision: spam or no spam.

We see a pattern emerging here. Spam classification is very similar to image classification and stock price prediction: data scientists collects many spam emails, extract features from these emails, and use fancy tools to find the best weighting of all the features. The main difference is that not a continuous number (stock price), or an object category, but a binary output (spam / no spam) is computed.

Robot Skill Learning

The last and maybe most exciting application of machine intelligence is robotics. Interestingly, you can (to some extent) apply exactly the same techniques - as before - to make a robot do cool things. The following video shows how to teach a robot to play table tennis.

The type of learning slightly differs from the approaches seen before. Instead of telling the robot exactly what the right motion, the researchers let the robot try out motions on its own and give the robot a reward if it performs a good stroke, or a penalty if it performs bad (in fact, the robot doesn’t get lollipops as rewards, but delicious large numbers). The robot will then repetitively generate motions, collect rewards/penalties and step by step correct its motions in order to get more reward and finally execute an excellent table tennis stroke.

I hope I was able to get you excited about machine intelligence! In the next couple of posts I will try to convey the basic ideas of how machine intelligence approaches all of these problems.

The next post will tell you in more detail why you don’t need to understand complex math and programming to understand machine intelligence.

High-dimensional Spaces

Sat, 25 Jul 2015 16:02:00 +0000

In the last post we have seen examples for how numbers can encode information, becoming data. In this post we will talk about a very important way to look at data. This view will allow us to play around with data in a powerful way, and this view lies at the core of machine intelligence and the science of data.

Data samples as points in space

I have previously told you that any image can be represented by a table of numbers where each number encodes the gray scale value of the image. For the trick that I would like to teach you, this image is actually a bit too large. So for the sake of the argument, let us shrink the original 27x35 pixel image drastically until it gets really really tiny, namely 3x1, that means only 3 pixels wide and 1 pixel high - bear with me for a second even if this sounds silly to you. We won’t see anything on this image but that doesn’t matter right now. Our 3x1 image expressed in gray scale values now looks like this:

0.909	1.000	0.860

I have set the background color of each table cell to correspond to the actual gray scale value the number encodes. Here comes the trick: although we know that these three numbers encode gray scale values we can pretend they were encoding the location of a point in 3D space. So instead of encoding “luminosity” we think of the numbers are being in “meters” or “centimeters”. As you might remember from high school we draw a location as an arrow in an coordinate system. So let’s draw our 3x1 “image” in a 3D coordinate system:

So you might think: “drawing arrows is really fun but why the heck are we doing this?” There are two reasons for that: the first reason is this that we humans are really good at manipulating objects in 3D space. We know how to move objects, we know how to rotate them, how to distort and mirror them, how to project them on a 2D planar surface (by taking a picture of it), and much more. Thus, if we treat data as locations in space we can apply all our spatial 3D knowledge to it. In fact, linear algebra has got all the math worked out to simulate these 3D operations on computers (as you can admire in Toy Story, Madagascar, and so on) [1].

Now the question is, what does the 3D arrow sketched above have to do with the original 27x35 pixel wide image of my face? Here’s the gist: linear algebra is so general that it does not care about the dimensionality of the data - it works for 3-dimensional spaces in the same way as for 5-dimensional or 500-dimensional spaces - even though our brains are not capable of imagining 500-dimensional data visually! This allows us to treat the 27x35 picture of my face as a point in some crazy unimaginable 945-dimensional space [2]! We will see later that applying such operations as moving and rotating a point in this huge space will be the basis for extracting information from it, as for example detecting what is in the image represented by this point [3].

So just to make this clear: I will now continue drawing 3D arrows, but these arrows are in fact just sketches of 945-dimensional arrows (since they are impossible to draw). Therefore I will write about 945-dimensional, but draw 3-dimensional arrows.

Discriminating between categories = finding separating planes

The really cool thing is that we can now understand how the computer can discriminate between images that depict different objects. Imagine we have another image of, say a blobfish. By the same procedure as before we can make the blobfish a point in 945-dimensional space. Now let’s assume we do not only have one picture of me and one of the blobfish, but a whole bunch of pictures for each category, Sebastians and blobfish.

How could a computer automatically discriminate between these two categories? Geometrically, of course! By treating each image as a point in a 945-dimensional space, we can now come up with a geometrical interpretation of discriminating between categories: we find a plane which separates these two sets of points (blue is me, red the blobfish):

Using this plane, the computer can now automatically discriminate between blobfish and Sebastians by checking on which side of the plane the point lies.

Sit back for a second and make sure that you have understood this. Because if so you can be proud - you have just understood the key principle behind image classification, the problem that I have presented in the intro article! The majority of all approaches in machine intelligence draws such planes through the high-dimensional image space, assigns a category to each side of the plane, and then checks on which side of the plane a new, previously unseen image lies.

Caveats

Of course, there are some details that I have omitted. As mentioned before, the sketch is in 3 dimensions, although we should draw 945 dimensions, which is not possible; and also the thing separating the two object categories in 945 dimensions is not really a plane, but a hyperplane (that’s what a plane is called in four or more dimensions - not to be confounded with hyperspace). But still, the math - which I have not shown you, so you need to trust me on this one - assures us that the concepts of points and planes also exist in 945-dimensional space, so it is totally valid to think of it in 3 dimensions only.

Another important detail that I have omitted is how to find this (hyper-)plane. And in fact this is the central problem of machine intelligence, or more precisely machine learning. We will talk about how this learning works in the post after the next one.

Finally, I have to lower your expectations about this method a bit: discriminating Sebastians and blowfish by finding a plane in this 945-dimensional space does not really work well in practise. The reason is that pictures of Sebastians and blowfish are not as nicely separable as I have suggested in the 3D arrow picture above, but rather scattered all around the space. But the general idea of finding discriminating hyperplanes in high-dimensional spaces still holds. And we can get it to work by first bringing the image into a different representation, that is a different space, and then finding the hyperplane in this new space (at this point you should remember our discussion about representations in the previous post). You can either try to transform images in these representations explicitly, i.e. think of a good representation and program it yourself, or implicitly, trying to find a way of doing this automatically. We will talk about both approaches later, too.

Terminology

Some last notes on terminology.

Often, sequence of numbers, such as the 3d point or the 945-dimensional representation of the image is called a vector, and it then lives in a vector space of some dimensionality. Although being basically the same thing, the term vector has a slightly different connotation which we will ignore for now. But I will mostly use the vector mainly because it sounds cooler.

Secondly, mathematics tries to be parsimonious with concepts, mainly because it makes talking about things easier. Therefore, we get rid of the concept “numbers” by treating them as 1-dimensional vectors. So in the future if I talk about vectors, you can often think of them as just being numbers.

Summary

To summarize, the approach of machine intelligence is the following: we transform some input, for example an image into a table of numbers, call this a vector and then treat this vector as a point in a high-dimensional space. We then apply our knowledge of how to manipulate points in 3D to transform this high-dimensional vector in order to extract information from it. Moreover, we can classify between different types of data, for example object categories, by finding discriminating hyperplanes in this space.

In the next post we will look at how be more precise about what we mean by transformations and how to describe these hyperplanes by introducing the concept of functions.

TL;DR:

Data can be viewed as points in high-dimensional spaces
We can apply the same transformations to points in this space as in 3D
Such points are called vectors
Hyperplanes in space discriminate between vectors, and hence categories

Footnotes:

Actually, linear algebra can do even a bit more than what is “physically possible” in the 3D world, e.g. mirroring and skewing objects.
To turn images into an arrow/point, we need to remove the column and row structure of the image and write all of the numbers in one very long row. And since 27 times 35 = 945, the original image becomes a 945-dimensional point. This has indeed been common practise when learning from images (although more recent algorithms exploit the spatial, i.e. “grid” structure of the image, rather than just stitching the columns together).
It is important to notice that moving or rotating a point in 945-dimensional space which represents an image is not the same as shifting or rotating the image! Since moving in 3D is equivalent to “adding a number to one or more dimensions”, this is also the definition of moving in 945-dimensional space. If you push a 3D object 3m up and 5m to the right you effectively add these two numbers some coordinates of the object. Hence, in the image example, moving is equivalent to changing the gray scale value of certain pixels. Neither do vector rotations actually rotate the image.

Data, Numbers and Representations

Sun, 19 Jul 2015 15:07:27 +0000

I would like to start the intuitive introduction to machine intelligence by looking at the term data. Let us try to gain an informal but sufficient understanding of how we could define data.

Data are numbers

Big data, data mining, data analysis, data, data, data - everyone wants data. But what is data? The philosophical answer could be: information. The data scientist’s answer would probably be: a bunch of numbers. Indeed, we are going to stick to the latter definition, namely that data is everything that can be encoded by numbers [1].

Some examples: a picture you have taken with your digital camera is data. You can look at it, see the things depicted on it. But your digital camera uses numbers to store the pictures. Also the MP3 sound file that your are currently listening to is data. Even this text here is data! So how is data numbers?

Let’s look at the concrete example of a picture:

What do we see here? It’s a picture of my face of course! Ok, if you don’t see it you should definitely watch The Matrix more often. In case that doesn’t work, how do we get the picture back? The trick is that I have taken the gray scale picture of my face and replaced every dot in the original picture with a number between 0 and 1 where 0 means “black” and 1 means “white”, and the numbers inbetween represent the different shades of gray between these black and white tones. When showing the image on the screen the computer translates these numbers back to gray values.

Data is numbers + what + encoding

However, the way how I have chosen the numbers to substitute gray values in the picture is entirely arbitrary. We could have done the same by representing black with 0.5 and white with 12, reverse the order, or even mix up the order of the numbers, etc. The way how you choose the numbers is completely up to you. But it is very important to know that: somebody else will not know how to make sense of the numbers unless you tell her that it is a picture and which numbers correspond to which gray scale. The latter information, of how to translate gray scale values to numbers and back is what we call the representation, and is also sometimes called the encoding of the data.

What about the sound file example? This one is a bit more tricky. You have to know quite a lot about the physics of sound. Let’s take a 10 seconds sound file. The easy way to think about how to encode sound by numbers is to divide the 10 seconds a sound file into say 10000 equally sized parts à 1 millisecond and assign a number to each part. Every number corresponds to the “loudness” of a part. That’s basically it! You might wonder now how a human voice, a guitar and the engine of a car can all be broken done to these few numbers. Well, it’s really just that! The real “magic” is happening in your brain which collects these pieces of information and generates a certain sensation which you identify as an opera singer or Jimi Hendrix playing his guitar. If you want to learn more about these things I recommend you to look into this book [2] which does a great job at explaining what sound and music really is and how the brain comes at perceiving it the way it is.

What is finally left is the text example. This one is easy, too. I leave it as an exercise to you to think of how to encode or represent text as numbers. (A hint: think about how many different letters and characters we have in our alphabet).

Summary

To summarize, we have learned that data is numbers plus information about what these numbers mean, called the representation or encoding.

As a final remark, I cannot overemphasize how crucial the representation of data is. The representation of a problem can drastically influence the ability to solve it (as exemplified very well in [3]), and corresponds directly to the question of how to represent data. Lots of the stuff you will read about in future articles deals with transforming numbers from one representation into another. In the next article we will come up with an interesting way of looking at data in order to extract information from it.

TL;DR:

Data is numbers + representation (what these numbers stand for)
How data is represented is crucial

Understanding Machine Intelligence without Formal Math

Sun, 19 Jul 2015 15:00:00 +0000

In this post I want to defend the basic idea of this series of articles, namely why I am trying to avoid mathematical formalism for explaining data science.

If you don’t care about math anyway, you can skip this post and start reading about data.

I do not have anything against formulas. Quite the contrary: mathematical formalisms are a fantastic way to condense and sort out the necessary ingredients of a method. Without formalism, we could not program a computer to operationalize any method. Think of a formula as a packing list for your camping trip. It has to be complete and all the things you need have to be on it, and only on site you might realize if you did something wrong (“Oh no, I forgot to bring my hiking heels”). In a similar way a formula has to contain everything required to describe your mathematical concept or method, and only when applying it / executing your method you realize whether something in your mathematical description is wrong or missing.

However, I find formulas a very bad way to understand things. I see three main reasons for that.

The first reason is that after studying and working in academia for a couple of years, I realized that the magic “ahhh, THAT’S how it works!” moments while trying to grasp a complex method always happened when I could find analogies in “bodily” or linguistic terms: while rotating three-dimensional structures in my head, while visualizing probabilities as a contour (like a mountainous landscape), while breaking down complex relationships to sets of causal if-then statements, and so on. Maybe not everybody thinks of mathematics like this. It is possible that some mathematical geniuses think differently when reasoning about math. But the vast majority of researchers and students that I have met so far strive for analogies in order to explain or grasp complex mathematical concepts. I think there is a lot of empirical evidence for that, for example that scientific papers plot graphs, bar charts and the like in order to visualize results instead of just printing some numbers. Two famous cognitive scientists even have made an endeavor to argue that the holy grail of mathematics, Euler’s formula, can be derived only from the way we perceive the world (in contrast to a Platonic view of universal mathematical truths) [1,2].

The second reason is that formulas often do not reflect the way how people figured out the concept or method underlying the formula [3]. It is like not telling the whole story but just the end of it. You will have a hard time reconstructing the rest of the story from that little information. With formulas it is at least not impossible as they condense all information, only in a very concise manner; but you will still have a hard time figuring out its meaning without some context.

The third reason is much simpler: Formulas scare many people. I believe one reason for this lies in the condensed nature of formulas that I have mentioned before. Formulas use a very precise and non-redundant language, which is far less verbose than natural language. Therefore, you can spend an entire day looking at one single formula without getting it. I believe this being the reason for some people thinking that they are not good at math, although they probably could be. They get disappointed too quickly because they cannot grasp the formula within an eyelash. Relax guys, that holds true for most of us.

So let us now jump right into the topic and talk a bit about data!

Update 09/11/2016: I have received a series of similar comments regarding this article, which I would like to respond to. The first critique was that I am claiming that all researchers think in intuitions, rather than in formulas, when devising mathematical equations and algorithms. That is of course not true; there are people who prefer to and are very good at thinking in abstract formulas, rather than in intuitions. And indeed, certain problems can be more easily solved “syntactically”, that means only by applying mathematical rules rather than thinking in intuitions (but we won’t do that in this series of articles). Secondly, I want to admit that, in some cases, the full aha-that’s-how-it-works moment comes when reconciling the intuitive and the formal, mathematical levels. But - at least for me - the intuitive level was always required to make full understanding happen.

TL;DR:

Formulas are good for condensing and bullet-proofing concepts and methods, but less often for understanding
Most people conceptualize complex things in analogies, relating them to perception (e.g. vision) and language
Formulas scare people

Sebastian Höfer's Blog

Yet Another Machine Learning 101

General setting

Supervised Learning

Overfitting

Example: Linear regression

Gradient descent

Ordinary least squares

Computational example

Overfitting

Regression vs. Classification

(Deep) Neural Networks

Unsupervised Learning

Reinforcement Learning

The Curse of Dimensionality

Parameters

Adding dimensions

Hughes effect

Summary

TL;DR:

Further reading:

Learning Functions from Data: A Primer

Learning Functions from Data

How to extrapolate from data?

Defining the learning problem

Caveats

Terminology

TL;DR:

Footnotes:

Functions as Data Translators

Mathematical Functions

Object description functions

Stock price prediction function

Vectorial functions

Image classification function

Relationship between functions

Summary

TL;DR:

Footnotes:

Stop Teaching Math in School?

The Curse of Overfitting

Occam’s Razor

Overfitting

Random fluctuations

Battling Overfitting

Underfitting

TL;DR:

Further Reading:

The Power of Machine Intelligence

Image Understanding

Algorithmic Trading: Stock Price Prediction

Spam Classification

Robot Skill Learning

High-dimensional Spaces

Data samples as points in space

Discriminating between categories = finding separating planes

Caveats

Terminology

Summary

TL;DR:

Footnotes:

Data, Numbers and Representations

Data are numbers

Data is numbers + what + encoding

Summary

TL;DR:

Further reading and thinking:

Understanding Machine Intelligence without Formal Math

TL;DR:

Further reading and thinking: