CSI 4106 - Fall 2026
5 amazing visuals show how the male fruit fly’s brain map is advancing neuroscience, Google Research, 2026-09-03.
Researchers at HHMI’s Janelia Research Campus, Google Research, and collaborating institutions have generated the first complete map of a male fruit fly brain. It comprises over 166,000 neurons.
As described in the article below, developers created game-playing demonstrations in less than a day by routing game inputs through the fly’s existing, biologically evolved neural circuitry. These demonstrations enabled the fly to play Minecraft, Doom, Beat Saber, and other games while allowing developers to examine how the network processed and responded to these inputs.
2 years ago, the timing for this lecture could not have been better since John J. Hopfield and Geoffrey E. Hinton were awarded the 2024 Nobel Prize in Physics “for their fundamental discoveries and inventions enabling machine learning with artificial neural networks.”
The course introduced symbolic AI and connectionism as two major approaches to artificial intelligence. Neural networks provide a connectionist model: many simple computational units work together to map inputs to outputs.
Two questions guide this introduction. Computation asks what a network does when its parameters are fixed. Learning asks how data can be used to choose those parameters. Understanding the computation comes first; the following lectures develop the training methods.
TensorFlow Playground is an interactive demonstration of feedforward neural networks for two-dimensional classification. Its network diagram shows how inputs pass through hidden units to produce an output, while the background shows the resulting decision regions.
The weights determine the computation performed by the network. During training, changes to those weights alter the decision regions. The distinction between computing a prediction and learning the parameters also applies to the larger networks used later in this lecture.
Experiment with TensorFlow Playground to build intuition about how feature transformations and network architecture affect classification. For the nonlinearly separable circle and exclusive OR (XOR) datasets, first try to construct classifiers with no hidden layers by incorporating additional input features. Then, using only the original input features, explore how increasing the number of hidden layers and units enables the network to learn nonlinear decision boundaries.
Artificial neural networks are machine learning models built from interconnected computational units, inspired by biological neurons.
AKA artificial neural networks or neural nets, abbreviated as ANN or NN.
Supervised Learning: Classification, Regression
Unsupervised Learning: Autoencoders, Self-Supervised
Reinforcement Learning: Now an Integral Component
We will begin our exploration within the framework of supervised learning.
Attribution: Jennifer Walinga, CC BY-SA 4.0
Biological neurons receive signals through their connections and transmit signals to other neurons. Artificial neural networks borrow the idea of interconnected units, but use deliberately simplified mathematical computations.
For the models studied here, the useful abstraction is a unit that combines its inputs and produces an output. Composing these units gives a network that performs a more complex computation.
Attribution: Molecular Mechanism of Synaptic Function from the Howard Hughes Medical Institute (HHMI). Published on YouTube on 2018-11-15.
From biology, we essentially adopt the concept of simple computational units that are interconnected to form a network, which collectively performs complex computations.
While research into understanding biological neural networks is undeniably important, the field of artificial neural networks has incorporated only a limited number of key concepts from this research.
A single unit combines its inputs using a weighted sum and a bias, then applies an activation function. For a binary threshold unit, the equation \mathbf{w}^{T}\mathbf{x}+b=0 defines a decision hyperplane. A network composes many such computations, allowing intermediate units to transform the inputs before the output is computed.
Attribution: LeNail, (2019). NN-SVG: Publication-Ready Neural Network Architecture Schematics. Journal of Open Source Software, 4(33), 747, https://doi.org/10.21105/joss.00747 (GitHub)
The term connectionism emphasizes the connections between computational units. In the feedforward networks studied here, units are organized into layers, and each connection has an associated weight.
The architecture specifies which units are connected. The weights and biases determine the function computed by that architecture. Training adjusts these parameters using data, so that the network’s predictions better match the targets.
Attribution: LeCun et al. (2015)
In the book “Deep Learning” (Goodfellow et al. 2016), authors Goodfellow, Bengio, and Courville define deep learning as a subset of machine learning that enables computers to “understand the world in terms of a hierarchy of concepts.”
This hierarchical approach is one of deep learning’s most significant contributions. It reduces the need for manual feature engineering and redirects the focus toward the engineering of neural network architectures.
where x_1, x_2 \in \{0,1\} and f(z) is a step function:
f(z)= \begin{cases}0, & z<\theta \\ 1, & z \geq \theta\end{cases}
McCulloch and Pitts (1943) connected neural activity with Boolean computation.
McCulloch and Pitts studied networks of idealized neurons whose activity is binary: a unit is either active or inactive. Their model used fixed thresholds and excitatory and inhibitory connections (McCulloch and Pitts 1943). The threshold-unit examples here provide a simplified illustration of the connection to logic.
The function f(z) is the indicator of the condition z\geq\theta: it returns 1 when the condition holds and 0 otherwise. A unit therefore maps binary inputs to a binary output.
Units implementing AND, OR, and NOT can be composed into circuits that compute Boolean functions. The parameters in these examples are chosen by hand so that the computation is easy to follow.
f(x_1 + x_2)= \begin{cases}0, & x_1 + x_2 <\theta \\ 1, & x_1 + x_2 \geq \theta\end{cases}
With \theta = 2, the unit implements an AND logic gate.
With \theta = 1, the unit implements an OR logic gate.
NOT can also be implemented with a negative input weight and an appropriate threshold.
For binary x, a unit with input weight -1 and threshold 0 computes NOT: f(-x)=1 when x=0 and f(-x)=0 when x=1. A negative weight alone does not specify a gate; its effect depends on the threshold and the other inputs.
AND and OR require no learning in this illustration: their thresholds have been selected to implement the desired truth tables.
Networks of threshold units can implement Boolean functions by composing logical operations.
With the connections and thresholds fixed, the network performs a specified computation.
Learning introduces a different question: how can data determine the parameters?
McCulloch and Pitts connected idealized neural networks to the mathematical study of computation (McCulloch and Pitts 1943). In the examples here, composing threshold units provides a concrete way to compute a Boolean function.
Their broader discussion of Turing computation included networks supplied with a tape and mechanisms for reading and writing it. A fixed feedforward network implements a chosen function. The key idea for this lecture is that simple units can be composed into a network that maps inputs to outputs.
Rosenblatt’s perceptron shifts the emphasis toward learning from examples: some of the parameters are adjusted using data rather than specified entirely by hand.
The buzz around artificial intelligence isn’t a new trend; in fact, it was already generating excitement back in 1958, as highlighted by a quote from the New York Times.
The historical footage illustrates both the early enthusiasm for learning machines and the gap between demonstrations and ambitious predictions about their future capabilities.
Rosenblatt (1958)
Rosenblatt’s perceptron introduced a model that could learn from examples by adjusting parameters (Rosenblatt 1958). A weighted threshold unit provides the simplified computational model used here.
Its inputs can be real-valued, but its output is discrete. With the Heaviside step function, the output is either 0 or 1. Each input connection has an associated weight.
The left section of the neuron, denoted by the sigma symbol, represents the computation of a weighted sum of its inputs, expressed as \theta_1 x^{(1)} + \theta_2 x^{(2)} + \ldots + \theta_D x^{(D)} + b.
This sum is then processed through a step function, right section of the neuron, to generate the output.
Here, x and \theta are column vectors. The transpose x^T is a row vector, so x^T\theta is their dot product.
The dot product x^T \theta is then a scalar given by:
x^T \theta = x^{(1)} \theta_1 + x^{(2)} \theta_2 + \cdots + x^{(D)} \theta_D
where x^{(j)} and \theta_j are the components of the vectors x and \theta, respectively. The superscripts on x identify features in these diagrams.
\text{Heaviside}(t) =
1, if t \geq 0
0, if t < 0
\text{sign}(t) =
1, if t > 0
0, if t = 0
-1, if t < 0
Common step functions include the Heaviside step function (0 if the input is negative and 1 otherwise) or the sign function (-1 if the input is negative, 0 if the input is zero, 1 otherwise).
Add an extra feature with a fixed value of 1 to the input. Associate it with weight b = \theta_{0}, where b is the bias/intercept term.
\theta_{0} = b is the bias/intercept term.
The weighted sum and bias are familiar from logistic regression. Logistic regression passes that score through a sigmoid to obtain a probability; a binary threshold unit applies a step function to obtain a class decision directly. Their decision boundaries are both linear in the input features, but their training procedures differ.
A perceptron consists of one or more threshold logic units arranged in a single layer, with each unit connected to all inputs. This configuration is referred to as fully connected or dense.
Since the threshold logic units in this single layer also generate the output, it is referred to as the output layer.
As this perceptron generates multiple outputs simultaneously, it performs multiple binary predictions, making it as a multilabel classifier (can also be used as multiclass classifier).
Classification tasks, can be further divided into multilabel and multiclass classification.
Multiclass Classification:
In multiclass classification, each instance is assigned to one and only one class out of a set of three or more possible classes. The classes are mutually exclusive, meaning that an instance cannot belong to more than one class at the same time.
Example: Classifying an image as either a cat, dog, or bird. Each image can only belong to one of these categories.
Multilabel Classification:
In multilabel classification, each instance can be associated with multiple classes simultaneously. The classes are not mutually exclusive, allowing for the possibility that an instance can belong to several classes at once.
Example: Tagging an image with multiple attributes such as “outdoor,” “sunset,” and “beach.” The image can simultaneously belong to all these labels.
The key difference lies in the relationship between classes: multiclass classification deals with a single label per instance, while multilabel classification handles multiple labels for each instance.
As before, introduce an additional feature with a value of 1 to the input. Assign a bias b to each neuron. Each incoming connection implicitly has an associated weight.
X is the input data matrix where each row corresponds to an example and each column represents one of the D features.
W is the weight matrix, structured with one row per input (feature) and one column per neuron.
Bias terms can be represented separately; both approaches appear in the literature. Here, b is a vector with a length equal to the number of neurons.
With neural networks, the weights are often denoted by w (vector) or W (matrix), rather than \theta.
The distinction is familiar from logistic regression: evaluating a model with fixed parameters and fitting those parameters to data are different operations. This lecture establishes the computations performed by neural networks; the next two lectures develop methods for training multilayer networks.
The perceptron learning rule is outside the scope of this lecture. For modern networks, backpropagation computes gradients, and an optimizer such as stochastic gradient descent uses them to update the parameters.
A single threshold unit cannot separate the two XOR classes with one straight line. XOR is not linearly separable in the original input space.
For binary inputs, XOR is 1 at (0,1) and (1,0), and 0 at (0,0) and (1,1). A threshold unit computes one weighted sum plus a bias, so its decision boundary in this two-dimensional space is a straight line. No such line separates these two classes.
Logistic regression has the same limitation when applied directly to these two inputs. Additional nonlinear features can change what a linear classifier can represent.
Minsky and Papert’s Perceptrons analyzed the computational limitations of restricted perceptron models (Minsky and Papert 1969). These results should not be interpreted as showing that multilayer networks cannot represent XOR. Representing a function and learning suitable parameters are separate questions.
At least one input is true, and they are not both true:
A\oplus B=(A\lor B)\land\neg(A\land B)
The minimal expression above connects directly to the AND and OR units in the network diagram. Two other standard expressions are useful for comparison:
\begin{aligned} A\oplus B &= (A\land\neg B)\lor(\neg A\land B) &&\text{(disjunctive normal form)},\\ A\oplus B &= (A\lor B)\land(\neg A\lor\neg B) &&\text{(conjunctive normal form)}. \end{aligned}
The first expression says that exactly one input is true. In the second, A\lor B requires at least one true input, while \neg A\lor\neg B excludes the case where both inputs are true. By De Morgan’s law, \neg A\lor\neg B=\neg(A\land B).
A Boolean connection: Geoffrey Hinton is the great-great-grandson of George Boole, whose work underlies Boolean algebra. This family connection links the logic used in this example with one of the researchers who helped popularize training multilayer neural networks. University College Cork’s George Boole bicentenary project
A multilayer perceptron (MLP) has one or more hidden layers between its inputs and output layer. Here, the hidden units compute AND (top) and OR (bottom).
The input nodes pass the features to the hidden units. The hidden units compute intermediate values, which become inputs to the output unit. For XOR, one hidden layer and one output layer are sufficient: these are two layers of computation, excluding the input nodes.
This hand-configured example uses threshold units to make the logical operations explicit. Modern MLPs typically use activations such as ReLU in their hidden layers; their weights and biases are learned jointly from data.
| A | B | Target y | o_1: AND | o_2: OR | o_3: XOR |
|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 1 | 0 | 1 | 1 |
| 1 | 0 | 1 | 0 | 1 | 1 |
| 1 | 1 | 0 | 1 | 1 | 0 |
The output unit computes o_2\land\neg o_1. The hidden representation makes XOR linearly separable.
In the diagram, A=x^{(1)} and B=x^{(2)}. The outputs o_1 and o_2 identify the AND and OR hidden units, and o_3 is the network’s prediction. The table confirms that o_3=y for all four input combinations.
With the convention H(z)=1 for z\geq0 and H(z)=0 otherwise, AND and OR can be implemented as
o_1=H(A+B-1.5),\qquad o_2=H(A+B-0.5).
An output unit such as o_3=H(o_2-2o_1-0.5) computes o_2\land\neg o_1. The negative weight suppresses the output when both original inputs are true. Several choices of weights and biases produce the same truth table.
The transformed inputs (o_1,o_2) are (0,0), (0,1), and (1,1). The two positive XOR cases both map to (0,1), which a straight line can separate from the negative cases. The network therefore computes an intermediate representation that makes the final classification possible.
Equivalent OR/NAND construction. A negative weight with a suitable bias can implement NOT: H(-A+0.5)=\neg A. Likewise, a single threshold unit implements NAND as H(-A-B+1.5). Its preactivation is 1.5,0.5,0.5,-0.5 for inputs (0,0),(0,1),(1,0),(1,1), yielding outputs 1,1,1,0.
The hidden units can therefore also compute h_1=H(A+B-0.5) (OR) and h_2=H(-A-B+1.5) (NAND). An AND output H(h_1+h_2-1.5) then computes XOR. This is an equivalent network; the displayed diagram uses AND and OR hidden units instead.
Representation and learning. The weights in this example are supplied by hand. For a general task, the desired outputs of the hidden units are not supplied: training must determine how changing their incoming weights affects the final prediction. This is the credit assignment problem.
Minsky and Papert established representational limitations for the restricted models they analyzed and expressed skepticism about multilayer extensions (Minsky and Papert 1969). Their results did not establish that all neural networks share those limitations. These mathematical results are distinct from the broader historical question of why interest and funding in connectionist research declined. Bottou’s foreword to the reissued Perceptrons discusses this distinction and the historical context.
Hard step functions pose an additional obstacle for ordinary gradient-based training: their derivatives are zero away from the threshold and undefined at the threshold. Backpropagation with suitable activation functions provides a way to compute how the loss depends on hidden-layer parameters. Rumelhart, Hinton, and Williams helped popularize this approach in 1986 (Rumelhart et al. 1986). The learning procedure can discover intermediate features instead of requiring a person to choose them.
Information in this architecture flows unidirectionally—from left to right, moving from input to output. Consequently, it is termed a feedforward neural network.
The diagram shows input, hidden, and output layers. There are two input nodes, three hidden units, and two output units. Counting only layers that perform parameterized computations, this is a two-layer network: one hidden layer and one output layer. Additional hidden layers and units can be added.
It is often useful to include explicit input nodes that do not perform calculations, known as input units or input neurons. These nodes act as placeholders to introduce input features into the network, passing data directly to the next layer without transformation. In the network diagram, these are the light blue nodes on the left, labeled 1 and 2. Typically, the number of input units corresponds to the number of features.
For clarity, nodes are labeled to facilitate discussion of the weights between them, such as w_{1,5} between nodes 1 and 5. Similarly, the output of a node is denoted by o_k, where k represents the node’s label. For example, for k=3, the output would be o_3.
Each computational unit performs two operations:
z_j=\sum_{i\to j} w_{i,j}o_i+b_j, \qquad o_j=\varphi(z_j).
The sum runs over the nodes connected to unit j. The weight w_{i,j} connects source node i to destination node j, and b_j is the bias of unit j. The input nodes pass the features unchanged: in the preceding diagram, o_1=x^{(1)} and o_2=x^{(2)}.
The quantity z_j is also called the preactivation. The output o_j is the activation; some references denote this output by a_j. The notation o_j is retained here to match the diagrams and the XOR example.
The symbol \varphi denotes a general activation function. When the activation is the logistic sigmoid, it is written \sigma. The summation symbol \sum in the left half of a unit represents aggregation, a different operation from the activation in the right half.
o_3 = \sigma(w_{1,3}x^{(1)}+w_{2,3}x^{(2)}+b_3)
o_4 = \sigma(w_{1,4}x^{(1)}+w_{2,4}x^{(2)}+b_4)
o_5 = \sigma(w_{1,5}x^{(1)}+w_{2,5}x^{(2)}+b_5)
o_6 = \sigma(w_{3,6}o_3+w_{4,6}o_4+w_{5,6}o_5+b_6)
o_7 = \sigma(w_{3,7}o_3+w_{4,7}o_4+w_{5,7}o_5+b_7)
Compute the hidden outputs first, then use them to compute the final outputs.
Every right-hand side first forms a weighted sum z_j, then applies the sigmoid to obtain o_j. The output units depend on the hidden units, so the hidden outputs must be available before the final outputs can be computed.
The diagram omits biases for clarity, but each computational unit has its own bias. A bias shifts the unit’s preactivation independently of its inputs. For a threshold unit, it moves the decision boundary away from the origin; for a sigmoid or ReLU unit, it shifts the position of the activation’s transition. Its role does not imply that training will automatically set an unnecessary bias to zero or that adding biases always makes training faster.
These two sigmoid outputs are separate values. A categorical distribution over mutually exclusive classes requires an output function such as the softmax used in the Fashion-MNIST example.
The same sequence of computations applies to a deep neural network, with more hidden layers.
Produced using NN-SVG, LeNail (2019).
The same network with bias inputs shown.
Produced using NN-SVG, LeNail (2019).
An affine transformation is a weighted sum plus a bias. For a one-dimensional example, composing h=w_1x+b_1 with y=w_2h+b_2 gives
y=(w_2w_1)x+(w_2b_1+b_2).
This is another affine transformation. The same reasoning applies to matrix-valued computations, so stacking affine layers alone does not give a network the ability to represent XOR in its original inputs. The hidden threshold functions in the XOR example supplied the necessary nonlinearity.
Activation functions have two distinct roles. Their nonlinearity affects what a network can represent; their derivatives affect how it can be trained. The classroom emphasis here is on computation and representation. Backpropagation and the detailed derivative comparisons are developed in Lecture 13.
A hard step has derivative zero away from the threshold and is not differentiable at the threshold. Sigmoid and tanh provide smooth alternatives. ReLU is differentiable except at zero, where implementations choose a convention; this isolated point does not prevent ordinary backpropagation in practice.
Nonlinearity and trainability are separate considerations: the hard threshold already supplies enough nonlinearity for the hand-configured XOR network, even though its ordinary derivatives are unsuitable for learning its parameters by gradient descent.
The universal approximation theorem (UAT) establishes that, on a compact input domain, a network with a sufficiently wide single hidden layer, sigmoid activations, and a linear output can approximate any specified continuous target function to any desired accuracy.
A compact set in \mathbb R^n is both closed and bounded. A closed interval such as [-4,2] is a simple example.
For every continuous target function on this domain and every positive error tolerance, the theorem guarantees a finite network whose error stays below that tolerance throughout the domain. The required width and parameter values can depend on the target and the tolerance (Cybenko 1989; Hornik et al. 1989).
This is an existence result about representation. It does not guarantee that a training algorithm will find the appropriate parameters, that the network will be small, or that learning from a finite sample will generalize well. It also gives no guarantee about extrapolation outside the specified domain.
The sigmoid version is a convenient classical statement. Broader results cover other activations, including the ReLU used in the visual code example. Nonlinearity alone is not a sufficient activation assumption. Leshno et al., Multilayer feedforward networks with a nonpolynomial activation function can approximate any function
Approximate the same target curve using networks with different numbers of hidden units.
# Defining the function to be approximated
def f(x):
return 2 * x**3 + 4 * x**2 - 5 * x + 1
# Generating a dataset, x in [-4,2), f(x) as above
approximation_rng = np.random.default_rng(42)
X = approximation_rng.uniform(-4, 2, size=(1000, 1))
y = f(X.flatten())y = f(x) = 2 x^3 + 4 x^2 - 5 x + 1
from sklearn.neural_network import MLPRegressor
from sklearn.model_selection import train_test_split
X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.1, random_state=42)
models = []
sizes = [1, 2, 3, 5, 50]
for n in sizes:
regressor = MLPRegressor(hidden_layer_sizes=(n,), max_iter=5000, random_state=42)
regressor.fit(X_train, y_train)
models.append(regressor) MLPRegressor is a multi-layer perceptron regressor from sklearn. Its default activation function is relu.
This video effectively conveys the underlying intuition of the universal approximation theorem. (18m 53s)
The video provides a supplementary visual explanation of how weighted activation outputs can combine to approximate a target curve. An output weight scales a hidden unit’s contribution and can reverse its sign. An input bias inside an activation, such as \sigma(wx+b), shifts its transition horizontally when w\ne0; a bias at a linear output shifts the combined curve vertically.
PyTorch, TensorFlow, and JAX provide tools for numerical computation and automatic differentiation.
Keras provides a high-level API for building and training models with these backends.
A framework supplies operations for computing model outputs and gradients efficiently. A high-level API supplies familiar interfaces for defining a model, fitting it to data, evaluating it, and making predictions.
The example uses Keras through tf.keras, with TensorFlow as its backend. The distinction between a network’s forward computation, gradient calculation, and optimizer updates remains the same across frameworks. Getting started with Keras
The main steps are:
Keras was created by François Chollet. Its Sequential API describes a stack of layers in which each layer receives the preceding layer’s output. It is suitable for this feedforward classifier.
The code that follows forms a complete workflow when executed in order. Softmax and categorical cross-entropy are introduced here through their roles in the API; their mathematical details are developed in Lecture 14.
Fashion-MNIST contains images of clothing and accessories:
Example adapted from Géron (2022), Hands-On Machine Learning, Chapter 10 notebook.
import tensorflow as tf
(X_train_full, y_train_full), (X_test, y_test) = (
tf.keras.datasets.fashion_mnist.load_data()
)
X_train, y_train = X_train_full[:-5000], y_train_full[:-5000]
X_valid, y_valid = X_train_full[-5000:], y_train_full[-5000:]Use 55,000 examples for training and 5,000 for validation. Reserve the supplied test set for final evaluation.
The validation examples are taken from the supplied training partition. They are used to monitor performance during development, but are not used to update the model’s parameters. The test partition remains separate from training and model selection, as discussed in Lecture 8.
X_train.shape, X_train.dtype((55000, 28, 28), dtype('uint8')). . .
Convert pixel intensities from 0–255 integers to 0–1 floating-point values in every split.
X_train = X_train.astype("float32") / 255.0
X_valid = X_valid.astype("float32") / 255.0
X_test = X_test.astype("float32") / 255.0The model must receive inputs in the same scale during training, validation, testing, and later prediction. Using raw 0–255 test pixels after training on 0–1 pixels changes the inputs to every weighted sum and invalidates the comparison.
Division by the known pixel bound 255 does not estimate a statistic from the data. Preprocessing that learns a mean, variance, or other statistic must instead be fitted on training data and applied to the other splits using the fitted transformation.
tf.keras.utils.set_random_seed(42)
model = tf.keras.Sequential([
tf.keras.Input(shape=(28, 28)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(300, activation="relu"),
tf.keras.layers.Dense(100, activation="relu"),
tf.keras.layers.Dense(10, activation="softmax")
])The Input specifies the shape of one example. Flatten rearranges the 28 × 28 pixels into 784 input features without learning any parameters. The two hidden Dense layers compute weighted sums plus biases, followed by ReLU activations.
The last Dense layer produces ten scores and applies softmax to form a probability distribution over the ten mutually exclusive classes. These output probabilities sum to 1. Hidden-layer activations and output activations serve different purposes.
The seed utility sets the Python, NumPy, and TensorFlow seeds for reproducibility; exact numerical results can still depend on the execution environment. Keras random-seed utility
model.summary()Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ flatten (Flatten) │ (None, 784) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense (Dense) │ (None, 300) │ 235,500 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 100) │ 30,100 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 10) │ 1,010 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 266,610 (1.02 MB)
Trainable params: 266,610 (1.02 MB)
Non-trainable params: 0 (0.00 B)
A Dense layer with n_{\mathrm{in}} inputs and n_{\mathrm{out}} units has (n_{\mathrm{in}}+1)n_{\mathrm{out}} trainable parameters, including biases.
There is one weight for every input-to-unit connection and one bias for every unit. The first hidden layer therefore has 784\times300+300=235{,}500 parameters. The second has 300\times100+100=30{,}100, and the output layer has 100\times10+10=1{,}010. The total is 266{,}610 trainable parameters. Flatten has none.
For a batch of m examples, the first hidden layer receives an m\times784 array, uses a 784\times300 weight matrix and a 300-element bias vector, and produces an m\times300 output. This follows the earlier convention of one row per example and one column per unit in the weight matrix.
model.compile(
loss="sparse_categorical_crossentropy",
optimizer=tf.keras.optimizers.SGD(learning_rate=0.01),
metrics=["accuracy"]
)Loss: compare the predicted distribution with the integer target label.
Optimizer: update parameters using gradients. Accuracy: report correct class predictions.
Sparse categorical cross-entropy is appropriate because each example has one integer class label and the model outputs a categorical probability distribution. With one-hot encoded targets, the corresponding loss is categorical cross-entropy. The loss here receives probabilities from softmax, rather than raw logits. Keras probabilistic losses
The compile method configures training; it does not fit the parameters. Stochastic gradient descent uses gradients computed by backpropagation to update the weights and biases. The learning rate controls the size of those updates. Accuracy is reported as a metric, while the differentiable loss supplies the training objective.
history = model.fit(
X_train, y_train, epochs=30, batch_size=32,
validation_data=(X_valid, y_valid), verbose=2
)Epoch 1/30
1719/1719 - 2s - 874us/step - accuracy: 0.7616 - loss: 0.7239 - val_accuracy: 0.8298 - val_loss: 0.5026
Epoch 2/30
1719/1719 - 1s - 715us/step - accuracy: 0.8299 - loss: 0.4894 - val_accuracy: 0.8418 - val_loss: 0.4472
Epoch 3/30
1719/1719 - 1s - 715us/step - accuracy: 0.8448 - loss: 0.4434 - val_accuracy: 0.8480 - val_loss: 0.4233
Epoch 4/30
1719/1719 - 1s - 693us/step - accuracy: 0.8548 - loss: 0.4154 - val_accuracy: 0.8550 - val_loss: 0.4088
Epoch 5/30
1719/1719 - 1s - 707us/step - accuracy: 0.8607 - loss: 0.3951 - val_accuracy: 0.8584 - val_loss: 0.3970
Epoch 6/30
1719/1719 - 1s - 702us/step - accuracy: 0.8667 - loss: 0.3790 - val_accuracy: 0.8594 - val_loss: 0.3889
Epoch 7/30
1719/1719 - 1s - 704us/step - accuracy: 0.8713 - loss: 0.3656 - val_accuracy: 0.8616 - val_loss: 0.3824
Epoch 8/30
1719/1719 - 1s - 710us/step - accuracy: 0.8748 - loss: 0.3539 - val_accuracy: 0.8642 - val_loss: 0.3780
Epoch 9/30
1719/1719 - 1s - 705us/step - accuracy: 0.8783 - loss: 0.3432 - val_accuracy: 0.8666 - val_loss: 0.3740
Epoch 10/30
1719/1719 - 1s - 720us/step - accuracy: 0.8817 - loss: 0.3336 - val_accuracy: 0.8678 - val_loss: 0.3684
Epoch 11/30
1719/1719 - 1s - 719us/step - accuracy: 0.8851 - loss: 0.3247 - val_accuracy: 0.8702 - val_loss: 0.3641
Epoch 12/30
1719/1719 - 1s - 695us/step - accuracy: 0.8875 - loss: 0.3165 - val_accuracy: 0.8726 - val_loss: 0.3601
Epoch 13/30
1719/1719 - 1s - 701us/step - accuracy: 0.8898 - loss: 0.3089 - val_accuracy: 0.8736 - val_loss: 0.3578
Epoch 14/30
1719/1719 - 1s - 716us/step - accuracy: 0.8922 - loss: 0.3017 - val_accuracy: 0.8724 - val_loss: 0.3554
Epoch 15/30
1719/1719 - 1s - 720us/step - accuracy: 0.8942 - loss: 0.2951 - val_accuracy: 0.8736 - val_loss: 0.3543
Epoch 16/30
1719/1719 - 1s - 688us/step - accuracy: 0.8962 - loss: 0.2887 - val_accuracy: 0.8734 - val_loss: 0.3520
Epoch 17/30
1719/1719 - 1s - 682us/step - accuracy: 0.8983 - loss: 0.2829 - val_accuracy: 0.8738 - val_loss: 0.3522
Epoch 18/30
1719/1719 - 1s - 692us/step - accuracy: 0.9002 - loss: 0.2771 - val_accuracy: 0.8746 - val_loss: 0.3495
Epoch 19/30
1719/1719 - 1s - 734us/step - accuracy: 0.9021 - loss: 0.2717 - val_accuracy: 0.8758 - val_loss: 0.3478
Epoch 20/30
1719/1719 - 1s - 727us/step - accuracy: 0.9043 - loss: 0.2664 - val_accuracy: 0.8760 - val_loss: 0.3465
Epoch 21/30
1719/1719 - 1s - 729us/step - accuracy: 0.9060 - loss: 0.2615 - val_accuracy: 0.8754 - val_loss: 0.3465
Epoch 22/30
1719/1719 - 1s - 753us/step - accuracy: 0.9081 - loss: 0.2565 - val_accuracy: 0.8772 - val_loss: 0.3449
Epoch 23/30
1719/1719 - 1s - 709us/step - accuracy: 0.9102 - loss: 0.2519 - val_accuracy: 0.8782 - val_loss: 0.3454
Epoch 24/30
1719/1719 - 1s - 724us/step - accuracy: 0.9121 - loss: 0.2471 - val_accuracy: 0.8780 - val_loss: 0.3445
Epoch 25/30
1719/1719 - 1s - 737us/step - accuracy: 0.9145 - loss: 0.2427 - val_accuracy: 0.8790 - val_loss: 0.3437
Epoch 26/30
1719/1719 - 1s - 704us/step - accuracy: 0.9158 - loss: 0.2384 - val_accuracy: 0.8792 - val_loss: 0.3446
Epoch 27/30
1719/1719 - 1s - 698us/step - accuracy: 0.9172 - loss: 0.2341 - val_accuracy: 0.8810 - val_loss: 0.3445
Epoch 28/30
1719/1719 - 1s - 710us/step - accuracy: 0.9193 - loss: 0.2299 - val_accuracy: 0.8804 - val_loss: 0.3441
Epoch 29/30
1719/1719 - 1s - 704us/step - accuracy: 0.9210 - loss: 0.2260 - val_accuracy: 0.8808 - val_loss: 0.3445
Epoch 30/30
1719/1719 - 1s - 711us/step - accuracy: 0.9223 - loss: 0.2220 - val_accuracy: 0.8800 - val_loss: 0.3440A training step processes one mini-batch and updates the parameters once. An epoch is one complete pass through the training examples. The batch size and learning rate connect this example to the stochastic and mini-batch gradient-descent methods introduced in Lecture 4.
With this call, Keras evaluates the validation set at the end of each epoch without updating parameters on it. The returned History stores the training and validation loss and accuracy for each epoch. Keras training API
Training metrics aggregate mini-batch results while the parameters change during an epoch; validation metrics use the parameters at the end of that epoch. Their curves are therefore not evaluated under exactly the same conditions. Validation results can guide model and hyperparameter choices; the test set is reserved for the final evaluation.
test_loss, test_accuracy = model.evaluate(X_test, y_test, verbose=0)
print(f"Test loss: {test_loss:.3f}")
print(f"Test accuracy: {test_accuracy:.3f}")Test loss: 0.362
Test accuracy: 0.874The test set estimates performance on held-out examples after the model and training configuration have been chosen. Its inputs receive the same scaling as the training and validation inputs. Test results should not be used to select the architecture or the number of epochs.
X_new = X_test[:3]
y_proba = model.predict(X_new, verbose=0)
y_proba.round(2)array([[0. , 0. , 0. , 0. , 0. , 0.33, 0. , 0.01, 0. , 0.65],
[0. , 0. , 1. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],
[0. , 1. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]],
dtype=float32). . .
y_pred = y_proba.argmax(axis=1)
y_predarray([9, 2, 1]). . .
y_observed = y_test[:3]
y_observedarray([9, 2, 1], dtype=uint8)The predicted class has the highest output probability. Compare it with the observed class.
Each row of y_proba contains ten class probabilities for one image. argmax(axis=1) selects the index of the largest probability in each row. The integer indices correspond to the entries in class_names.
A high output probability expresses the model’s confidence in its prediction; it does not establish that the prediction is correct. The observed labels are needed to assess these examples.
from sklearn.metrics import classification_report
y_test_proba = model.predict(X_test, verbose=0)
y_test_pred = y_test_proba.argmax(axis=1)
print(classification_report(y_test, y_test_pred,
target_names=class_names, digits=3)) precision recall f1-score support
T-shirt/top 0.842 0.810 0.826 1000
Trouser 0.992 0.964 0.978 1000
Pullover 0.744 0.850 0.793 1000
Dress 0.824 0.929 0.873 1000
Coat 0.806 0.794 0.800 1000
Sandal 0.885 0.986 0.933 1000
Shirt 0.777 0.618 0.689 1000
Sneaker 0.944 0.911 0.927 1000
Bag 0.961 0.963 0.962 1000
Ankle boot 0.983 0.918 0.949 1000
accuracy 0.874 10000
macro avg 0.876 0.874 0.873 10000
weighted avg 0.876 0.874 0.873 10000
Overall accuracy can hide differences between classes. The classification report gives the precision, recall, F1 score, and support for each clothing category, using the definitions introduced in Lecture 7. The readable class names make it easier to identify which categories are most difficult for this model.
Marcel Turcotte
School of Electrical Engineering and Computer Science (EECS)
University of Ottawa