Posts

In the first part of these series we began by a brief review of essential mathematical concepts to helps us build a foundation for understanding backpropagation using matrix derivatives. In this post we will express our neural network model in matrix form and symbolically derive its backpropagation equations using gradient descent. A simple neuron model In the first part, we observed that our neural model is represented as follows; $\displaystyle \hat{y} = f(w*x+b)$ ...

A simple neuron with a single input can be represented mathematically as; $\displaystyle \hat{y} = f(w*x +b)$ $\displaystyle L = MSE(y, \hat{y})$ Here, the $\displaystyle f$ is an activation function that transforms its input into some class label. MSE stands for Mean Squared Error, and it’s a function that measures the difference between the predicted output $\displaystyle \hat{y}$ and the actual output $\displaystyle y$. The goal of this series of posts is to express a neural model using matrix notation and apply matrix calculus to derive the backpropagation algorithm using gradient descent. ...

Backward propagation or more precisely Error Backward Propagation is a technique in neural networks used to optimize a neural network. An output error is propagated backward within the network and as the error propagates the network parameter are adjusted with the aim to reduce resulting output error. Our Model: Consider the following graph of our model of the neuron; graph LR A@{shape: circ, label: "w"} --> B@{shape: rect, label: "g(.)"} --> C@{shape: circ, label: "y"} --> D@{shape: rect, label: "h(.)"} --> E@{shape: circ, label: "f"} F@{shape: circ, label: "b"} --> B In this model; ...

Wikipedia defines Automatic Differentiation (AD) as a technique to evaluate partial derivative of a function. I will not repeat the definition, but I will quote an important point that’s relevant to what we are doing; … By applying the chain rule repeatedly to these operations, partial derivatives of arbitrary order can be computed automatically … The take away here is that with AD; we breaks down a long expression into small sub-expressions. ...

Computational graph is a directed graph used to represent a sequence of operations to some input data, until its eventual output. This sequence of operations could be any form of transformations or manipulation to the input data. In this article we will restrict ourselves to those operations that involve mathematical expressions. The key elements of a computational graph are; a node and an edge Since the graph is used to represent data flow along a sequence of operations, I will expand the concept of a computational graph to also incorporate an additional element; rules. The significance of this addition will become apparent as we proceed. Thus, for our purpose a computational graph will be composed of the following elements; ...

The standard formula for a cirlce is $\displaystyle x^2+y^2=r^2$. This can also be written as $\displaystyle x^2+y^2-r^2=0$ or more properly as $\displaystyle C(x,y)=x^2+y^2-r^2=0$. There are situations where the parametric form of the circle is useful. Here, I will show how to derive the parametric form of the circle using angles on the circle and a parameter $\displaystyle t$. Given a circle $\displaystyle C$, with radius $r=1$; we can derive its parametric form in 2 ways; ...