Neural Networks

AI had been really influencing our daily life, It’s a powerful tool to help us with a lot of tasks. As of 2024 There are prototypes of self driving cars to humanoid robots that does chores.

Neural Network is the core of the whole Machiene Learning and AI all around us. Let’s break down Neural Networks with examples in pytorch.

What is a Neural Network?

Neural Network is a programmable network of neurons called nodes with layers to compute and return few outputs. It’s a model that can be trained to recognize patterns in data.

Structure of a Neural Network

Layers

A neural network is made up of layers, each layer is a collection of nodes. There are mainly 3 types of layers in a neural network.

1. Input Layer
2. Hidden Layer
3. Output Layer

example code in pytorch

import torch
import torch.nn as nn

class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.input_layer = nn.Linear(2, 4)
        self.hidden_layer = nn.Linear(4, 1)
        self.output_layer = nn.Sigmoid()

There are different types of layers to choose from

The numbers in the brackets are the parameters that the layer takes.

eg:

• nn.Linear(2, 4) means the layer takes 2 inputs and gives 4 outputs.
• nn.Conv2d(3, 16, 3, 1) means the layer takes 3 input channels and gives 16 output channels with a kernel size of 3 and stride of 1.

Activation Functions

Activation functions are used to introduce non-linearity to the network. There are different types of activation functions to choose from.

1. ReLU (Rectified Linear Unit) - Most commonly used activation function. eg: nn.ReLU()
2. Sigmoid - Used in the output layer for binary classification. eg: nn.Sigmoid()
3. Softmax - Used in the output layer for multi-class classification. eg: nn.Softmax()

example code in pytorch

import torch
import torch.nn as nn

class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.input_layer = nn.Linear(2, 4)
        self.hidden_layer = nn.Linear(4, 1)
        self.output_layer = nn.Sigmoid()

    def forward(self, x):
        x = self.input_layer(x)
        x = nn.ReLU(x) # Activation Function
        x = self.hidden_layer(x)
        x = self.output_layer(x)
        return x

The forward method is used to define the forward pass of the network. Forward pass defines the flow of input data through the layers of the network to get the output.

Steps To Train a Neural Network

Common steps to train a neural network are:

1. Forward Pass - Pass the input data through the layers of the network to get the output.
2. Calculate Loss - Calculate the difference between the predicted output and the actual output.
3. Backward Pass - Calculate the gradients of the loss with respect to the weights of the network.
4. Update Weights - Update the weights of the network using an optimization algorithm like Adam or SGD.

graph LR
    A[Input Data] --> B[Forward Pass]
    B --> C[Calculate Loss]
    C --> D[Backward Pass]
    D --> E[Update Weights]

example code in pytorch

import torch
import torch.nn as nn
import torch.optim as optim

class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.input_layer = nn.Linear(2, 4)
        self.hidden_layer = nn.Linear(4, 1)
        self.output_layer = nn.Sigmoid()

    def forward(self, x):
        x = self.input_layer(x)
        x = nn.ReLU(x) # Activation Function
        x = self.hidden_layer(x)
        x = self.output_layer(x)
        return x

# Simple Training function
def train(model, optimizer, criterion, x, y):
    optimizer.zero_grad()       # Zero the gradients
    output = model(x)           # Forward Pass
    loss = criterion(output, y) # Calculate Loss
    loss.backward()             # Backward Pass
    optimizer.step()            # Update Weights

model = NeuralNetwork()     # Create the model
optimizer = optim.Adam(model.parameters(), lr=0.001) # Create the optimizer with learning rate 0.001
criterion = nn.BCELoss()    # Create the loss function

x = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=torch.float32) # Input Data
y = torch.tensor([[0], [1], [1], [0]], dtype=torch.float32) # Actual Output

for epoch in range(1000):
    train(model, optimizer, criterion, x, y)
    print(f'Epoch: {epoch+1}, Loss: {criterion(model(x), y).item()}')

new_data = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=torch.float32)
predictions = model(new_data)
print(predictions)

Forward Pass

In the forward pass, we simply pass the input data through the layers of the network to get the output.

When starting, the weights and biases of the network are randomly initialized which will be updated during the training process.

where, \(W\) is the weight matrix, \(x\) is the input data, \(b\) is the bias, \(z\) is the output of the linear layer, \(\sigma\) is the activation function and \(a\) is the output of the layer.

output = model(x)

Calculate Loss

In the loss calculation step, we calculate the difference between the predicted output and the actual output.

There are different types of loss functions to choose from based on the problem we are trying to solve.

example: Mean Squared Error Loss

where, \(L\) is the loss, \(N\) is the number of samples, \(y_{pred}\) is the predicted output and \(y_{true}\) is the actual output.

1. Binary Cross Entropy Loss - Used for binary classification problems. eg: nn.BCELoss()
2. Mean Squared Error Loss - Used for regression problems. eg: nn.MSELoss()
3. Cross Entropy Loss - Used for multi-class classification problems. eg: nn.CrossEntropyLoss()

criterion = nn.BCELoss()

loss = criterion(output, y)

Backward Pass

In the backward pass, we calculate the gradients of the loss for weight and bias parameters of the network.

This is where the magical adjustment of weights and biases happens.

example:

where, \(L\) is the loss, \(W\) is the weight matrix, \(b\) is the bias, \(a\) is the output of the layer, \(z\) is the output of the linear layer.

loss.backward()

Update Weights

The optimization algorithm is used to update the weights and biases of the network.

The optimization depends on the gradients calculated in the backward pass and the learning rate.

example:

where, \(W_{new}\) is the new weight, \(W_{old}\) is the old weight, \(b_{new}\) is the new bias, \(b_{old}\) is the old bias, \(\alpha\) is the learning rate.

optimizer = optim.Adam(model.parameters(), lr=0.001)

optimizer.step()

Training Loop

The training loop is where we put all the steps together to train the neural network.

for epoch in range(1000):
    train(model, optimizer, criterion, x, y)
    print(f'Epoch: {epoch+1}, Loss: {criterion(model(x), y).item()}')

Evaluation

After training the network, we can use it to make predictions on new data.

new_data = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=torch.float32)
predictions = model(new_data)
print(predictions)

The training and evaluation can be further improved by splitting the data into training and validation sets.

Summary

1. Neural Networks are a class of machine learning models inspired by the human brain.
2. Neural Networks can be used for a wide range of tasks like classification, regression, and more.
3. A neural network consists of layers of neurons that process input data to make predictions.
4. The training process involves forward pass, loss calculation, backward pass, and weight update.
5. PyTorch provides a simple and efficient way to create and train neural networks.