blog.dopana

Back

Neural networks power many of the AI applications we use daily—from image recognition to language translation. But at their core, they’re just mathematical functions that learn patterns from data.

Understanding how to build a basic neural network from scratch is one of the best ways to grasp the fundamentals of deep learning. You’ll see exactly what happens behind the scenes when you train a model, without relying on black-box libraries that hide the details.

In this tutorial, we’ll build a simple neural network using Python and NumPy, step by step. No prior deep learning experience required—just basic Python knowledge.

What Is a Neural Network?#

Imagine your brain has billions of tiny cells called neurons that communicate with each other. When you see a cat, specific neurons fire together to recognize the pattern: pointy ears, whiskers, fur. This pattern recognition happens automatically through experience.

An artificial neural network mimics this idea using mathematical operations. It consists of:

  • Neurons: Simple processing units that receive input, do some math, and produce output
  • Layers: Groups of neurons arranged in sequences (input layer, hidden layers, output layer)
  • Weights: Numbers that determine how important each input is (like the strength of a connection between neurons)
  • Biases: Numbers that help the network make decisions independently of the input
  • Activation functions: Mathematical functions that add non-linearity, allowing the network to learn complex patterns

[!NOTE] A neural network is essentially a chain of mathematical operations that transforms input data into predictions.

How a Single Neuron Works#

Let’s start with the simplest building block: a single neuron.

Think of a neuron like a decision-making machine. It takes several inputs, multiplies each by a weight (importance), adds them together with a bias, then passes the result through an activation function to decide whether to “fire” (output a value).

Here’s the math:

output = activation(sum(inputs × weights) + bias)
text

Let’s build one in Python:

What happened?

  • Input: [2, 3]
  • Weighted sum: (2 × 0) + (3 × 1) + 4 = 7
  • Sigmoid(7) ≈ 0.999 (squashed to nearly 1)

The sigmoid function is just one type of activation function. It’s useful for classification because it outputs values between 0 and 1, which we can interpret as probabilities.

Building a Simple Neural Network#

A single neuron is limited—it can only learn simple patterns. To solve more complex problems, we connect multiple neurons into layers.

Let’s build a network with:

  • 2 input neurons
  • 1 hidden layer with 2 neurons
  • 1 output neuron

This network takes 2 inputs, processes them through a hidden layer, and produces 1 output. But there’s a problem: the weights are random! The network has no idea what it’s doing.

Training a Neural Network#

Training is the process of adjusting weights and biases so the network makes accurate predictions. Here’s how it works:

  1. Forward pass: Pass input through the network to get a prediction
  2. Calculate loss: Compare prediction to the actual answer (how wrong are we?)
  3. Backward pass (backpropagation): Calculate how much each weight contributed to the error
  4. Update weights: Adjust weights slightly to reduce the error
  5. Repeat: Do this many times until the network learns

[!TIP] This is called “gradient descent”—we’re slowly walking down a hill (the error) to find the lowest point (the best weights).

Let’s implement training for a simple task: classify numbers as either “small” (< 0.5) or “large” (≥ 0.5).

After training, the network should predict values close to 0 for small numbers and close to 1 for large numbers.

Understanding the Components#

Activation Functions#

Activation functions introduce non-linearity, allowing neural networks to learn complex patterns. Without them, a neural network would just be a linear regression model.

Common activation functions:

  • Sigmoid: Outputs between 0 and 1 (good for binary classification)
  • ReLU: Outputs 0 for negative inputs, passes positive inputs unchanged (most common in hidden layers)
  • Softmax: Converts outputs to probabilities (good for multi-class classification)

Loss Functions#

The loss function measures how wrong the network’s predictions are.

  • Mean Squared Error (MSE): Average of squared differences between predictions and targets (for regression)
  • Cross-Entropy Loss: Measures difference between predicted and actual probability distributions (for classification)

Learning Rate#

The learning rate controls how big of a step we take when updating weights.

  • Too high: Might overshoot the optimal solution
  • Too low: Training takes forever
  • Typical values: 0.001 to 0.1

Moving to Real-World Applications#

Building a neural network from scratch is great for learning, but in practice, you’ll use deep learning frameworks like:

  • TensorFlow/Keras: Industry-standard, lots of tutorials
  • PyTorch: Popular in research, Pythonic design
  • JAX: High-performance, composable

Here’s the same simple network using Keras:

Keras handles all the math, optimization, and bookkeeping for you. But now you understand what’s happening under the hood.

Common Challenges#

When building neural networks, you’ll encounter several challenges:

Vanishing Gradients#

In deep networks, gradients can become extremely small during backpropagation, making early layers learn very slowly. Solutions: use ReLU activation, careful weight initialization, or residual connections.

Overfitting#

The network memorizes training data but fails to generalize to new data. Solutions: add dropout, use more training data, apply regularization, or early stopping.

Choosing Architecture#

How many layers? How many neurons? There’s no formula—experimentation is key. Start simple, then add complexity if needed.

Next Steps#

Now that you understand the basics, you can:

  1. Experiment with different architectures: Try deeper networks, different activation functions
  2. Work with real datasets: MNIST (handwritten digits), CIFAR-10 (images), IMDB (text)
  3. Learn advanced techniques: Convolutional Neural Networks (CNNs) for images, Recurrent Neural Networks (RNNs) for sequences
  4. Explore frameworks: Dive deeper into TensorFlow, PyTorch, or JAX

Conclusion#

Building a neural network from scratch demystifies the “magic” of deep learning. You’ve seen that at its core, it’s just math—multiplication, addition, and function composition—arranged in a clever way to learn patterns from data.

The concepts you’ve learned—neurons, layers, activation functions, forward/backward propagation—are the foundation of all modern AI systems. Whether you’re building image classifiers, language models, or recommendation systems, these fundamentals remain the same.

Start simple, experiment often, and don’t be afraid to break things. That’s how you learn.

References#