AmberLJC/activation_functions

🧠 Activation Functions: Deep Neural Network Analysis

Empirical evidence for the vanishing gradient problem and why modern activations (ReLU, GELU) dominate deep learning.

This repository provides a comprehensive comparison of 5 activation functions in deep neural networks, demonstrating the vanishing gradient problem with Sigmoid and why modern activations enable training of deep networks.

---

🎯 Key Findings

Activation	Final MSE	Gradient Ratio (L10/L1)	Status
ReLU	0.008	1.93 (stable)	✅ Excellent
Leaky ReLU	0.008	0.72 (stable)	✅ Excellent
GELU	0.008	0.83 (stable)	✅ Excellent
Linear	0.213	0.84 (stable)	⚠️ Cannot learn non-linearity
Sigmoid	0.518	2.59×10⁷ (vanishing!)	❌ Failed

🔬 The Vanishing Gradient Problem - Visualized

Sigmoid Network (10 layers):
Layer 1  ████████████████████████████████████████  Gradient: 5.04×10⁻¹
Layer 5  ████████████                              Gradient: 1.02×10⁻⁴  
Layer 10 ▏                                         Gradient: 1.94×10⁻⁸  ← 26 MILLION times smaller!

ReLU Network (10 layers):
Layer 1  ████████████████████████████████████████  Gradient: 2.70×10⁻³
Layer 5  ██████████████████████████████████████    Gradient: 2.10×10⁻³
Layer 10 ████████████████████████████████████████  Gradient: 1.36×10⁻³  ← Healthy flow!

---

📊 Visual Results

Learned Functions

ReLU, Leaky ReLU, and GELU perfectly approximate the sine wave. Linear learns only a straight line. Sigmoid completely fails to learn.

Training Dynamics

Gradient Flow Analysis

Comprehensive Summary

---

🧪 Experimental Setup

Architecture

• Network: 10 hidden layers × 64 neurons each
• Task: 1D non-linear regression (sine wave approximation)
• Dataset: y = sin(x) + ε, where x ∈ [-π, π] and ε ~ N(0, 0.1)

Training Configuration

optimizer = Adam(lr=0.001)
loss_fn = MSELoss()
epochs = 500
batch_size = full_batch (200 samples)
seed = 42

Activation Functions Tested

Function	Formula	Gradient Range
Linear	f(x) = x	Always 1
Sigmoid	f(x) = 1/(1+e⁻ˣ)	(0, 0.25]
ReLU	f(x) = max(0, x)	{0, 1}
Leaky ReLU	f(x) = max(0.01x, x)	{0.01, 1}
GELU	f(x) = x·Φ(x)	Smooth, ~(0, 1)

---

🚀 Quick Start

Installation

git clone https://huggingface.co/AmberLJC/activation_functions
cd activation_functions
pip install torch numpy matplotlib

Run the Experiment

# Basic 5-activation comparison
python train.py

# Extended tutorial with 8 activations and 4 experiments
python tutorial_experiments.py

# Training dynamics analysis
python train_dynamics.py

---

📁 Repository Structure

activation_functions/
├── README.md                          # This file
├── report.md                          # Detailed analysis report
├── activation_tutorial.md             # Educational tutorial
│
├── train.py                           # Main experiment (5 activations)
├── tutorial_experiments.py            # Extended experiments (8 activations)
├── train_dynamics.py                  # Training dynamics analysis
│
├── learned_functions.png              # Predictions vs ground truth
├── loss_curves.png                    # Training loss over epochs
├── gradient_flow.png                  # Gradient magnitude per layer
├── hidden_activations.png             # Activation patterns
├── summary_figure.png                 # 9-panel comprehensive summary
│
├── exp1_gradient_flow.png             # Extended gradient analysis
├── exp2_activation_distributions.png  # Activation distribution analysis
├── exp2_sparsity_dead_neurons.png     # Sparsity and dead neuron analysis
├── exp3_stability.png                 # Training stability analysis
├── exp4_predictions.png               # Function approximation comparison
├── exp4_representational_heatmap.png  # Representational capacity heatmap
│
├── activation_evolution.png           # Activation evolution during training
├── gradient_evolution.png             # Gradient evolution during training
├── training_dynamics_functions.png    # Training dynamics visualization
├── training_dynamics_summary.png      # Training dynamics summary
│
├── loss_histories.json                # Raw loss data
├── gradient_magnitudes.json           # Gradient measurements
├── gradient_magnitudes_epochs.json    # Gradient evolution data
├── exp1_gradient_flow.json            # Extended gradient data
└── final_losses.json                  # Final MSE per activation

---

📖 Key Insights

Why Sigmoid Fails in Deep Networks

The vanishing gradient problem occurs because:

1. Sigmoid derivative is bounded: max(σ'(x)) = 0.25 at x=0
2. Chain rule multiplies gradients: For 10 layers, gradient ≈ (0.25)¹⁰ ≈ 10⁻⁶
3. Early layers don't learn: Gradient signal vanishes before reaching input layers

# Theoretical gradient decay for Sigmoid
gradient_layer_10 = gradient_output * (0.25)^10
                  ≈ gradient_output * 0.000001
                  ≈ 0  # Effectively zero!

Why ReLU Works

ReLU maintains unit gradient for positive inputs:

# ReLU gradient
f'(x) = 1 if x > 0 else 0

# No multiplicative decay!
gradient_layer_10 ≈ gradient_output * 1^10 = gradient_output

Practical Recommendations

Use Case	Recommended
Default choice	ReLU or Leaky ReLU
Transformers/LLMs	GELU
Very deep networks	Leaky ReLU + skip connections
Output (classification)	Sigmoid/Softmax
Output (regression)	Linear

---

📚 Extended Experiments

The tutorial_experiments.py script includes 4 additional experiments:

1. Gradient Flow Analysis - Depths 5, 10, 20, 50 layers
2. Activation Distributions - Sparsity and dead neuron analysis
3. Training Stability - Learning rate and depth sensitivity
4. Representational Capacity - Multiple target function approximation

---

🔗 References

• Deep Learning Book - Chapter 6.3: Hidden Units
• Glorot & Bengio (2010): Understanding the difficulty of training deep feedforward neural networks
• He et al. (2015): Delving Deep into Rectifiers
• Hendrycks & Gimpel (2016): GELU

---

📄 Citation

@misc{activation_functions_analysis,
  title={Activation Functions: Deep Neural Network Analysis},
  author={Orchestra Research},
  year={2024},
  publisher={HuggingFace},
  url={https://huggingface.co/AmberLJC/activation_functions}
}

---

📜 License

MIT License - feel free to use for education and research!

---

Generated by Orchestra Research Assistant