Automatic Differentiation Engine (Autograd)

1Concept

PyTorch Autograd automatically computes partial derivatives (gradients) during the backward pass. Setting `requires_grad=True` builds a dynamic computational graph; calling `loss.backward()` computes gradients stored in `tensor.grad`.

2Architecture Diagram

Forward Pass:  x -> [ Node: y = 2x^2 ] -> Loss
Backward Pass: loss.backward() ---> Computes dy/dx = 4x stored in x.grad!

3Code Example

autograd_demo = '''
import torch

# Parameter requiring gradient calculation
w = torch.tensor([3.0], requires_grad=True)
b = torch.tensor([1.0], requires_grad=True)

# Forward pass: y = w * x + b
x = torch.tensor([2.0])
y = w * x + b
loss = (y - 10.0) ** 2 # Loss = (3*2 + 1 - 10)^2 = (-3)^2 = 9.0

# Backward pass: compute gradients
loss.backward()

print(f"Gradient d(loss)/dw: {w.grad.item()}") # dLoss/dw = 2*(y-10)*x = 2*(-3)*2 = -12.0
'''
print("=== Autograd Computational Graph ===")
print(autograd_demo.strip())

4Expected Output

=== Autograd Computational Graph ===
import torch

# Parameter requiring gradient calculation
w = torch.tensor([3.0], requires_grad=True)
b = torch.tensor([1.0], requires_grad=True)

# Forward pass: y = w * x + b
x = torch.tensor([2.0])
y = w * x + b
loss = (y - 10.0) ** 2 # Loss = (3*2 + 1 - 10)^2 = (-3)^2 = 9.0

# Backward pass: compute gradients
loss.backward()

print(f"Gradient d(loss)/dw: {w.grad.item()}") # dLoss/dw = 2*(y-10)*x = 2*(-3)*2 = -12.0

5Key Takeaways

• ✓Use `with torch.no_grad():` during inference to disable graph building and save GPU memory.
• ✓Gradients accumulate by default; call `optimizer.zero_grad()` before each backward pass.
• ✓`tensor.detach()` severs a tensor from the computation graph.