Homework 1

Overview

Four analytic problems on the forward pass of a small MLP, partial derivatives, hyperplane geometry, and convexity, followed by two Monte Carlo simulations in pure Python.

Getting Started

Download the starter: hw1-starter.zip

unzip hw1-starter.zip
cd hw1-starter

It contains one directory per problem and no code.

Problem 1: MLP Forward Pass

An MLP has two input nodes, one hidden layer, and two outputs. The two sets of weights and biases are given by:

The non-linear activation for the hidden layer is ReLU (rectified linear unit) – that is \(h(x) = \max(x, 0)\). The output layer is linear (i.e., identity activation function). The output for layer \(l\) is given by \(a^{(l)} = h_l(W_l a^{(l-1)} + b_l)\).

What is the output activation for input \(x = \begin{bmatrix} +1 \\ -1 \end{bmatrix}\)?

Deliverables

See Submission. q1.pdf contains your calculations and the final output activation.

Problem 2: Partial Derivatives and Minimization

Let \(f(x, y) = 4x^2 + y^2 - xy - 13x\)

a. Find \(\frac{\partial f}{\partial x}\), the partial derivative of \(f\) with respect to \(x\). Find \(\frac{\partial f}{\partial y}\).

b. Find \((x, y) \in \mathbb{R}^2\) that minimizes \(f\).

Deliverables

See Submission. q2.pdf contains your worked solution, typeset or scanned. Show all work and box final answers.

Problem 3: Distance to a Hyperplane

A hyper-plane in \(\mathbb{R}^n\) is the set, \(\{ \mathbf{x} : \mathbf{x} \in \mathbb{R}^n, w^T x + b = 0 \}\), where \(w \in \mathbb{R}^n\) and \(b\) is a real scalar.

a. The solution of the following optimization problem describes the distance between a point \(x_0 \in \mathbb{R}^n\) and the hyperplane \(w^T x + b = 0\):

Derive an analytic solution for the distance between \(x_0\) and \(w^T x + b = 0\).

b. What is the distance between two hyperplanes, \(w^T x + b_1 = 0\) and \(w^T x + b_2 = 0\)?

Deliverables

See Submission. q3.pdf contains your worked solution, typeset or scanned. Show all work and box final answers.

Problem 4: Convexity

A function \(f(x)\) is convex if

for all \(x, y\) and \(0 < \lambda < 1\).

a. Use this definition to prove that \(f(x) = x^2\) is a convex function. Verify that \(f(x) = x^3\) is not a convex function.

b. An \(n \times n\) matrix \(A\) is positive semi-definite if \(\mathbf{x}^T A \mathbf{x} \geq 0\) for all \(\mathbf{x} \in \mathbb{R}^n\). Prove that the function \(f(\mathbf{x}) = \mathbf{x}^T A \mathbf{x}\) is convex if \(A\) is positive semi-definite.

Deliverables

See Submission. q4.pdf contains your worked solution, typeset or scanned. Show all work and box final answers.

Problem 5: Biased Coin Simulation

Simulate tossing a biased coin (a Bernoulli trial) where \(P[\text{HEAD}] = 0.70\).

a. Count the number of heads in 50 trials. Record the longest run of heads.

b. Repeat the 50-flip experiment 20, 100, 200, and 1000 times. Use matplotlib to generate a histogram showing the observed number of heads for each case. Comment on the limit of the histogram.

c. Simulate tossing the coin 500 times. Generate a histogram showing the heads run lengths.

Deliverables

See Submission. coin_sim.py is your simulation. q5.pdf contains the histograms and your comments for each part.

Problem 6: Sum of Uniforms Exceeding a Threshold

Define the random variable \(N = \min \{ n: \sum_{i=1}^{n} X_i > 4\}\) as the smallest number of standard uniform random samples whose sum exceeds four. Generate a histogram using 100, 1000, and 10000 realizations of \(N\). Comment on the expected value \(E[N]\).

Deliverables

See Submission. uniform_sum.py is your simulation. q6.pdf contains the histograms and your comment on \(E[N]\).

README.md
.gitignore
q1/
└── q1.pdf
q2/
└── q2.pdf
q3/
└── q3.pdf
q4/
└── q4.pdf
q5/
├── coin_sim.py
└── q5.pdf
q6/
├── uniform_sum.py
└── q6.pdf