# MATH 5312: Mathematical Statistics I

Fall 2016

## Table of Contents

## Course Syllabus

## Course Schedule

The due date of HW 10 has changed to 11/29.

The instructor reserves the right to adjust this schedule in any way that serves the educational needs of the students enrolled in this course.

Week | Date | Topics | Reading | Notes |
---|---|---|---|---|

Week1 | 8/25 | Introduction | ||

Week2 | 8/30 | Probability set function | HMC 1.2-1.3 | HW 1 - Due 9/6 in class |

Week2 | 9/1 | No class | ||

Week3 | 9/6 | Random variable, distribution function and density | HMC 1.5 | HW 2 - Due 9/13 in class |

Week3 | 9/8 | Distribution function, conditional distribution | HMC 1.4,6 | |

Week4 | 9/13 | Transformation of random variables | HMC 1.7-8 | HW 3 - Due 9/22 in class |

Week4 | 9/15 | Moment generating function, important inequalities | HMC 1.9-10 | |

Week5 | 9/20 | Review | ||

Week5 | 9/22 | Midterm 1 |
||

Week6 | 9/27 | No Class | HW 4 - Due 10/4 in class | |

Week6 | 9/29 | Distribution of two random variables | HMC 2.1 | |

Week7 | 10/4 | Transformation of bivariate random variables | HMC 2.2 | HW 5 - Due 10/11 in class |

Week7 | 10/6 | Conditional distribution, correlation coefficient | HMC 2.3-4 | |

Week8 | 10/11 | Independence, linear combination of random variables | HMC 2.5,8 | HW 6 - Due 10/20 in class |

Week8 | 10/13 | Binomial distribution | HMC 3.1 | |

Week9 | 10/18 | Poisson distribution | HMC 3.2 | |

Week9 | 10/20 | Midterm 2 |
||

Week10 | 10/27 | \(\Gamma\), \(\chi^{2}\), and \(\beta\) distributions | HMC 3.3 | HW 7 - Due 11/1 in class |

Week10 | 10/25 | Normal distribution | HMC 3.4 | |

Week10 | 11/1 | Multivariate normal distribution | HMC 3.5 | HW 8 - Due 11/8 in class |

Week11 | 11/3 | t/F-distribution | HMC 3.6 | |

Week11 | 11/8 | Statistical model | HW 9 - Due 11/15 in class | |

Week12 | 11/10 | Statistical model & inference, introduction to estimation | HMC 4.1 | |

Week12 | 11/15 | Likelihood, MLE, posterior distribution | HMC 4.1, 6.1 | HW 10 - Due 11/29 in class |

Week13 | 11/17 | Mean, variance, MSE | HMC 11.1,2 | |

Week13 | 11/22 | Midterm 3 |
||

Week14 | 11/24 | No class | Thanksgiving holidays | |

Week14 | 11/29 | Loss and risk functions | HMC 7.1 | HW 11 - Due 12/6 in class |

Week15 | 12/1 | Order statistics | HMC 4.4 | |

Week15 | 12/6 | Review | ||

Week16 | 12/13 | Final exam |
11-1:30pm, PKH 302 |

## Homework

### HW 2

- Do Exercise 1.4.11,12,25
- Do Exercise 1.5.3,5,8. Sketch graphs of CDFs.
- Do Exercise 1.7,9,12,14
- Prove or disprove: two events are mutually exclusive if and only if they are independent.

### HW 4

- Do Exercise 2.1.6,7,9,10,13-16.
- Two random variables \(X_{1}\) and \(X_{2}\) has a joint pdf, \[ f(x_{1},x_{2}) = 4 x_{1} x_{2}, \, 0 < x_{1} < 1, \, 0 < x_{2} < 1. \] Let \(Y = X_{1}+X_{2}\). Find the CDF of \(Y\).

### HW 5

- Do Exercise 2.2.3-7.
- Using Example 2.2.2, find the CDF of \(Y\).
- Use the transformation.
- Use the direct integration.

- Do Exercise 2.3.1-3,5,12
- Do Exercise 2.4.3,4

### HW 6

- Do Exercise 2.5.1,2,4,8
- Do Exercise 2.8.2-7,10,18
- Do Exercise 3.1.2,3.1.28
- Do Exercise 3.1.4
- Find \(P(\min X \ge 0.5)\) and \(P(\max X \ge 0.5)\).

- Do Exercise 3.2.1-2,5,11-12,14

### HW 7

- Do Exercise 3.3.1-3,6,9,16-17,23-24.
- Do Exercise 3.4.2,4,6,10,12-13,16,19,28-29,32.
- Do Exercise 3.5.1-2,5,8,10.

### HW 8

- Do Exercise 3.6.8-10,12-14.
- Let \(X_{1},\ldots,X_{n}\) are iid samples from N(\(\mu,\sigma^2\)). Prove \(\bar X \sim \text{N}(\mu,\sigma^2 / n)\).

### HW 9

- Suppose \(X_{1},\ldots,X_{n}\) are iid Uniform(\(\theta\),\(\theta+1\)), \(\theta \in \mathbb{R}\).
- Find the method of moment estimator of \(\theta\).
- Consider the following data: \(3.2, 3.5, 4.0, 3.8, 3.3\). What is the estimate of \(\theta\)?

- Suppose \(X_{1},\ldots,X_{n}\) are iid Gamma(\(\alpha\),\(\beta\)), \((\alpha,\beta) \in (0,\infty) \times (0,\infty)\). The MGF of Gamma distribution is \(M(t) = (1-\beta t) ^{-\alpha}, \, t < 1/\beta\).
- Find the mean and variance of the Gamma distribution.
- Find the method of moments estimator of \(\alpha\) and \(\beta\).

### HW 10

- Let \(X_{1}, X_{2}, \ldots , X_{n}\) are iid samples from the distribution with probability density function
\[
f(x|\lambda)=2\lambda^{2} x^{3}e^{-\lambda x^{2}}, x>0, λ>0.
\]
- Find the log-likelihood function (you can drop a constant term).
- Find the MLE of \(\lambda\).

- Suppose \(X_{1}, \ldots, X_{n}\) are iid Uniform(\(\theta\),\(\theta+1\)), \(\theta \in \mathbb{R}\).
- What is the space of \(X_{i}\)?
- Find the likelihood function.
- Consider the following data: 3.2, 3.5, 4.0, 3.8, 3.3. The likelihood is maximized for θ in an interval (so the MLE is not unique). What is the interval?

- Suppose that number of spam emails you received follows a Poisson process with a rate of \(\lambda\) per minute, where \(\lambda \in (0,\infty)\). That is, if \(X\) is the number of spams coming in over the course of t minutes, then
\[X \sim \text{Poisson}(t\lambda).\]
- Assuming t is known, what is the MLE of \(\lambda\)?
- Assuming t is known, is \(\hat \lambda\) (MLE) unbiased? (i.e. \(E(\hat \lambda) = \lambda\)?)
- (
*Hint:*First find \(\theta\) in terms of \(\lambda\).) Assuming t is known, what is the MLE of the parameter \[\theta = P_{\lambda}[\text{no spams in the next two minutes}]?\]

- Suppose \(X_{1}, \ldots , X_{n}\) are iid with pdf
\[ f(x_{i}; \alpha, \lambda) = λ\exp[−λ(x_{i}−α)]I_{x_{i}≥α}(x_{i}), \]
where \((α,λ) ∈ \mathbb{R}×(0,∞)\). Find the MLE of \((α,λ)\) when \(n\) = 4 and the data are 10,7,12,15. (
*Hint:*First find the MLE of \(α\) for fixed \(λ\), and note that it does not depend \(λ\).) - Suppose \(X | \theta \sim \text{Binomial}(100,\theta)\) and \(\theta \sim \text{Uniform(0,1)}\).
- Find the prior mean of \(\theta\).
- Find the posterior mean of \(\theta\) (Bayes estimator).
- Find the MSE of the posterior mean.

- Let \(X_{1},...,X_{n}\) be iid N\((μ,1)\), with \(μ ∈ \mathbb{R}\). A modification to least squares uses the estimator \(δ_{c}(\underline x)\) that is the value of \(μ\) that minimizes
\[ l(μ; x_{1}, \ldots, x_{n})= \sum_{i=1}^{n} (x_{i} − μ)^{2} +c μ^{2}\]
over \(μ\), where \(c ≥ 0\) is some fixed constant. (Adding that extra term to the sum of squares is a form of
*regularization*that is popular in machine learning. This particular function is a special case of the objective function in ridge regression.)- What is \(\delta_{c} (\underline x)\)? Is this unbiased for \(\mu\)?
- For which value of \(c\) is \(δ_{c}\) the MLE?
- For \(c > 0\), \(δ_{c} = E [ μ | \underline X = \underline x ] = \frac{n\bar x + \mu_{0} / \sigma_{0}^{2}}{n + 1/\sigma_{0}^{2}}\) is the Bayes posterior mean using the \(N(μ_{0},σ_{0}^{2})\) prior. What are \(μ_{0}\) and \(σ_{0}^{2}\)?
- Obatin the MSE for \(\delta_{c}\).

Let \(X_{1},...,X_{n}\) be iid Poisson\((\theta)\), \(\theta >0\).

- Find the MLE of \(\theta\).
- Find the MSE of the MLE.
- Suppose the prior distribution on \(\theta\) is Gamma(\(\alpha, lambda\)), \(\alpha,\lambda >0\) with pdf

\[ \pi (\theta) = \frac{\lambda^{\alpha}}{\Gamma(\alpha)} \theta^{\alpha - 1} \exp( - \lambda \theta), \, \theta >0. \] . Show that the posterior distribution of \(\theta\) is Gamma(\(\sum_{i}^{n}x_{i} + \alpha, n+\lambda\)).

### HW 11

- Do Exercise 6.1.1, 6.1.5.
- Let \(X_{1},...,X_{n}\) be iid Uniform\((0,\theta)\), \(\theta \in (0, \infty)\).
- What is the space of \(X_{i}\)?
- Find the likelihood function of \(\theta\).
- Show that the MLE of \(\theta\) is \(\hat \theta = \max X_{i}\).
- Find \(E_{\theta}(\hat \theta)\). Is it unbiased? (
*Hint:*find the probability density of \(\hat \theta\) using \(P(\max X_{i} \le x)\).) - Find the risk of \(\hat \theta\) under squared-error loss when \(\theta = 1\).

- Suppose \(X\) and \(Y\) are independent, with \(X \sim\) Poisson(\(1 + θ\)) and \(Y \sim\) Poisson(\(1 − θ\)), where \(θ ∈ (−1, 1)\).
- Find constants \(a\) and \(b\) so that \(δ_{1}(x) = a + bx\) is an unbiased estimator of \(θ\). What is \(Var[δ_{1}(X)]\)?
- Find constants \(c\) and \(d\) so that \(δ_{2}(x,y)\) = \(cx+dy\) is an unbiased estimator of \(θ\). What is \(Var[δ_{2}(X,Y)]\)?
- Sketch \(MSE(θ,δ_{i})\)’s for the estimators found in parts 1) and 2). Is either one always better than the other?

- Suppose \(X_{1},...,X_{n}\) are iid N(\(μ,σ^{2}\)), where \((μ,σ^{2})∈R×(0,∞)\). Let \(U= \sum (X_{i} - \bar X)^{2}\), so that \(\frac{U}{\sigma^{2}} \sim \chi^{2}(n−1)\).
- Let \(δ_{c}(u) = cu\) be an estimator of \(σ^{2}\) for some constant c. Find the mean and variance of \(δ_c\), and show that the mean square error of \(δ_{c}\) is \[ MSE(σ^{2};δ_{c})= σ^{4} ((c(n−1)−1)^{2} +2c^{2}(n−1)). \]
- Find the \(c\) that minimizes the MSE in part (a).
- For \(c\) in part 2), find \(δ_{c}\) and \(E[δ_{c}(U)]\). Is this estimator unbiased?

- Suppose \(X_{1},...,X_{n}\) are iid Bernoulli(\(\theta\)), \(0 < \theta < 1\). The goal is to estimate \(g(\theta) = \theta^{2}\).
- Let \(\delta_{1} = X_{1} X_{2}\). Is this unbiased for \(g(\theta)\)?
- Let \(Y = \sum_{i=1} ^{n} X_{i}\). What is the distribution of \(Y\)?
- Let \(\delta_{2} = (Y^{2} - Y) / ( n(n-1))\). Is this unbiased for \(g(\theta)\)?
- Find the variance of \(\delta_{1}\) and \(\delta_{2}\).
- In the light of MSE, which estimator is better?

- Let \(X_{1},...,X_{n}\) be iid Poisson\((\theta)\). The prior on \(\theta\) is Gamma(\(\alpha, \lambda\)) with pdf
\[ \pi (\theta) = \frac{\lambda^{\alpha}}{\Gamma(\alpha)} \theta^{\alpha - 1} \exp( - \lambda \theta), \, \theta >0.
\]
The posterior \(\pi(\theta | \underline{x})\) is given in the last HW. Suppose the loss function is \(L_{2}(\theta ,\delta) = (\theta - \delta)^{2}/ \theta\). This loss function may be more appropriate for the estimation of a variance of \(X_{i}\).
- Given \(\underline{X} = \underline{x}\), \(\delta(\underline{X})\) becomes \(\delta(\underline{x})\) (constant). The posterior risk is \(E[L_{2}(\theta ,\delta(\underline{X})) | \underline{X} = \underline{x}] = A - B \delta(\underline{x}) + C \delta(\underline{x}) ^{2}\). What are \(A\),\(B\), and \(C\)?
- Find the Bayes estimator that minimizes the risk above.

## Exams

Exams will be closed book, open calculators. Calculators may not be shared on exams. Cell phone calculators are not permitted on exams. A single page (on US letter size paper), hand-written cheat-sheet is permitted for each exam.