Variance and Standard Error of the Mean

Prerequisite concepts

Expectation is the probability-weighted mean of the sum of all the possible outcomes of an experiment. It is also known as the expected value, mathematical expectation, EV, average, mean value, mean, or first moment. It is denoted as $E\left ( X \right )=\sum _{i=1}^{n}x_{i}p_{i}$ where X is a random variable, $x_{1},x_{2},x_{3},...x_{n}$ are the possible outcomes and $p_{1},p_{2},p_{3},...p_{n}$ are their corresponding probabilities. For example, let X represent the outcome of the roll of a single six-sided fair die. The possible values for X are 1, 2, 3, 4, 5, and 6, all equally likely and each having the probability of $\frac{1}{6}$ . The expectation of X is given by: $E\left ( X \right )=\frac{1}{6}\left ( 1 \right ) + \frac{1}{6}\left ( 2 \right ) +\frac{1}{6}\left ( 3 \right ) +\frac{1}{6}\left ( 4 \right ) + \frac{1}{6}\left ( 5 \right ) + \frac{1}{6}\left ( 6 \right )= 3\cdot 5$
A probability experiment is something that has an uncertain result. In the context of probability, this is often just referred to as an experiment.
An outcome is a possible result of an experiment.
The set of all outcomes in an experiment is called the sample space of that experiment. For example, the sample space of flipping a fair coin is $\left \{ heads, tails \right \}$
A Random Variable is the set of possible values from a random experiment. For example, an experiment of tossing a fair coin can give either a heads or a tail. If we assign values to these outcomes; $Heads=100$ and $Tails=200$ , then X is the set of these values; $X= \left \{ 100, 200 \right \}$

The variance of the Sampling Distribution of the Mean is given by $\frac{\sigma ^{2}}{n}$
where, $\sigma^{2}$ is the population variance and, n is the sample size.

Let’s derive the above formula. Variance is the expectation of the squared deviation of a random variable from its mean. It is denoted by $\sigma^{2}, s^{2}$ or Var(X)

From the above definition of Variance, we can write the following equation:
$Var(X) = E[(X - E[X])^2]$
Since we have to find the variance of the mean of samples, let’s replace the random variable X in the above equation with its mean, $\bar{X}$
$Var(\bar{X}) = E[(\bar{X} - E[\bar{X}])^2]$
We know that $\bar{X}= \frac{\sum_{i=1}^n{X_i}}{n}$ , therefore, we can expand the above equation as:
$Var(\bar{X}) = E[(\frac{\sum_{i=1}^n{X_i}}{n} - E[\frac{\sum_{i=1}^n{X_i}}{n}])^2]$
Taking out the $\frac{1}{n}$ from the above equation, we get:
$Var(\bar{X}) = E[\frac{1}{n^2}(\sum_{i=1}^n{X_i} - E[\sum_{i=1}^n{X_i}])^2]$
$Var(\bar{X}) = \frac{1}{n^2}E[(\sum_{i=1}^n{X_i} - E[\sum_{i=1}^n{X_i}])^2] \rightarrow$ let’s call this equation 1

$E[\sum_{i=1}^n{X_i}]$ in the above equation can be expanded as:
$E[{X_1} + {X_2} + {X_3} + ...... + {X_n}]$
Expectation is linear therefore the above equation can be rewritten as:
$E[{X_1}] + E[{X_2}] + E[{X_3}] + ...... + E[{X_n}]$ … n times
Since all the samples in the distribution are random; also known as IID (Independent and Identically Distributed), the mean of each of them is the same. Therefore we can write the above equation as:
$E[\bar{X}] + E[\bar{X}] + E[\bar{X}] + ...... + E[\bar{X}]$ … n times
$\Rightarrow \sum_{i=1}^nE[\bar{X}]$
Substituting this in equation 1, we get:
$Var(\bar{X}) = \frac{1}{n^2}E[(\sum_{i=1}^n{X_i} - \sum_{i=1}^nE[\bar{X}])^2]$
$\Rightarrow Var(\bar{X}) = \frac{1}{n^2}\sum_{i=1}^nE[({X_i} - E[\bar{X}])^2]$
$\Rightarrow Var(\bar{X}) = \frac{1}{n^2}[E({X_1} - E[\bar{X}])^2 + E({X_2} - E[\bar{X}])^2 + ...... + E({X_n} - E[\bar{X}])^2]$
The variance of the all the random variables} ${X_1}, {X_2}...{X_n}$ is also equal. Therefore:
$\Rightarrow Var(\bar{X}) = \frac{1}{n^2}[nE({X} - E[\bar{X}])^2]$
$\Rightarrow Var(\bar{X}) = \frac{1}{n^2}[n\sigma^2]$
$\therefore \textup{ Variance of Sampling Distribution of the Mean = Var}(\bar{X}) = \frac{\sigma^2}{n}$
$\textup{and, } \textup{Standard Error} = \sqrt{Var(\bar{X})}=\frac{\sigma}{\sqrt{n}}$