We have our equation
$$ y_i = \beta_0 + \beta_1 x_i + u_i $$
- in this model we need to fine $\{\beta_0, \beta_1\}$ so that we can explain how change in $x$ would affect $y$
[IMPORTANT] Next question is: Can we find out $\{\beta_0, \beta_1\}$?
- We need to make a mental note that the $\{\beta_0, \beta_1\}$ in $y_i = \beta_0 + \beta_1 x_i + u_i$ describes the TRUE relationship
- HOWEVER, due to data limitation (we cannot survey everyone in the population) we can’t find the exact value of $\{\beta_0, \beta_1\}$
- THEREFORE, we can only estimate (make a best educated guess) of $\{\beta_0, \beta_1\}$
- To differentiate between the ideal truth and our estimate, we use the notation $\hat{ }$ to denote our estimate i.e. we get try to find $\{\hat{\beta_0}, \hat{\beta_1}\}$ which is an estimate of $\{\beta_0, \beta_1\}$
- Note that the raw $\beta$ can also represent free variable, but $\hat \beta$ always denote the estimate and $\bar \beta$ the mean
- Note other circumflex might be placed over parameters e.g. $\tilde \beta$
- That brings us to the first big concept, econometrics only estimate the values of the parameters.
- Note that our best guess, i.e. estimates, may be wrong, and we have to validate that they are indeed “correct” later.
How are we going to estimate the parameters — find $\{\hat{\beta_0}, \hat{\beta_1}\}$
- Let’s look work through the example below
- We know that regression is just a linear model i.e. $y = \beta_0 + \beta_1 x + u$ how would you fit a line to “estimate” the $\beta_0$ and $\beta_1$ ?
- Without doing any math, we can try to draw a straight line and then see where is the y intercept ($\hat{\beta_0}$) and the slope of the line ($\hat{\beta_1}$)
- With this, you have understood the fundamental concept of regression: we are trying to fit the best linear line to the data
-
We can look at the graph but maths have no eyes, and economist want a bit more granularity than drawing a line so we need to try something else.
-
Let’s go back to our example and try to see if we can find something find something useful to estimate ****$\{\hat{\beta_0}, \hat{\beta_1}\}$
-
Assume that we randomly pick 2 sets of $\{\hat{\beta_0}, \hat{\beta_1}\}$ and draw out the line they produce
-
I think we can intuitively see that the yellow line fits better than the blue one. But we need a way to formalise this intuition
-
One way to evaluate the fit is the look at the distance between our observations and the line (predicted value)
-
If we can add up the distances between the observation and the line, aka the error, then we can compare two numeric value and conclude that the yellow line is a better fit.
-
But the error have both positive value and negative value and we don’t want the numbers to cancel when we are summing up the errors, we have square it.
-
⭐️ Putting our intuition together, we can construct a statistic called Sum of Squared Residuals where residual is just another name of error/ distance between the line and observation.
$$ SSR = u^2_1 + u^2_2 + u^2_3 + ... $$
In the example
$SSR_{yellow} = (−2.8477)^2 + (+2.5275)^2 + (−1.7413)^2 = 17.529$
$SSR_{blue} = (−5.3477)^2 + (+3.5275)^2 + (+2.2587)^2 = 46.142$
Yellow line’s SSR is smaller → better fit.
-
‼️You should fully understand the example before continuing on with the text because it’s getting a bit dry and an intuitive understanding is critical to continue on
-
‼️Another important note: pay close attention to the notation and try to distinguish between the true value vs. estimate value
-
In the example we mention that we can use the Sum of Squared Residuals (SSR/ or sometimes called RSS) to compare the fit of 2 ****$\{\hat{\beta_0}, \hat{\beta_1}\}$ estimations.
-
Let’s dive a bit deeper into the residuals. Recall the model equation in estimation form:
$y_i = \hat{\beta_0} + \hat{\beta_1} x_i + \hat{u}_i$
$\hat{u_i}$ is the residual
we can rewrite the equation as
$$ \hat{u}_i = y_i - \hat{\beta_0} - \hat{\beta_1} x_i $$
Important note on notation
- $\hat{\theta}$ denotes estimates
- $y_i$ and $x_i$ (wage and education) are true value we can observe
- $\{\hat{\beta_0}, \hat{\beta_1}\}$ are our guesses of the relationship
- ‼️ $\hat{u_i}$ is also estimate because it is what the error we get from subtracting $y_i - \hat{\beta_0} + \hat{\beta_1} x_i$ : $\hat{\beta}$s are estimated → $\hat{u_i}$ changes with different estimates of $\hat{\beta}$s, therefore, it is also a estimate
-
Next remember to square the residual and add them up across all sample from $1$ to $N$
$$ \mathrm{SSR}(\hat\beta_0,\hat\beta_1)= \sum_{i=1}^n \hat{u}^2_i = \sum_{i=1}^n \big(y_i - \hat\beta_0 - \hat\beta_1 x_i\big)^2 $$
-
In some text they will take the mean and call is Mean Squared Errors, but it’s just adding $\frac{1}{N}$ in front. For subsequent calculation it does not matter whether you add it or not. → if your prof like it then add it if not you can decide whether you want to add it
-
Now that we know how to compare the fit of the estimates, we want to use math to find out the BEST $\{\hat{\beta_0}, \hat{\beta_1}\}$ pair
-
Since we have a nice equation for $SSR$ and it’s a continuous function (in our wage and education example). Let’s plot the change in SSR when we change $\hat{\beta_1}$
-
Can you see that there’s a minima?
-
Recall your 1st year math! If we have a continuous function $f(x)$ how do we find $x$ that give us the minimum value of $f(x)$?
-
First order condition!
- $x^\star \ \text{s.t.}\ \frac{df}{dx}(x^\star)=0$
- Take the first derivative, find $x$ that set the first derivative as zero
-
Let’s apply this logic to SSR
$$ \begin{aligned}\mathrm{SSR}(\beta_0,\beta_1) &= \sum_{i=1}^n u_i^2 = \sum_{i=1}^n \big(y_i - \beta_0 - \beta_1 x_i\big)^2 \\[6pt]\frac{\partial \mathrm{SSR}}{\partial \beta_0}&= -2\sum_{i=1}^n \big(y_i-\beta_0-\beta_1 x_i\big) = 0 \\[6pt]\frac{\partial \mathrm{SSR}}{\partial \beta_1}&= -2\sum_{i=1}^n x_i \big(y_i-\beta_0-\beta_1 x_i\big) = 0\end{aligned} $$
⚠️ Note that here we don’t use $\hat{.}$ because it’s still unknown parameters not an estimate yet, it is confusing, I know
-
Let’s drive the estimates:
Expand and rearrange to get normal equations - Eq 1 and Eq 2
$$ \begin{aligned}\sum_{i=1}^n y_i &= n\,\beta_0 + \beta_1 \sum_{i=1}^n x_i, \\\sum_{i=1}^n x_i y_i &= \beta_0 \sum_{i=1}^n x_i + \beta_1 \sum_{i=1}^n x_i^2\end{aligned} $$
From Eq1
$$ \beta_0 = \bar y - \beta_1 \bar x $$
Substitute $\beta_0$ equation into Eq 2
$$ \begin{aligned} \sum x_i y_i &= (\bar y-\beta_1\bar x)\sum x_i + \beta_1 \sum x_i^2 \\&= n\bar x\,\bar y - \beta_1 n\bar x^2 + \beta_1 \sum x_i^2\\ \sum x_i y_i - n\bar x\,\bar y &= \beta_1\Big(\sum x_i^2 - n\bar x^2\Big)\\ \hat \beta_1 &= \frac{\sum x_i y_i - n\bar x\,\bar y}{\sum x_i^2 - n\bar x^2} \end{aligned}\\ $$
Now recognised that, ask ChatGPT if you cannot proof it
$$ \sum (x_i-\bar x)^2 = \sum x_i^2 - n\bar x^2 \\ \sum (x_i-\bar x)(y_i-\bar y) = \sum x_i y_i - n\bar x\,\bar y $$
We have
$$ \begin{aligned} \hat\beta_1 &= \frac{\sum (x_i-\bar x)(y_i-\bar y)}{\sum (x_i-\bar x)^2}\\ \text{Equivalent to:}\\ \hat\beta_1 &= \frac{Cov(x,y)}{Var(x)} \end{aligned} $$
Plugging $\hat \beta_1$ to $\beta_0$equation
$$ \hat \beta_0 = \bar y = \hat \beta_1 \bar x $$
That is the full chain: FOC → normal equations → $\beta_0$ equation → solve for $\hat \beta_1$ → back-substitute for $\hat \beta_0$