Regression Performance Measures

Regression Performance Metrics

To analyze the performance of regression, several evaluation measures are used. These performance metrics are important for performing quantitative analysis of the results produced by the regression models.

The various metrics used to evaluate the results of the prediction are :

Mean Squared Error (MSE)
Root-Mean-Squared-Error (RMSE)
Mean-Absolute-Error (MAE)
R² or Coefficient of Determination
Adjusted R²

They are described below one-by-one.

1. Mean Squared Error (MSE)

MSE is a widely-used evaluation measure of the difference between the values predicted by a regression model and the actual values found from the system. It is simply the average of the squared difference between the target value e_actualand the value predicted by the regression model e_predicted as:

MSE = ∑_k (e_actual,k– e_predicted,k)² / n ———> (1) [for k=1, 2, … , n]

where e_actualare the actual values and e_predicted are the predicted values for ∀k. Here, ‘n’ denotes the number of data records present in the database.

2. Root-Mean-Squared Error (RMSE)

RMSE is a well-known performance measure of the dissimilarity between the values predicted by a regression model and the values actually found from the system being modeled. The RMSE of a regression model’s estimation with regard to the predicted variable e_predicted is the square root of the Mean Squared Error (MSE):

RMSE = √[∑_k (e_actual,k– e_predicted,k)²] / n ———> (2) [for k=1, 2, … , n]

where e_actualare the actual values and e_predicted are the predicted values for ∀k. Here, ‘n’ denotes the number of data records present in the database.

3. Mean Absolute Error (MAE)

MAE is the absolute difference between the target value t and the value y predicted by the model. It is more robust to outliers as compared to MSE. The formula for MAE is shown below:

MAE = ∑ (t – y) / n ———> (3)

Here, ‘n’ denotes the number of data records present in the database.

4. R² or Coefficient of Determination

R² (pronounced as “R-Squared“) or Coefficient of Determination is a very useful metric for evaluating the performance of a regression model. It is a statistical measure of fit that indicates how much variation of a dependent variable is explained by the independent variable(s) in a regression model. In other words, R² shows how well the data fit the regression model (i.e. the goodness of fit).

It is basically the square of the correlation (R) and measures the proportion of variation in the dependent variable that can be attributed to the independent variable. Mathematically, R-squared is calculated by dividing sum of squares of residuals (SSe_res) by total sum of squares (SS_tot) and then subtract it from 1. In this case, SS_tot measures total variation. SS_regmeasures explained variation and SS_res measures unexplained variation.

As SS_res + SS_reg = SS_tot and R² = Explained variation / Total Variation, the formula for R² is given by:

$R² = SS reg / SS tot = 1 - SS res / SS tot ———> (4)$

R² values range from 0 to 1 and are commonly stated as percentages from 0% to 100%. The most common interpretation of this metric is how well the regression model fits the observed data. For example, an R² of 70% reveals that 70% of the data fit the regression model. Generally, a higher value indicates a better fit for the model.

However, it is not always the case that a high R² is good for the regression model. The quality of the statistical measure depends on many factors, such as the nature of the variables employed in the model, the units of measure of the variables, and the applied data transformation. It basically suffers from the problem that the scores improve on increasing terms even though the model is not improving which may misguide the data analysts.

⇒ N.B. The value of R² may be negative sometimes as because actually it ranges from -∞ to 1, though the theoretical value lies between 0 and 1.

5. Adjusted R²

In order to solve the problem mentioned above, an improvement of R² has been made known as Adjusted R² which indicates the same meaning as R². It is always lower than R² as it adjusts for the increasing predictor variables and only shows improvement if there is a real improvement.

Basically, the use of an Adjusted R² (denoted by ${\bar R}^{2}$ and pronounced “R bar squared“) is an attempt to account for the phenomenon of the R² automatically and spuriously increasing when extra predictor variables are added to the model.

The Adjusted R² is defined as:

{\bar {R}}^{2}=1-(1-R^{2}){n-1 \over n-p-1}

where p is the total number of predictor variables in the model (excluding the constant term), and n is the sample size.