Objective parameters of models

Posted on by

the Diploma thesis learning –

There are multiple objectives of models in machine learning. One is percentage of explained variance in model, the next is speed of convergence of models (both in time as well as iteration) and the next is Accuracy of model.

It is viewed as a trade-off between the convergence efficiency and speed of convergence. Either the model is close to perfect but it is very slow to get there, or it is relatively fast yet relatively imperfect. The difference in trade-off is even bigger when parallelism is introduced.

Percentage of variance

The quality of a model is often written as finding the percentage of variance explained by model. It ts based on basic mean model. Mean model is model where our prediction is based only on mean value.

  
 (1)    \begin{align*}    base\_model &= \bar{y}, & \bar{y} &= \frac{1}{m}\sum_{i=1}^{m}{y_{i}}  \end{align*}

The R^2 comparison is coefficient of determination and has equation:

  
 (2)    \begin{align*}     R^2 &= \frac{MSE_{base\_model} - MSE_{model}}{MSE_{base\_model}}\\         &= \frac{\frac{(\bar{y} - y)^2}{m} - \frac{(\hat{y} - y)^2}{m}}{\frac{(\bar{y} - y)^2}{m}}\\         &= \frac{1}{m}\frac{(\bar{y} - y)^2 - (\hat{y} - y)^2}{(\bar{y} - y)^2}   \end{align*}

To get percentage of the variance of feature variable we need to multiply R^2 by 100.

Speed of convergence

When the convergence is good enough the main problem is speed of convergence of model. The speed of convergence is dependent on many factors with one, the optimization function in update of parameters, being one of the biggest factors. Choosing too big steps and the iteration is jumping back and forth with little improvements. Choosing too small steps and the number of iterations needed to converge to correct result is lim_{\infty} = \infty.

Accuracy of prediction

The model is function made to predict the outcome of dependent variable based on some independent variables. It can be useful to know how fast is the model improving in iterations on training as well as testing data set. There exist many metrics to judge the quality of model but one method stands out. The accuracy of prediction.

It is defined as summary of all classified predictions equal \hat{y_i} to expected result y_i for single value.

  
 (3)    \begin{align*}      Accuracy &= \frac{1}{n} \sum_{i=1}^{n}{z_i} \\      z_i &= iff(\hat{y}_{i} == y_{i}) = 1 \\           &= iff(\hat{y}_{i} \neq y_{i}) = 0   \end{align*}