Linear Regression

From Machine Learning
Jump to: navigation, search

The following image shows what a linear regression model could look like:

Linear regression.jpg

Linear regression essentially aims to find the line of best fit between two variables. The line can be linear or curvilinear. There are many techniques that can be used to estimate the coefficients βn, some of which are described below:

Ordinary Least Squares

Used when there is more than one input x. It treats the data as a matrix, and it looks for the optimal regression line (the optimal values for the coefficients) using linear algebra. There must be enough memory available to hold all the data and perform matrix operations. This method is very common and will be available in many libraries, ready to implement.

Gradient Descent

Works by selecting random values for the coefficients to begin with and uses a learning rate as a scale factor. Learning rate (α) is a selected hyperparameter that determines the size of the improvement step each iteration. The whole dataset does not need to be available at once in memory as the sum of squared errors are calculated for input and output pairs at a time.

Regularisation

  • Lasso Regression (L1 Regression), where Ordinary Least Squares is modified to also minimise the absolute sum of the coefficients
  • Ridge Regression (L2 Regression), where Ordinary Least Squares is modified to also minimise the squared absolute sum of the coefficients


Preparing data for a linear regression model

• Keep in mind that the relationship between the input and output must be linear – this may require a transformation of the data.

• Removing noise is important, especially removing outliers in the output values y

• Linear regression will overfit the data if the input variables are highly correlated

• If the input and output variables have a Gaussian (normal) distribution the model’s predictions will be more accurate. Transforming the data to have a more gaussian distribution may be beneficial.

• Rescaling inputs through standardisation or normalisation can also lead to more accurate predictions