How to select columns of a data base to call a linear regression (OLS and lasso) in sklearn

Question

I am not comfortable with Python - much less intimidated and at ease with R. So indulge me on a silly question that is taking me a ton of searches without success.

I want to fit in a regression model with sklearn both with OLS and lasso. In particular, I like the mtcars dataset that is so easy to call in R, and, as it turns out, also very accessible in Python:

import statsmodels.api as sm
import pandas as pd
import statsmodels.formula.api as smf


mtcars = sm.datasets.get_rdataset("mtcars", "datasets", cache=True).data
df = pd.DataFrame(mtcars)

It looks like this:

                      mpg  cyl   disp   hp  drat  ...   qsec  vs  am  gear  carb
Mazda RX4            21.0    6  160.0  110  3.90  ...  16.46   0   1     4     4
Mazda RX4 Wag        21.0    6  160.0  110  3.90  ...  17.02   0   1     4     4
Datsun 710           22.8    4  108.0   93  3.85  ...  18.61   1   1     4     1
Hornet 4 Drive       21.4    6  258.0  110  3.08  ...  19.44   1   0     3     1

In trying to use LinearRegression() the usual structure found is

import numpy as np
from sklearn.linear_model import LinearRegression

model = LinearRegression().fit(x, y)

but to do so, I need to select several columns of df to fit into the regressors x, and a column to be the independent variable y. For example, I'd like to get an x matrix that includes a column of 1's (for the intercept) as well as the disp and qsec (numerical variables), as well as cyl (categorical variable). On the side of the independent variable, I'd like to use mpg.

It would look if it were possible to word this way as

model = LinearRegression().fit(mpg ~['disp', 'qsec', C('cyl')], data=df)

But how do I go about the syntax for it?

Similarly, how can I do the same with lasso:

from sklearn.linear_model import Lasso
lasso = Lasso(alpha=0.001)
lasso.fit(mpg ~['disp', 'qsec', C('cyl')], data=df)

but again this is not the right syntax.

I did find that you can get the actual regression (OLS or lasso) by turning the dataframe into a matrix. However, the names of the columns are gone, and it is hard to read the variable corresponding to each coefficients. And I still haven't found a simple method to run diagnostic values, like p-values, or the r-square to begin with.

StupidWolf · Accepted Answer · 2021-05-10T07:25:28.550

You can maybe try patsy which is used by statsmodels:

import statsmodels.api as sm
import pandas as pd
import statsmodels.formula.api as smf
from patsy import dmatrix

mtcars = sm.datasets.get_rdataset("mtcars", "datasets", cache=True).data

mat = dmatrix("disp + qsec + C(cyl)", mtcars)

Looks like this, we can omit first column intercept since it is included in sklearn:

mat
 
DesignMatrix with shape (32, 5)
  Intercept  C(cyl)[T.6]  C(cyl)[T.8]   disp   qsec
          1            1            0  160.0  16.46
          1            1            0  160.0  17.02
          1            0            0  108.0  18.61
          1            1            0  258.0  19.44
          1            0            1  360.0  17.02

X = pd.DataFrame(mat[:,1:],columns = mat.design_info.column_names[1:])

from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X,mtcars['mpg'])

But the parameters names in model.coef_ will not be named. You just have to put them into a series to read them maybe:

pd.Series(model.coef_,index = X.columns)
 
C(cyl)[T.6]   -5.087564
C(cyl)[T.8]   -5.535554
disp          -0.025860
qsec          -0.162425

Pvalues from sklearn linear regression, there's no ready method to do it, you can check out these answers, maybe one of them is what you are looking for.

It strikes me as this whole package is not meant for statistical analysis - the objective is predictions and classification, i.e. feature selection and weight assignment for machine learning algos. Is that your impression? If this is true, it makes no sense to even try to match the detail of R in its output. — Antoni Parellada, May 10 '21 at 13:26
Well in statistics we do either inference or prediction. It's not meant for inference.. If you are looking for something with similar output to R, statsmodels is easier — StupidWolf, May 10 '21 at 13:27
I keep on thinking of the statement at the beginning of your last comment, because I never thought of it that way. Am I reading you correctly if I paraphrased it as... Classical statistics in a tacit way always wants to suggest association or cause-effect (inference), whereas "ML stats" wants to predict (self-driven cars, Google search, NNs)? If so, it makes a lot of sense. — Antoni Parellada, May 10 '21 at 17:31

score 0 · Answer 2 · answered May 09 '21 at 14:46

Here are two ways - unsatisfactory, especially because the variables labels seem to be gone once the regression gets going:

import statsmodels.api as sm
import pandas as pd
import statsmodels.formula.api as smf


mtcars = sm.datasets.get_rdataset("mtcars", "datasets", cache=True).data
df = pd.DataFrame(mtcars)

import numpy as np
from sklearn.linear_model import LinearRegression

Single variable regression mpg (i.v.) ~ hp (d.v.):

lm = LinearRegression()
 
mat = np.matrix(df)
 
lmFit = lm.fit(mat[:,3], mat[:,0])
 
print(lmFit.coef_)
print(lmFit.intercept_)

For multiple regression drat ~ wt + cyl + carb:

lmm = LinearRegression()
wt = np.array(df['wt'])
cyl = np.array(df['cyl'])
carb = np.array(df['carb'])
stack = np.column_stack((cyl,wt,carb))
stackmat = np.matrix(stack)

lmFit2 = lmm.fit(stackmat,mat[:,4])
print(lmFit2.coef_)
print(lmFit2.intercept_)

How to select columns of a data base to call a linear regression (OLS and lasso) in sklearn

2 Answers2