46

When choosing the number of principal components (k), we choose k to be the smallest value so that for example, 99% of variance, is retained.

However, in the Python Scikit learn, I am not 100% sure pca.explained_variance_ratio_ = 0.99 is equal to "99% of variance is retained"? Could anyone enlighten? Thanks.

  • The Python Scikit learn PCA manual is here

http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA

CCoder
  • 2,305
  • 19
  • 41
Chubaka
  • 2,933
  • 7
  • 43
  • 58

3 Answers3

104

Yes, you are nearly right. The pca.explained_variance_ratio_ parameter returns a vector of the variance explained by each dimension. Thus pca.explained_variance_ratio_[i] gives the variance explained solely by the i+1st dimension.

You probably want to do pca.explained_variance_ratio_.cumsum(). That will return a vector x such that x[i] returns the cumulative variance explained by the first i+1 dimensions.

import numpy as np
from sklearn.decomposition import PCA

np.random.seed(0)
my_matrix = np.random.randn(20, 5)

my_model = PCA(n_components=5)
my_model.fit_transform(my_matrix)

print my_model.explained_variance_
print my_model.explained_variance_ratio_
print my_model.explained_variance_ratio_.cumsum()

[ 1.50756565  1.29374452  0.97042041  0.61712667  0.31529082]
[ 0.32047581  0.27502207  0.20629036  0.13118776  0.067024  ]
[ 0.32047581  0.59549787  0.80178824  0.932976    1.        ]

So in my random toy data, if I picked k=4 I would retain 93.3% of the variance.

Curt F.
  • 4,690
  • 2
  • 22
  • 39
  • 4
    Thank you so much! Great explanation! Sometimes Python manual is poorly documented / explained. I am totally enlightened! – Chubaka Sep 30 '15 at 05:12
  • One more question: when we do PCA(n_components=1), the scikit learn "PCA" commands perform the "Compute covariance matrix from the normalized data" & "Use single value decomposition (SVD) to compute eigenvectors "? I don't see any where to choose other methods to compute eigenvectors in the Python scikit learn PCA module. – Chubaka Sep 30 '15 at 05:17
  • 1
    Great answer! this really helped me – Chaos Monkey Jun 11 '17 at 08:35
  • I think some "*e-int" went missing from first members of the copied results. That might be confusing. Otherwise, This is great! +1 – AChervony Oct 22 '17 at 20:04
  • @AChervony, not sure what you mean. – Curt F. Nov 05 '17 at 08:28
  • You can disregard it, just thanks and +1. The numbers I got were showing in scientific notation. E.g. 3.45e-4, so the scale of the result seemed wrong. All in all: 'Thanks!' – AChervony Nov 06 '17 at 17:59
  • isn't it enough to sum, like sum(my_model.explained_variance_ratio_)? why cumsum? – kilgoretrout Apr 12 '21 at 10:14
  • @kilgoretrout If you want to see the intermediate sums you have to do cumsum. `pca.explained_variance_ratio_.sum()` will always give you 1.0 if you are doing a full-rank PCA (same number of PCs as original dimensions). – Curt F. Apr 16 '21 at 17:02
  • @chubaka you will probably not find any other place doing the eigen_vector calc as it is really a single classical implementation available through LAPACK in FORTRAN. – Mário de Sá Vera Dec 04 '21 at 19:31
55

Although this question is older than 2 years i want to provide an update on this. I wanted to do the same and it looks like sklearn now provides this feature out of the box.

As stated in the docs

if 0 < n_components < 1 and svd_solver == ‘full’, select the number of components such that the amount of variance that needs to be explained is greater than the percentage specified by n_components

So the code required is now

my_model = PCA(n_components=0.99, svd_solver='full')
my_model.fit_transform(my_matrix)
Yannic Bürgmann
  • 6,301
  • 5
  • 43
  • 77
1

This worked for me with even less typing in the PCA section. The rest is added for convenience. Only 'data' needs to be defined in an earlier stage.

import sklearn as sl
from sklearn.preprocessing import StandardScaler as ss
from sklearn.decomposition import PCA 

st = ss().fit_transform(data)
pca = PCA(0.80)
pc = pca.fit_transform(st) # << to retain the components in an object
pc

#pca.explained_variance_ratio_
print ( "Components = ", pca.n_components_ , ";\nTotal explained variance = ",
      round(pca.explained_variance_ratio_.sum(),5)  )
Julian
  • 154
  • 1
  • 11