Dataframe.unstack() equivalent in numpy

Asked Nov 12 '19 at 14:19

Active Nov 12 '19 at 14:32

Viewed 358 times

I am running a code that calculates the correlation (Pearson) between columns in a DataFrame and then unstacks it to build a graph on it.

However the pandas dataframe.corr() is quite slow atm (21 000 columns) and I saw that numpy.corrcoef() might be much faster as I don't have NaNs.

However I use df.unstack() on the correlation results and I couldn't find a numpy equivalent

edited Nov 12 '19 at 14:30

mayosten

asked Nov 12 '19 at 14:19

Mayeul sgc

Don't reinvent the wheel. Have a look into `pandas_profiling` https://pandas-profiling.github.io/pandas-profiling/examples/titanic/titanic_report.html – alec_djinn Nov 12 '19 at 14:29
Wouldn’t reshape or flatten suffice? – mayosten Nov 12 '19 at 14:29
@alec_djinn I don't see your point... – Mayeul sgc Nov 12 '19 at 14:33
1

You can insert results of `numpy.corrcoef()` to pandas data frame as a quick workaround – Artyom Akselrod Nov 12 '19 at 14:33
@ArtyomAkselrod I didn't think about it, very good point – Mayeul sgc Nov 12 '19 at 14:34
@Mayeulsgc The point is that there is a library doing the correlation plot of the database for you with a single line of code already. – alec_djinn Nov 13 '19 at 10:49
I didn't look for the plot, I was looking to build graph of all features related to each other above a given correlation threshold – Mayeul sgc Nov 13 '19 at 10:50
@Mayeulsgc that is exactly what the library does. Install it and try `df.profile_report()` – alec_djinn Nov 13 '19 at 10:52

0 Answers0