1

I have Python 3.7 installed. Trying to install janitor library in Azure DataBricks. It works properly in my local machine, but have difficulty to be installed in Azure DataBricks.

I run dbutils.library.installPyPI('janitor'), but got the below error: ModuleNotFoundError: No module named 'ConfigParser'. I tried butils.library.installPyPI('mysqlclient'), as mentioned in 'https://stackoverflow.com/questions/14087598/python-3-importerror-no-module-named-configparser', but didn't work.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Amn Kh
  • 531
  • 3
  • 7
  • 19
  • Did you install `ConfigParser` ? – The Singularity Oct 07 '21 at 06:16
  • I did `Requirement already satisfied: ConfigParser in /databricks/python3/lib/python3.7/site-packages (5.0.2)`, but it gives me the same error `ModuleNotFoundError: No module named 'ConfigParser'` when I run `import janitor` – Amn Kh Oct 08 '21 at 02:55

3 Answers3

0

dbutils.library.installPyPI is a deprecated method (doc). You need to use %pip install janitor instead (see docs)

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • I did `%sh pip install janitor`, but still shows error. I also run `%pip install janitor`, but it gave me `org.apache.spark.SparkException: Conda and pip magic commands are currently not supported in Databricks. To modify the environment on driver only, you can use %sh. ` – Amn Kh Oct 08 '21 at 02:58
  • What DBR version? – Alex Ott Oct 08 '21 at 04:49
  • DBR 6.4, Spark 2.4.5, Scala 2.11 – Amn Kh Oct 09 '21 at 04:47
0

You can install Janitor library using %pip install janitor.

enter image description here

Based on your Databricks Runtime, if you are unable to install using the above command you can use any alternative method to install library: different methods to install packages in Azure Databricks.

CHEEKATLAPRADEEP
  • 12,191
  • 1
  • 19
  • 42
  • It gives me `ModuleNotFoundError: No module named 'ConfigParser'`, however I run `%sh pip install ConfigParser`. It seems `ConfigParser` has been installed as `Collecting ConfigParser Using cached https://files.pythonhosted.org/packages/fd/01/ff260a18caaf4457eb028c96eeb405c4a230ca06c8ec9c1379f813caa52e/configparser-5.0.2-py3-none-any.whl Installing collected packages: ConfigParser Successfully installed ConfigParser-5.0.2 You should consider upgrading via the 'pip install --upgrade pip' command.`, but NO. I'm still getting `ModuleNotFoundError: No module named 'ConfigParser'` – Amn Kh Oct 09 '21 at 04:40
  • @AmnKh Could you please share the Databricks Runtime which you are using? – CHEEKATLAPRADEEP Oct 11 '21 at 04:01
0

The python implementation of this great library is pyjanitor:

pip install pyjanitor

https://pypi.org/project/pyjanitor/

Douglas Wiley
  • 483
  • 1
  • 7
  • 10