Cronjob is not working due to module missing

Question

I saw similar threads but it could not solve my issue so I try a new one. I can run my script from a shell script file from the terminal but I have an error when it is from a cronjob. I can give addition information as needed. Any help would be super appreciated. I am stuck here for too long.

I am using Anaconda3 and all modules are installed on Conda.

the shell script:

#!/bin/bash
export PATH=/usr/local/bin:${PATH}
echo $1 'report'
if [ -z "$1" ]; then
    echo 'no parameter about report type'
    exit 1
fi
TOPDIR=~/extra/cronjobs
JOBDIR=unify_reports
REPORT_TYPE=$1
DAY=$(date +"%Y%m%d")
SOURCE_PATH_base=~/market_risk
SUB_PATH=$REPORT_TYPE/src/code
INTERAPI_PATH=control_tdrive_mail
OUTPATH="${SOURCE_PATH_base}/$REPORT_TYPE/output"
OUT=$(date +%Y-%m-%d -d "1 day ago")$REPORT_TYPE.xlsx
echo 'output ' $OUTPATH $OUT
FILENO_FLAG=SINGLE
SUBJECT="[DATA BATCH] $REPORT_TYPE daily report for ""${OUT}"
TO="xxxxxxxxxxxxx@xxxxx.com"
CC=${TO}
top_dir=${TOPDIR}/${JOBDIR}
dir=$top_dir
if [ -e $dir ]; then
    echo 'exists' $dir
else
    mkdir $dir
    # log folder
    dir=$dir/logs
    if [ -e $dir ]; then
        echo 'exists' $dir
    else
        mkdir $dir
    fi
    echo 'log dorectory created' $dir
fi
cd ${TOPDIR}/${JOBDIR}
(
    # run each code to generate files
    cd "${SOURCE_PATH_base}/${SUB_PATH}"
    python3 $REPORT_TYPE.py
    # output file link
    dir=$top_dir/output_$REPORT_TYPE
    echo $dir 'output link'
    if [ -e $dir ]; then
        echo 'exists' $dir
    else
        # create link to real path
        echo 'no output link' $dir
        echo 'real output path' ${OUTPATH}
        cd ${TOPDIR}/${JOBDIR}
        trap `ln -s "${OUTPATH}" "output_${REPORT_TYPE}"` 1 2 3 15
        echo 'created output link' $dir
    fi
    # upload to gdrive
    echo 'test'  "${SOURCE_PATH_base}/${INTERAPI_PATH}"
    cd "${SOURCE_PATH_base}/${INTERAPI_PATH}"
    echo 'test2'   "${OUTPATH}/${OUT}" "${TO}" "${FILENO_FLAG}" "${SUBJECT}"
    python3 teamdrive_control.py Market_Risk $REPORT_TYPE "${OUTPATH}/${OUT}" "${TO}" "${FILENO_FLAG}" "${SUBJECT}"
) 2>&1 | xz -9ec > logs/${JOBDIR}-$(date +%s)_$REPORT_TYPE.log.xz

the error:

    Traceback (most recent call last):
  File "risk_position.py", line 3, in <module>
    import psycopg2 as ps
  File "/home/ubuntu/.local/lib/python3.6/site-packages/psycopg2/__init__.py", line 50, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: libpq.so.5: cannot open shared object file: No such file or directory
/home/ubuntu/extra/cronjobs/unify_reports/output_risk_position output link
exists /home/ubuntu/extra/cronjobs/unify_reports/output_risk_position
test /home/ubuntu/market_risk/control_tdrive_mail
test2 /home/ubuntu/market_risk/risk_position/output/2020-02-16risk_position.xlsx  SINGLE [DATA BATCH] risk_position daily report for 2020-02-16risk_position.xlsx
----------base src:   /home/ubuntu/market_risk/control_tdrive_mail
Traceback (most recent call last):
  File "teamdrive_control.py", line 10, in <module>
    import common_config
  File "/home/ubuntu/market_risk/control_tdrive_mail/common_config.py", line 15, in <module>
    from rpt_utils import Parameters
  File "/home/ubuntu/market_risk/control_tdrive_mail/utils/rpt_utils.py", line 1, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

EDIT: new error after 'pip3 install pandas'

Traceback (most recent call last):
  File "risk_position2.py", line 3, in <module>
    import psycopg2 as ps
  File "/home/ubuntu/.local/lib/python3.6/site-packages/psycopg2/__init__.py", line 50, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: libpq.so.5: cannot open shared object file: No such file or directory
/home/ubuntu/extra/cronjobs/unify_reports/output_risk_position2 output link
exists /home/ubuntu/extra/cronjobs/unify_reports/output_risk_position2
test /home/ubuntu/market_risk/control_tdrive_mail
test2 /home/ubuntu/market_risk/risk_position2/output/2020-02-16risk_position2.xlsx SINGLE [DATA BATCH] risk_position2 daily report for 2020-02-16risk_position2.xlsx
----------base src:   /home/ubuntu/market_risk/control_tdrive_mail
Traceback (most recent call last):
  File "teamdrive_control.py", line 10, in <module>
    import common_config
  File "/home/ubuntu/market_risk/control_tdrive_mail/common_config.py", line 16, in <module>
    from db_utils import DBQuery
  File "/home/ubuntu/market_risk/control_tdrive_mail/utils/db_utils.py", line 1, in <module>
    import psycopg2 as ps
  File "/home/ubuntu/.local/lib/python3.6/site-packages/psycopg2/__init__.py", line 50, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: libpq.so.5: cannot open shared object file: No such file or directory

The basic answer is that your interactive shell has a richer environment which is not exported to noninteractive shells. Where is `libpq.so.5` and where have you installed Pandas? Amending `LD_LIBRARY_PATH` withe the former and `PYTHONPATH` with the latter should get you past these errors, but might expose new ones. — tripleee, Feb 17 '20 at 10:23
As such, this is basically a duplicate of https://stackoverflow.com/questions/22743548/cronjob-not-running — tripleee, Feb 17 '20 at 10:24
Tangentially see also [When to wrap quotes around a shell variable?](https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) — tripleee, Feb 17 '20 at 10:31

score 1 · Answer 1 · edited Feb 17 '20 at 09:41

1

Your error is: ModuleNotFoundError: No module named 'pandas' which means your environment does not have pandas yet. Install the library using:

Python 2.x : pip install pandas
Python 3.x : pip3 install pandas

And run it again.

edited Feb 17 '20 at 09:41

Robo Mop

3,485
1
10
23

answered Feb 17 '20 at 09:02

Nikolai Kamenskiy

129
1
3

I have added an edit. Pandas was installed in Conda and I use conda but the pandas error disappear when i pip3 install it. Still 'psycopg27 is install both with Anaconda and pip3 but it is not detected. Any though please? – delalma Feb 17 '20 at 09:47
Actually it is not looking into in '/home/ubuntu/.local' and not in '/home/ubuntu/anaconda3/lib'. I don't understand why. – delalma Feb 17 '20 at 09:53
you install pandas at conda, but run it on python3. Change"python3 teamdrive_control.py..." to "conda teamdrive_control.py ..." [Try this](https://stackoverflow.com/questions/18675907/how-to-run-conda) – Nikolai Kamenskiy Feb 17 '20 at 13:44

Cronjob is not working due to module missing

1 Answers1