python script is not running without sudo - why?

Question

This is my python script which downloads the most recent image from my S3 bucket. When I run this script using sudo python script.py it does as expected, but not when I run it as python script.py. In this case, the script finishes cleanly without exceptions or process lockup, but there is no image file.

Why does this happen? Is there something I could do at boto's library end or any other thing?

import boto
import logging


def s3_download():
    bucket_name = 'cloudadic'
    conn = boto.connect_s3('XXXXXX', 'YYYYYYY')
    bucket = conn.get_bucket(bucket_name)

    for key in bucket.list('ocr/uploads'):
        try:
            l = [(k.last_modified, k) for k in bucket]
            key = sorted(l, cmp=lambda x, y: cmp(x[0], y[0]))[-1][1]
            res = key.get_contents_to_filename(key.name)
        except:
            logging.info(key.name + ":" + "FAILED")

if __name__ == "__main__":
     s3_download()

What is "doesn't run" supposed to mean? And why do I even have to keep asking that? — Ignacio Vazquez-Abrams, Oct 14 '16 at 10:32
Do you get an exception? Do you get a zero-length file? Does the process lock up? WHAT DOES IT MEAN? — Ignacio Vazquez-Abrams, Oct 14 '16 at 10:36
no exception, no process lockup, the script execution ends and I do not get the file — prashantitis, Oct 14 '16 at 10:37
Have you considered getting rid of the garbage exception "handling" so you can see what's actually happening? — Ignacio Vazquez-Abrams, Oct 14 '16 at 10:38
Also, you're definitely not just running `python script.py` on *this* script since it just defines a function without calling it. Please post a minimal, complete and verifiable example. — dkasak, Oct 14 '16 at 10:40
You should always have a shebang line at the top: `#!/usr/bin/env python` — , Oct 14 '16 at 10:41
@IgnacioVazquez-Abrams: Sorry I don't know how to do garbage exception handling in python. neither I understood what you said in the last comment — prashantitis, Oct 14 '16 at 10:42
I supoose there is something I should look into boto library, but till now, I haven't found anything — prashantitis, Oct 14 '16 at 10:44
you should compare normal user and _sudo_ environment first place — agg3l, Oct 14 '16 at 10:45
@Guru What Ignacio meant was that you're catching all exceptions unconditionally with that `except` clause since you haven't limited it to a particular class of exceptions (e.g. `except SomeExceptionClass`). This is probably masking the real error. — dkasak, Oct 14 '16 at 10:45
try changing the directory you are trying to save the file to, I bet that is because you need root permissions to write a file where you are trying to save it — Stam Kaly, Oct 14 '16 at 10:45
@StamKaly On that image folder, I have already done `chmod -r 777` not working — prashantitis, Oct 14 '16 at 10:47
@Guru If your script _does_ run and throws an (silent) exception, you can catch it using the `try / except` keywords and then print it out. Read more about this [here](https://docs.python.org/3/tutorial/errors.html). If I was you I'd try to catch every exception, no matter the type, and then print out the type afterwards. — Nearoo, Oct 14 '16 at 10:55
Hey, your access key and secret access key are both fake, right? Otherwise you should to change them immediately! — Nearoo, Oct 14 '16 at 11:02
@Nearoo these are fake, how should I catch this silent exception? — prashantitis, Oct 14 '16 at 11:05
@Guru Just the way I told you, did you look at the page I linked? Search for "except Exception as inst" using Ctrl+F, it is explained there. — Nearoo, Oct 14 '16 at 11:06

score 1 · Answer 1 · answered Oct 14 '16 at 10:59

1

Presumably the problem is that you try to store things somewhere your user doesn't have permission to. The second issue is that your script is hiding the error. The except block completely ignores what sort of exceptions occur (and of course consumes them so you never see them), and uses logging.info which by default doesn't show; that should probably be at least a warning and would be better if it showed something about what went wrong. By the way, odds are you shouldn't be posting S3 authentication keys here.

answered Oct 14 '16 at 10:59

Yann Vernier

15,414
2
28
26

that should not be directory permission issue, I have already done `chmod -R 777` plus these are not actual S3 authentication key for exception handling I am triying, but how do I know which exception is occuring? I – prashantitis Oct 14 '16 at 11:04
If you don't catch the exception it will abort the script and produce a stack trace. If you want to keep running, a call to [traceback.print_exc](https://docs.python.org/3/library/traceback.html#traceback.print_exc) in your except block will print that same traceback for you but allow the script to continue. The logging message can be shown by either increasing severity (use `warning` rather than `info`) or [configuring logging](https://docs.python.org/2/howto/logging.html#configuring-logging) to be more verbose. – Yann Vernier Oct 14 '16 at 11:08

mootmoot · Answer 2 · 2016-10-14T12:53:29.787

Always install virtual environment before you start development in python.

The issue you face is typical python newbie problem : use sudo to install all the pypi package.

When you do sudo pip install boto3, it will install pypi package to system workspace and only accessible by sudo. In such case, sudo python script.py will works, because it has the rights to access the package.

To resolve this issues and isolation of development environment (so you don't pollute different project with differnt pypi package), python developer will install python virtual environment (from above link), then use mkvirtualenv to create your project workspace, run pip install and python setup.py install to install required package to the environment, then you can run python without the sudo python.

Python virtualenv is also deployed inside production environment for the same reason.

Important Note : avoid boto and boto2. AWS no longer support them nor with bug fixing(AWS is not officially support boto2, use it at your own risk). Switch to boto3.

For the exceptional handling issues, @Yann Vernier has mentioned it. And exception error will not logged by logging.info. You may try with logging.debug or simple use raise to raise the actual exception error.

score 0 · Accepted Answer · edited May 23 '17 at 12:06

As @Nearoo in comments had suggested to use except Exception as inst to catch exceptions.

catching the exception like this

except Exception as inst:
            print(type(inst))
            print(inst.args)
            print(inst)

Should fetch you this error if the script is compiled using python 3x If you would compile your script using python 2.7, this error wouldn't come.

Probably you might be having multiple versions of python on your system, which should be the reason for your difference in behavior of python script.py and sudo python script.py and for the same @mootmoot answer suggests to use virtualenv

'cmp' is an invalid keyword argument for this function.

Now if you would have googled this error, you should find that cmp has been deprecated in python 3x and suggestions would be to use key instead.

add these imports

import functools
from functools import cmp_to_key

and replace with these

key2 = sorted(l, key = cmp_to_key(lambda x,y: (x[0] > y[0]) - (x[0] < y[0])))[-1][1]
res = key2.get_contents_to_filename(key2.name)

(x[0] > y[0]) - (x[0] < y[0]) is an alternate to cmp(x[0], y[0])

cmp is replaced by key and cmp_to_key is used from functools lib

Check this out

Thanks man it worked. Actually I was working on openCV lib so installed Anaconda which installed different version python — prashantitis, Oct 14 '16 at 13:20
@Guru : In such case, you need to install `conda` for Anaconda to enable your virtual environment. Because many package under `conda install` is not up to date, it end up no clear advatanges using `conda` vs `virtualenv` for those who not using Anaconda. http://kylepurdon.com/blog/using-continuum-analytics-conda-as-a-replacement-for-virtualenv-pyenv-and-more.html — mootmoot, Oct 14 '16 at 14:12

python script is not running without sudo - why?

3 Answers3