Can't understand python variable scopes for callbacks and with blocks

Question

While writing a function that will show a progress bar while downloading a list of files from an ftp server I ended up with some strange script:

with ftplib.FTP(...) as ftp:
  files = list(ftp.mlsd(path=remote_dir_path))
  total_size = sum(int(f[1]['size']) for f in files)
  total_done = 0
  current_done = 0
  current_size = 0

  def callback(chunk):
    global current_done
    global total_done
    global current_size
    global total_size
    trg_file.write(chunk) # how? why it knows what is `trg_file`?
    current_done += len(chunk)
    total_done += len(chunk)
    print_progressbar(current_done,total_done,...)

  for file in files:
    current_done = 0
    current_size = int(file[1]['size'])
    with open(file[0],'wb') as trg_file:
      ftp.retrbinary('RETR %s/%s'%(remote_dir_path,file[0]),callback)

I had to write global in callback function so that python does not complain about it. BUT it knew what is trg_file no problem?!?!?! Trying to understand I wrote a small test:

def callback():
  print(foo)

def main():
  foo = 'FOOOO'
  callback()

main()

It failed as expected.

I get why the second script does not work. Intuition says that the first should not work too. But it works. Can somebody please explain why? EDIT: The question is not about global itself, but about with ... as ... block and scope (scope of ... as variable variable)

Answer 1

When you do something like this:

def abc():
    x = 5

The x variable is local, same applies to

def abc():
    x += 5

Because it basically is x = x + 5, assign operator again. Consider this one:

x = 1
def abc():
    print(x)

What will hapen here? x will be printed because x is available in the abc function scope.

Now consider this one:

x = 1
def abc():
    print(x)
    x = 5
    print(x)

abc()

What will happen? Will the first print print 1? The answer is no because later on in the function definition there is x = 5, so x will be treated as a local variable in whole the function, the first print included - it won't look to global scope, because there is an x variable in local scope. It wasn't assign yet when first print is executed, that's why there will be an error.

Now with statement doesn't create a scope. That's why you need global keyword if you want to assign to (global) variables: total_size, total_done, current_done, current_size.

To ilustrate why some variables doesn't need global keyword consider this snippet:

some_list = [1, 2, 3]
def abc():
    some_list = ['a', 'b', 'c']  # assignment to variable some_list, it is 
                                 # a local variable, it has nothing to do
                                 # with global some_list (it references a different object)
    print(some_list)

abc()
print(some_list)
>>> ['a', 'b', 'c']
>>> [1, 2, 3]

def cba():
    some_list.append(4)  # now there is no assignment so some_list is a global variable and references the same object in memory from the beginning

cba()
print(some_list)
>>> [1, 2, 3, 4]

If it's still confusing, there is this nice website when you can visualize which variable references which object in memory here: http://pythontutor.com/visualize.html.