append in a list causing value overwrite

Question

I am facing a peculiar problem. I will describe in brief bellow Suppose i have this piece of code -

class MyClass:

   __postBodies = []
    .
    .
    .
for the_file in os.listdir("/dir/path/to/file"):
      file_path = os.path.join(folder, the_file)
      params = self.__parseFileAsText(str(file_path)) #reads the file and gets some parsed data back
      dictData = {'file':str(file_path), 'body':params}
      self.__postBodies.append(dictData)
      print self.__postBodies
      dictData = None
      params = None

Problem is, when i print the params and the dictData everytime for different files it has different values (the right thing), but as soon as the append occurs, and I print __postBodies a strange thing happens. If there are thee files, suppose A,B,C, then

first time __postBodies has the content = [{'body':{A dict with some data related to file A}, 'file':'path/of/A'}]

second time it becomes = [{'body':{A dict with some data relaed to file B}, 'file':'path/of/A'}, {'body':{A dict with some data relaed to file B}, 'file':'path/of/B'}]

AND third time = [{'body':{A dict with some data relaed to file C}, 'file':'path/of/A'}, {'body':{A dict with some data relaed to file C}, 'file':'path/of/B'}, {'body':{A dict with some data relaed to file C}, 'file':'path/of/C'}]

So, you see the 'file' key is working very fine. Just strangely the 'body' key is getting overwritten for all the entries with the one last appended.

Am i making any mistake? is there something i have to? Please point me to a direction.

Sorry if I am not very clear.

EDIT ------------------------

The return from self.__parseFileAsText(str(file_path)) call is a dict that I am inserting as 'body' in the dictData.

EDIT2 ----------------------------

as you asked, this is the code, but i have checked that params = self.__parseFileAsText(str(file_path)) call is returning a diff dict everytime.

def __parseFileAsText(self, fileName):

        i = 0
        tempParam = StaticConfig.PASTE_PARAMS

        tempParam[StaticConfig.KEY_PASTE_PARAM_NAME] = ""
        tempParam[StaticConfig.KEY_PASTE_PARAM_PASTEFORMAT] = "text"
        tempParam[StaticConfig.KEY_PASTE_PARAM_EXPIREDATE] = "N"
        tempParam[StaticConfig.KEY_PASTE_PARAM_PRIVATE] = ""
        tempParam[StaticConfig.KEY_PASTE_PARAM_USER] = ""
        tempParam[StaticConfig.KEY_PASTE_PARAM_DEVKEY] = ""
        tempParam[StaticConfig.KEY_PASTE_FORMAT_PASTECODE] = ""

        for line in fileinput.input([fileName]):
            temp = str(line)
            temp2 = temp.strip()
            if i == 0:              
                postValues = temp2.split("|||")

                if int(postValues[(len(postValues) - 1)]) == 0 or int(postValues[(len(postValues) - 1)]) == 2:
                    tempParam[StaticConfig.KEY_PASTE_PARAM_NAME] = str(postValues[0])

                    if str(postValues[1]) == '':                                        
                        tempParam[StaticConfig.KEY_PASTE_PARAM_PASTEFORMAT] = 'text'
                    else:
                        tempParam[StaticConfig.KEY_PASTE_PARAM_PASTEFORMAT] = postValues[1]

                    if str(postValues[2]) != "N":
                        tempParam[StaticConfig.KEY_PASTE_PARAM_EXPIREDATE] = str(postValues[2])

                    tempParam[StaticConfig.KEY_PASTE_PARAM_PRIVATE] = str(postValues[3])
                    tempParam[StaticConfig.KEY_PASTE_PARAM_USER] = StaticConfig.API_USER_KEY
                    tempParam[StaticConfig.KEY_PASTE_PARAM_DEVKEY] = StaticConfig.API_KEY

                else:
                    tempParam[StaticConfig.KEY_PASTE_PARAM_USER] = StaticConfig.API_USER_KEY
                    tempParam[StaticConfig.KEY_PASTE_PARAM_DEVKEY] = StaticConfig.API_KEY
                i = i+1
            else:
                if tempParam[StaticConfig.KEY_PASTE_FORMAT_PASTECODE] != "" :           
                    tempParam[StaticConfig.KEY_PASTE_FORMAT_PASTECODE] = str(tempParam[StaticConfig.KEY_PASTE_FORMAT_PASTECODE])+"\n"+temp2
                else:
                    tempParam[StaticConfig.KEY_PASTE_FORMAT_PASTECODE] = temp2

        return tempParam

Can you show the code of `self.__parseFileAsText`? It looks like the same dictionary is being reused which leads to the value of the `'body'` key being altered each time. — Simeon Visser, Aug 27 '12 at 16:14
Do yourself a favor, don't use double-underscore names. A single underscore is enough to indicate that others shouldn't use it. — Ned Batchelder, Aug 27 '12 at 16:27

score 2 · Accepted Answer · edited May 23 '17 at 12:27

You are likely returning the same dictionary with every call to MyClass.__parseFileAsText(), a couple of common ways this might be happening:

__parseFileAsText() accepts a mutable default argument (the dict that you eventually return)
You modify an attribute of the class or instance and return that instead of creating a new one each time

Making sure that you are creating a new dictionary on each call to __parseFileAsText() should fix this problem.

Edit: Based on your updated question with the code for __parseFileAsText(), your issue is that you are reusing the same dictionary on each call:

tempParam = StaticConfig.PASTE_PARAMS
...
return tempParam

On each call you are modifying StaticConfig.PASTE_PARAMS, and the end result is that all of the body dictionaries in your list are actually references to StaticConfig.PASTE_PARAMS. Depending on what StaticConfig.PASTE_PARAMS is, you should change that top line to one of the following:

# StaticConfig.PASTE_PARAMS is an empty dict
tempParam = {}

# All values in StaticConfig.PASTE_PARAMS are immutable
tempParam = dict(StaticConfig.PASTE_PARAMS)

If any values in StaticConfig.PASTE_PARAMS are mutable, you could use copy.deepcopy but it would be better to populate tempParam with those default values on your own.

You are right. so silly of me. anyway, it is done now. Thanks for the help :) — SRC, Aug 27 '12 at 16:30

Pierre GM · Answer 2 · 2012-08-27T17:17:37.840

0

What if __postBodies wasn't a class attribute, as it is defined now, but just an instance attribute?

edited Aug 27 '12 at 17:17

answered Aug 27 '12 at 16:36

Pierre GM

19,809
3
56
67

i think in case of mine it is fine to be a class attribute, as i am using only one reference to MyClass elsewhere to call the function which does the job. but you are right for cases where this is not the condition. a good reference can be found here - http://stackoverflow.com/questions/207000/python-difference-between-class-and-instance-attributes – SRC Aug 27 '12 at 16:53

append in a list causing value overwrite

2 Answers2