unexpected behaviour of dictionary membership check

Question

An unhashable object cannot be inserted into a dict. It is documented, there is good reason for it.

However, I do not understand why it affects a membership test:

if value not in somedict:
    print("not present")

I was assuming that a membership test can return only True or False. But when the value is unhashable, it fails with TypeError: unhashable type. I would say True should be the correct answer of this not in test, because the value is clearly not contained in somedict and it is fully irrelevant that it cannot be inserted.

Another example:

try:
    result = somedict[value]
except KeyError:
    # handle the missing key

When the value is unhashable, it fails with TypeError: unhashable type I would expect the KeyError instead.

Also:

somedict.get(value, default)

does not return the default, but throws the TypeError.

---

So why unhashable in somedict does not evaluate to False and what is the correct test returning just True or False?

---

Update:

object.__contains__(self, item)

Called to implement membership test operators. Should return true if item is in self, false otherwise.

(from "Data Model - Python Documentation")

---

Appendix:

This is a simplified part of a user interface program, which failed when one of the arguments was a dict.

# args = list of function arguments created from user's input
# where "V1" stands for value1 and "V2" stands for value2
XLAT = {'V1': value1, 'V2': value2} 
args = [XLAT.get(a, a) for a in args]
function(*args)

Answer 1

The reason is that the test for a potential key being part of a dictionary is done by generating the hash value of the potential key. If the potential key cannot provide a hash value (if the object is not hashable), the test cannot take place.

You are right, in a way, that in this case the existence test could just say "no, not present" ("because it cannot be inserted anyway").

This is not done this way because it could cloak a lot of programming errors.

If you program cleanly, you will most likely never check for an unhashable object whether it is in a dictionary or not. I. e. it requires quite an amount of fantasy to come up with a case where you actually would. (I wouldn't say it is completely out of the question, though.) The amount of cases where such a check is only happening because a programming error leads to a situation where you do something accidentally, is way larger. So the exception indicates that you should look at the spot in the code.

If you know what you are doing (maybe the case in your situation), you should simply catch that error:

try:
    if strange_maybe_unhashable_value in my_dict:
        print("Yes, it's in!")
    else:
        print("No, it's not in!")
except TypeError:
    print("No, it's not even hashable!")

If you want to combine that with your KeyError handling:

try:
    result = somedict[value]
except (KeyError, TypeError):
    # handle the missing key

or

try:
    result = somedict[value]
except KeyError:
    # handle the missing key
except TypeError:
    # react on the thing being unhashable

To provide another aspect which is rather esoteric:

An object might be hashable at some time and unhashable at another (maybe later). This should of course never be the case but can happen, e. g. if the hash value relies on something external. Despite common assumptions, being hashable is independent from being immutable (although one often relies on the other). So an object could change while being part of a dictionary, and this could change its hash value. While this is an error in itself and will lead to having the dictionary not working properly, this might also be a design-reason not to simply say "not present" or raise a KeyError but raise the TypeError instead.

Answer 2

Maybe you could check if it's hashable, if it is: try your piece of code, if not: return False. This is not an answer to your 'why' question, but at least I think it'll work.

Answer 3

If you're trying to test a key which cannot possibly exist in a dict, it indicates a logic error or possibly a typo of you trying to test against the wrong variable. Why would you even attempt the test if it can never be true; likely you want to fix that issue in your code.

It makes sense to me to treat these as two different cases which can really aid in debugging. If you do have a case where you're mixing hashable and unhashable types and this is explicitly not in error, you should make that clear:

try:
    if value not in somedict:
        ...
except TypeError:
    ...

Answer 4

I think this is more of an implementation decision. Which makes sense : unhashable type should return a specific type of error.

Imagine you implement a Counter class, containing a Dictionary which just counts objects.

cnt.Add(x) # adds one to the counter
cnt.Count(x) # returns the number of occurences of x as seen by cnt, 0 if x has never been seen beofreL.

You run it on a collection of objects and one of them happens to be unhashable, but you do not know it. Then when you want to look at the number of occurences of this object, your program will return 0, which would not be true.

On the other hand, the fact that there is a specific exception when the type is unhashable let the developer chose the behavior to implement. If there was one single exception for these two cases, you would not have this choice.

Answer 5

Use it as:

if value not in somedict.values():
    print("not present")