Logically merging physically distinct dictionaries

Question

I am trying to write a wrapper around two dictionaries, so that they seem like one dictionary (for reading only; writing should raise Exceptions).

I am doing this to save memory, since one of the original dictionaries is needed elsewhere. I also think it's faster than merging the dictionaries, if less than half the elements in the combined dictionaries are going to be looked up.

Here's my attempt:

class LogicalMerge:
  def __init__(self, d1, d2):
    #d1 and d2 are dictionaries
    self.d1 = d1
    self.d2 = d2
  def __getitem__(self, x):
    if x in self.d1:
      return self.d1[x]
    else:
      return self.d2[x]

d1 = {1:2, 3:4}
d2 = {5:10}
d = LogicalMerge(d1, d2)
d[1] # == 2
d[5] # == 10

Are there any design, technical, or performance problems with this approach?

Answer 1

You can save yourself one lookup per call by rewriting __getitem__ as

try:
    return self.d1[x]
except KeyError:
    return self.d2[x]

This follows the pythonic idiom of "asking forgiveness, not permission".

I also think it's faster than merging the dictionaries

I strongly doubt that, but you should measure to be sure. Your approach introduces an extra level of indirection and requires the key to be hashed multiple times. It's also bound to take more memory.

Edit: here's an alternative approach. Replace one of your dicts with a DictWithBackup, which behaves like a dict except that when a key is missing, it looks into the other dict.

class DictWithBackup(dict):
    def __init__(self, backup):
         self._backup = backup

    def __missing__(self, key):
         return self._backup[key]

This versions avoids the exception handling.

Answer 2

For performance reasons I would prefer the following. Given None is an object that never validly occurs.

def __getitem__(self, k):
  v = self.d1.get(k, None)
  if v is None:
    v = self.d2[k] # if you're going to raise an error anyway ...
  return v

Otherwise you could default-get a custom object. Note, that you either need an object that implements __eq__ to test value equality (o1 == o2), or---performance-wise even better---that you use an immutable object, i.e. a certain string "error_key_not_found_string", that is not newly created every time. Then you may even compare by object identity id(o1) == id(o2), i.e. using the is operator. (You don't need to provide __eq__ then either.)

def __getitem__(self, k):
  v = self.d1.get(k, "error_key_not_found_string")
  # if id(v) == id("error_key_not_found_string":
  if v is "error_key_not_found_string": 
    v = self.d2[k] # if you're going to raise an error anyway ...
  return v

Have you thought about the case, where the item is in both dictionaries?

In conclusion, I find this a bit confusing from the design perspective. Is the performance gain really justifying the additional source of error and confusion? Plus you will lose all the other dict functionality... This could be as easy as d1.update(d2). Given d1 is the dictionary you don't use elsewhere (you could use a deepcopy then).