Why does copy of Pandas DataFrame Index maintain dependence?

Question

An issue appeared when concatenating two pandas DataFrames and proceeding the update the index. After boiling down the issue we can ignore the concatenation. Despite creating a copy of the new DataFrame or it's index, changing the elements of the copy still changes the original DataFrame index. Below is a basic example you can run to create the issue.

A few alternates that have been tried:

1. ind = df.copy().index.to_numpy(): changing ind alters df
2. ind = df.index.copy().to_numpy(): changing ind alters df
3. ind = df.copy(deep=True).index.to_numpy(): changing ind alters df
4. ind = df.index.copy(deep=True).to_numpy(): changing ind does not alter df.

Why don't options 1-3 behave like option 4?

import pandas as pd

# Define two data frames
df = pd.DataFrame(index=[0,1.,2.], data={'y':[0,0,0]})
print('Original DataFrame')
print(df)

# Update index
ind = df.copy().index.to_numpy() # Option 1
#ind = df.index.copy().to_numpy() # Option 2
#ind = df.copy(deep=True).index.to_numpy() # Option 3
#ind = df.index.copy(deep=True).to_numpy() # Option 4
ind[:] += 3

# Why does the index of (df) get updated?
print("\n\nAfter updating copy of index:")
print(df)

Output (Pandas v1.0.1, Python v3.7.4):

Original DataFrame
     y
0.0  0
1.0  0
2.0  0


After updating copy of index:
     y
3.0  0
4.0  0
5.0  0

Answer 1

The simple answer is, the culprit is to_numpy() (Emphasis mine):

copy: bool, default False
Whether to ensure that the returned value is a not a view on another array. Note that copy=False does not ensure that to_numpy() is no-copy. Rather, copy=True ensure that a copy is made, even if not strictly necessary.

>>> ind = df.copy().index.to_numpy(copy=True)
>>> ind
array([0., 1., 2.])
>>> df
     y
0.0  0
1.0  0
2.0  0
>>> ind += 3
>>> df
     y
0.0  0
1.0  0
2.0  0
>>> ind
array([3., 4., 5.])

Since to_numpy uses np.asarray, it's worthwhile to make note of this bit as well (Emphasis mine):

out : ndarray
Array interpretation of a. No copy is performed if the input is already an ndarray with matching dtype and order. If a is a subclass of ndarray, a base class ndarray is returned.

---

The deeper answer is: the underlying object reference of the index is carried over, unless a true copy is explicitly made on the index, not the df itself. Observe this test:

tests = '''df.index
df.copy().index
df.index.copy()
df.copy(deep=True).index
df.index.copy(deep=True)'''

print('Underlying object reference test...')
for test in tests.split('\n'):

    # !!! Do as I say not as I do  !!!
    # !!! eval will ruin your life !!!

    print(f'{"{:54}".format(f"With {test} is:")}{eval(test).values.__array_interface__["data"]}')
    print(f'{"{:54}".format(f"With {test}.to_numpy() is:")}{eval(test).to_numpy().__array_interface__["data"]}')
    print(f'{"{:54}".format(f"With {test}.to_numpy(copy=True) is:")}{eval(test).to_numpy(copy=True).__array_interface__["data"]}')

Results:

Underlying object reference test...
With df.index is:                                     (61075440, False) # <-- reference to watch for
With df.index.to_numpy() is:                          (61075440, False) # same as df.index
With df.index.to_numpy(copy=True) is:                 (61075504, False) # True copy
With df.copy().index is:                              (61075440, False) # same as df.index
With df.copy().index.to_numpy() is:                   (61075440, False) # same as df.index
With df.copy().index.to_numpy(copy=True) is:          (61075504, False) # True copy
With df.index.copy() is:                              (61075440, False) # same as df.index
With df.index.copy().to_numpy() is:                   (61075440, False) # same as df.index
With df.index.copy().to_numpy(copy=True) is:          (61075504, False) # True copy
With df.copy(deep=True).index is:                     (61075440, False) # same as df.index
With df.copy(deep=True).index.to_numpy() is:          (61075440, False) # same as df.index
With df.copy(deep=True).index.to_numpy(copy=True) is: (61075504, False) # True copy
With df.index.copy(deep=True) is:                     (61075504, False) # True copy
With df.index.copy(deep=True).to_numpy() is:          (61075504, False) # True copy
With df.index.copy(deep=True).to_numpy(copy=True) is: (61075472, False) # True copy of True copy

As you can see, unless the explicit true copy is made on the index directly, or on the to_numpy method, you'll always inadvertently change your existing data.

As to why the True Copies have the same reference (except True copy of True copy), I don't have a full appreciation of what's happening under the hood. But I'm guessing it has to do with some optimization magic to save memory. That however, is probably for another question.

Answer 2

Difference between Deep and Shallow Copy:

In a Shallow Copy: only the reference of the object gets copied, so any change to the original object or copied object causes a change to both objects.

In a Deep Copy :the entire object is copied along with the reference, hence any changes to either of the objects does not affect the other(i.e they are independent objects)

Cases Provide :

The first one is shallow copy and in shallow copy the index gets altered, there is no guarantee of keeping the index intact :

ind = df.copy().index.to_numpy(): changing ind alters df

The second one copies the dataframe and the index but by default it copies shallow and not deep. Hence the index does not stay intact:

ind = df.index.copy().to_numpy(): changing ind alters df

The third one copies all the elements of the dataframe with a deep Copy but since it does not take into account the index, the index gets altered:

ind = df.copy(deep=True).index.to_numpy(): changing ind alters df

As for this last one, the index is part of the deep copy hence it is completely copied keeping index intact, so df has no relation with ind except that ind is a full copy of it and exists independently :

ind = df.index.copy(deep=True).to_numpy(): changing ind does not alter df.

In the above cases, when you make a shallow copy any change in either df or ind results in change of the index. But with deep copy along with the index you have 2 completely independent dataframes.