Replacing existing column in dask map_partitions gives SettingWithCopyWarning

Question

I'm replacing column id2 in a dask dataframe using map_partitions. The result is that the values are replaced but with a pandas warning.

What is this warning and how to apply the .loc suggestion in the example below?

pdf = pd.DataFrame({
    'dummy2': [10, 10, 10, 20, 20, 15, 10, 30, 20, 26],
    'id2': [1, 1, 1, 2, 2, 1, 1, 1, 2, 2],
    'balance2': [150, 140, 130, 280, 260, 150, 140, 130, 280, 260]
})

ddf = dd.from_pandas(pdf, npartitions=3) 

def func2(df):
    df['id2'] = df['balance2'] + 1
    return df

ddf = ddf.map_partitions(func2)

ddf.compute()

C:\Users\xxxxxx\AppData\Local\Temp\ipykernel_30076\248155462.py:2: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy df['id2'] = df['balance2'] + 1

SultanOrazbayev · Accepted Answer · 2022-05-21T07:03:41.683

2

A quick fix is to add copy of the dataframe:

def func2(df):
    df = df.copy() # will make a copy of the dataframe
    df['id2'] = df['balance2'] + 1
    return df

However, as I understand, copying of the dataframe is not required as the delayed nature of the dask dataframe means that the changes are not propagated back to the dask dataframe partitions.

Update: there is a relevant question which explains the reason for .copy in pandas. In the snippet below applying the function will modify the original pandas dataframe, which might be undesirable:

from pandas import DataFrame

def addcol(df):
    df['a'] = 1
    return df

df = DataFrame()

df1 = addcol(df)
# without .copy, df is also modified, which might be undesirable

In the context of dask this warning is just that, a warning, so .copy is not needed.

from dask.dataframe import from_pandas
ddf = from_pandas(df, npartitions=1)
ddf1 = ddf.map_partitions(addcol)
# will show warning, but original ddf is not modified

edited May 21 '22 at 07:03

answered May 20 '22 at 18:11

SultanOrazbayev

14,900
3
16
46

What do you achieve by copying the dataframe? – ps0604 May 20 '22 at 18:13
If you were working with pandas dataframe, then running the original `func2` would modify the dataframe that is passed to `func2` (i.e. both the input dataframe and output dataframe would be modified). – SultanOrazbayev May 20 '22 at 18:14
This worked, however I'm still confused when should I copy the dataframe and when not. – ps0604 May 20 '22 at 18:25
This expands on it: https://stackoverflow.com/q/27673231/10693596 – SultanOrazbayev May 20 '22 at 18:30
Ok, so creating a new reference with copy will remove the warning, I get that, but you also said that "copy is not required"? – ps0604 May 20 '22 at 18:59
Kindly check the updated answer. – SultanOrazbayev May 21 '22 at 05:15

Replacing existing column in dask map_partitions gives SettingWithCopyWarning

1 Answers1