Standardize Pixel Input Data with many Zeros

Question

I wanna standardize my input data for a neural network.

Data looks like this:

data= np.array([[0,0,0,0,233,2,0,0,0],[0,0,0,23,50,2,0,0,0],[0,0,0,0,3,20,3,0,0]])

This is the function that I used. It doesn't work because of the zeros.

def standardize(data): #dataframe
    _,c = data.shape
    data_standardized = data.copy(deep=True)
    for j in range(c):
        x = data_standardized.iloc[:, j]
        avg = x.mean()
        std = x.std()
        x_standardized = (x - avg)/ std
        data_standardized.iloc[:, j] = x_standardized

    return data_standardized

What do you mean it doesnt work because of zeors. What error you are getting? — Poojan, Nov 20 '19 at 19:29
@Poojan if the first feature of all samples is zero then I get a sdt = 0 so i div. by zero — Steven, Nov 20 '19 at 19:32
Have a look at this answer it will help. https://stackoverflow.com/questions/26248654/how-to-return-0-with-divide-by-zero. When divided by zero you can put that as 0. — Poojan, Nov 20 '19 at 19:49

wwii · Answer 1 · 2019-11-21T15:07:13.290

Use boolean indexing to avoid dividing by zero:

In [90]: data= np.array([[0,0,0,0,233,2,0,0,0],[0,0,0,23,50,2,0,0,0],[0,0,0,0,3,20,3,0,0]])

In [91]: new = np.zeros(data.shape)

In [92]: m = data.mean(0)

In [93]: std = data.std(0)

In [94]: r = data-m

In [95]: new[:,std.nonzero()] = r[:,std.nonzero()]/std[std.nonzero()]

In [96]: new
Out[96]: 
array([[ 0.        ,  0.        ,  0.        , -0.70710678,  1.3875163 ,
        -0.70710678, -0.70710678,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        ,  1.41421356, -0.45690609,
        -0.70710678, -0.70710678,  0.        ,  0.        ],
       [ 0.        ,  0.        ,  0.        , -0.70710678, -0.9306102 ,
         1.41421356,  1.41421356,  0.        ,  0.        ]])

Or use sklearn.preprocessing.StandardScaler.

Your function refactored:

def standardize(data): #dataframe
    data = data.values
    new = np.zeros(data.shape)
    m = data.mean(0)
    std = data.std(0)
    new[:,std.nonzero()] = r[:,std.nonzero()]/std[std.nonzero()]
    return pd.DataFrame(new)

Standardize Pixel Input Data with many Zeros

1 Answers1