Image segmentation in Python + Processing

Question

I'm running into difficulties trying to run this image segmentation code.
The idea I have is to take an image such as:

https://i.stack.imgur.com/JFOrs.jpg

and extract all the black squigglies and save each individual squiggly as its own image.
It seems like the code is working, but it's not segmenting my images for some reason.
The error I am getting is: ('segments detected:', 0)

This is the code Im using:

import os, sys  
import numpy as np  
from scipy import ndimage as ndi  
from scipy.misc import imsave  
import matplotlib.pyplot as plt  

from skimage.filters import sobel, threshold_local  
from skimage.morphology import watershed  
from skimage import io  


def open_image(name):  
    filename = os.path.join(os.getcwd(), name)  
    return io.imread(filename, as_grey=True)  


def adaptive_threshold(image):  
    print(type(image))  
    print(image)  
    block_size = 41  
    binary_adaptive = threshold_local(image, block_size, offset=10)  
    binary_adaptive = np.asarray(binary_adaptive, dtype=int)  
    return np.invert(binary_adaptive) * 1.  


def segmentize(image):  
    # make segmentation using edge-detection and watershed  
    edges = sobel(image)  
    markers = np.zeros_like(image)  
    foreground, background = 1, 2  
    markers[image == 0] = background  
    markers[image == 1] = foreground  

    ws = watershed(edges, markers)  

    return ndi.label(ws == foreground)  


def find_segment(segments, index):  
    segment = np.where(segments == index)  
    shape = segments.shape  

    minx, maxx = max(segment[0].min() - 1, 0), min(segment[0].max() + 1, shape[0])  
    miny, maxy = max(segment[1].min() - 1, 0), min(segment[1].max() + 1, shape[1])  

    im = segments[minx:maxx, miny:maxy] == index  

    return (np.sum(im), np.invert(im))  


def run(f):  
    print('Processing:', f)  

    image = open_image(f)  
    processed = adaptive_threshold(image)  
    segments = segmentize(processed)  

    print('Segments detected:', segments[1])  

    seg = []  
    for s in range(1, segments[1]):  
        seg.append(find_segment(segments[0], s))  

    seg.sort(key=lambda s: -s[0])    

    for i in range(len(seg)):  
        imsave('segments/' + f + '_' + str(i) + '.png', seg[i][1])  

folder = os.path.join(os.getcwd(), 'segments')  
os.path.isfile(folder) and os.remove(folder)  
os.path.isdir(folder) or os.mkdir(folder)  
for f in sys.argv[1:]:  
    run(f)

I'll also mention I'm running this Python script from within Processing 3.3.5 using this as my sketch file:

import deadpixel.command.Command;  

static final String BASH =  
  platform == WINDOWS? "cmd /C " :  
  platform == MACOSX? "open" : "xdg-open";  

static final String CD = "cd ", PY_APP = "python ";  
static final String AMP = " && ", SPC = " ";  

static final String PY_DIR = "scripts/";  
//static final String PY_FILE = PY_DIR + "abc.py";  
static final String PY_FILE = PY_DIR + "segmenting.py";  

static final String PICS_DIR = "images/";  
static final String PICS_EXTS = "extensions=,png,jpg,jpeg,gif";  

void setup() {  
  final String dp = dataPath(""), py = dataPath(PY_FILE);  
  final String prompt = BASH + CD + dp + AMP + PY_APP + py;  

  final String pd = dataPath(PICS_DIR);  
  final String pics = join(listPaths(pd, PICS_EXTS), SPC);  

  final Command cmd = new Command(prompt + SPC + pics);  
  println(cmd.command, ENTER);  

  println("Successs:", cmd.run(), ENTER);  
  printArray(cmd.getOutput());  

  exit();  
}

And this in a new tab in processing:
https://github.com/GoToLoop/command/blob/patch-1/src/deadpixel/command/Command.java

Did you edit each individual line of code to be surrounded by tick marks? Note that you can just highlight your code and press the code button, which will indent the block by 4 spaces and mark the whole thing as code. — Kevin Workman, Jul 14 '17 at 22:58

WhoIsJack · Accepted Answer · 2017-07-17T18:23:50.873

A quick investigation reveals the problem: this function here

def adaptive_threshold(image):  
    print(type(image))  
    print(image)  
    block_size = 41  
    binary_adaptive = threshold_local(image, block_size, offset=10)  
    binary_adaptive = np.asarray(binary_adaptive, dtype=int)  
    return np.invert(binary_adaptive) * 1.

is supposed to create a mask of the image by adaptive thresholding - but this goes (very) wrong.

The main reason seems to be a misunderstanding of how threshold_local works: this code expects it to return a binarized segmented version of the input image, when in reality it returns a threshold image, see explanation here.

This is not the only problem, however. For images like the one in your example, offset=10 reduces the threshold produced by threshold_local way too far, so the entire image would be above the threshold.

Here's a working version of the function:

def adaptive_threshold(image):

    # Create threshold image
    # Offset is not desirable for these images
    block_size = 41 
    threshold_img = threshold_local(image, block_size)

    # Binarize the image with the threshold image
    binary_adaptive = image < threshold_img

    # Convert the mask (which has dtype bool) to dtype int
    # This is required for the code in `segmentize` (below) to work
    binary_adaptive = binary_adaptive.astype(int)   

    # Return the binarized image
    return binary_adaptive

If the code is run with this function (with python; this problem has nothing to do with Processing, as far as I can tell), it returns Segments detected: 108 and it produces a nice segmentation:

plt.imshow(segments[0],interpolation='none')
plt.show()

Side note: based on how you phrased your question, am I correct to assume that you did not write this code yourself and that you perhaps have limited expertise in this field?

If so, you may be interested in learning a bit more about python-based image processing and segmentation. I recently ran a short course on this topic that includes a completely self-explanatory hands-on tutorial of a pipeline similar to the one you are using here. The materials are openly accessible, so feel free to have a look.

Edit:

As per your comment, here is a solution that should allow the program to run with full paths as input.

First, remove all this:

folder = os.path.join(os.getcwd(), 'segments')  
os.path.isfile(folder) and os.remove(folder)  
os.path.isdir(folder) or os.mkdir(folder)

so that only this remains:

for f in sys.argv[1:]:  
    run(f)

Next, replace this:

    for i in range(len(seg)):  
        imsave('segments/' + f + '_' + str(i) + '.png', seg[i][1])

by this:

    # Get the directory name (if a full path is given)
    folder = os.path.dirname(f)

    # Get the file name
    filenm = os.path.basename(f)

    # If it doesn't already exist, create a new dir "segments" 
    # to save the PNGs
    segments_folder = os.path.join(folder,"segments")
    os.path.isdir(segments_folder) or os.mkdir(segments_folder)

    # Save the segments to the "segments" directory
    for i in range(len(seg)):
        imsave(os.path.join(segments_folder, filenm + '_' + str(i) + '.png'), seg[i][1])

This solution can handle both files-only input (e.g 'test.png') and path input (e.g. 'C:\Users\Me\etc\test.png').

Edit 2:

For transparency, scipy.misc.imsave allows an alpha layer if arrays are saved as RGBA (MxNx4), see here.

Replace this

        imsave(os.path.join(segments_folder, filenm + '_' + str(i) + '.png'), seg[i][1])

by this

        # Create an MxNx4 array (RGBA)
        seg_rgba = np.zeros((seg[i][1].shape[0],seg[i][1].shape[1],4),dtype=np.bool)

        # Fill R, G and B with copies of the image
        for c in range(3):
            seg_rgba[:,:,c] = seg[i][1]

        # For A (alpha), use the invert of the image (so background is 0=transparent)
        seg_rgba[:,:,3] = ~seg[i][1]

        # Save image
        imsave(os.path.join(segments_folder, filenm + '_' + str(i) + '.png'), seg_rgba)

Edit 3:

For saving into a different target folder with individual subfolders for each segmented image:

Instead of this line

    folder = os.path.dirname(f)

you can specify the target folder, for example

    folder = r'C:\Users\Dude\Desktop'

(Note the r'...' formatting, which produces a raw string literal.)

Next, replace this

    segments_folder = os.path.join(folder,"segments")

by this

    segments_folder = os.path.join(folder,filenm[:-4]+"_segments")

and to be extra-clean replace this

        imsave(os.path.join(segments_folder, filenm + '_' + str(i) + '.png'), seg_rgba)

by this

        imsave(os.path.join(segments_folder, filenm[:-4] + '_' + str(i) + '.png'), seg_rgba)

Wow, thankyou so much! I dont think I ever would have figured that out on my own. You are correct, I do have limited expertise in this field and yes its not my code either. Im getting 108 segments detected as well now. The original mentions "output all segments as PNG images" https://github.com/ntadej/CenturyOfTheSun/blob/master/segmenting.py How to I save each segment as a png? Im also getting a IndexError: tuple index out of range. Not sure if this matters. — , Jul 16 '17 at 17:19
Can you post the error? Saving of segments works fine for me. — WhoIsJack, Jul 16 '17 at 18:12
Here is the error: http://imgur.com/a/lp7yF (sorry, couldnt copy it) I didnt put in `plt.imshow(segments[0],interpolation='none') plt.show()` Yet, where was I supposed to put this? — , Jul 16 '17 at 18:50
There seems to be some sort of confusion: the line that causes the error is actually different from the code you posted in your question and the indexing is wrong. This is the (correct) version in your questions: `miny, maxy = max(segment[1].min() - 1, 0), min(segment[1].max() + 1, shape[1]) ` -- This is the wrong version in your current code: `miny, maxy = max(segment[1].min() - 1, 0), min(segment[2].max() + 1, shape[1]) ` -- Note that one index is changed from **1** (correct) to **2** (wrong). — WhoIsJack, Jul 16 '17 at 19:00
The `imshow` stuff does not strictly *need* to be added, but if you want to add it you can put it after `segments = segmentize(processed)`. — WhoIsJack, Jul 16 '17 at 19:03
Finally, I would get rid of the following line, since as far as I can tell it is not needed but may have the unwanted side-effect of deleting a file that happens to be named the same as the folder the program wants to create for saving the segments: `os.path.isfile(folder) and os.remove(folder) ` — WhoIsJack, Jul 16 '17 at 19:05
Oh whoops, just a typo! Seems to be working, but I think im getting a filename error now: http://imgur.com/a/cTMrY Im assuming this has to do with the last lines of the code? *Ive removed `os.path.isfile(folder) and os.remove(folder)` but im still getting the same error — , Jul 16 '17 at 19:07
The problem is that the code expects its arguments to be just filenames (without a path) and it also expects that it is executed with the directory that contains these files as working directory. I will edit my answer with a solution that works for full paths as input. — WhoIsJack, Jul 16 '17 at 19:20
Actually, one other question I'll ask while you're here is, do you know how I might go about making the background around the squigglies (the white) transparent? See, im taking all these segments and applying them to another image :3 — , Jul 16 '17 at 19:54
Hey again, I have another question for you. How would I go about saving the segments to a folder where I want it, like if I wanted it on the desktop instead? Also, if I load a second image after the first, the segments all go to the same segments folder, so would there be a way of creating a new segments folder each time a new image is loaded so all the segments dont get mixed up? — , Jul 17 '17 at 14:51
I will make an edit that allows you to specify the folder you want (like the Desktop) within the program and that will create multiple folders if you run different images. — WhoIsJack, Jul 17 '17 at 18:13

Image segmentation in Python + Processing

1 Answers1