tensorflow-image-convolution-edge-detection

The idea

So while on deep learning workshop in Melbourne. I was playing around with tensor flow and seeing it was really easy to do convolutions over images with it. (Workshop was at Silverpond and would highly recommend it)

So in short convolution is comparing two signals with each other the more they resemble each other the stronger the result.
This makes for super signal filters btw.(Electronic engineering for the win)

I remember seeing this trick in an old textbook called the Sobel_operator and wanted to give it a try.

The sobel operator is basicly an edge detection on image. But what I wanted to do it in horizontal and vertical direction then form a vector. I could then colour the image based on direction of the edge to make pretty images.

Loading image and libaries

For the image i picked a very plain cup with very clear edges. An image with heavy texture will have to many edges detected and not look as impresive.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

import skimage.io
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
%matplotlib inline
import math
image = skimage.io.imread('coffee.jpg', as_grey=True)
plt.imshow(image, cmap='gray')
print(image.shape) 

Note when installing tensor flow for the first time. Use the CPU version getting the GPU version to work took me a while getting all the correct version loaded and working together.(Think its even easier on AWS as there are prepared images)

Doing calculations

So the next code block I’m setting up the object in tensor flow. it isnt activated just yet. Its basicly building a pipeline for information to flow through. Note the 3 by 3 sobel vectors in code for horisontal and vertical.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

tf.reset_default_graph()
# Write the kernel weights as a 2D array. 
kernel_h = np.array([3, 3])
kernel_h = [ [1,2,1], [0,0,0], [-1,-2,-1] ]
kernel_v = np.array([3, 3])
kernel_v = [ [1,0,1], [2,0,-2], [-1,0,-1] ]
# Kernel weights
if len(kernel_h) == 0 or len(kernel_v) == 0:
    print('Please specify the kernel!')
input_placeholder = tf.placeholder(
    dtype=tf.float32, shape=(1, image.shape[0], image.shape[1], 1))
with tf.name_scope('convolution'):
    conv_w_h = tf.constant(kernel_h, dtype=tf.float32, shape=(3, 3, 1, 1))
    conv_w_v = tf.constant(kernel_v, dtype=tf.float32, shape=(3, 3, 1, 1))    
    output_h = tf.nn.conv2d(input=input_placeholder, filter=conv_w_h, strides=[1, 1, 1, 1], padding='SAME')
    output_v = tf.nn.conv2d(input=input_placeholder, filter=conv_w_v, strides=[1, 1, 1, 1], padding='SAME')

I the next code block i activate the tensor flow session doing the calculations and producing our output vectors.

1
2
3
4
5
6
7
8
9
10

with tf.Session() as sess:
    result_h = sess.run(output_h, feed_dict={
            input_placeholder: image[np.newaxis, :, :, np.newaxis]})
    result_v = sess.run(output_v, feed_dict={
            input_placeholder: image[np.newaxis, :, :, np.newaxis]})
#plt.imshow(result_h[0, :, :, 0]) # view horisontaal edges
#plt.imshow(result_v[0, :, :, 0]) # view vertical edges

Going some trigonometry

So time for some trig to calculate the strength and angle of the vector.
First we calculate the streng as shown in this image.

This was generated using the following code. (plain old Pythagoras)

1
2
3
4

result_lenght = ((result_v**2) + (result_h**2))**0.5  
plt.imshow(result_lenght[0, :, :, 0], cmap='hot')

Next we calculate the angle of the vector. Note that this image always has a value or angle.
In low strength areas its very noisy as the ratio between x and y swings easily. (a signal to noise example)

Its calculated with following code. I added very small value to avoid divide by zero issues. (Samll cheat alwas productive)

1
2
3
4

result_angle = (np.arctan(result_v/(result_h+0.00000001)))#*(2*math.pi)
plt.imshow(result_angle[0, :, :, 0], cmap='hot')

Final result

So in the end we combine the two.
We do this by having a strong angle for red , green and blue. Done by diffrent offsets in cosine value.
its also important to normalize each value.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#normalize like crazy
result_lenght_norm = (result_lenght[0,:,:,0] + (np.min(result_lenght)*-1) ) / (np.min(result_lenght)*-1 + np.max(result_lenght))
result_angle_norm = result_angle[0,:,:,0]
result_red = np.absolute(result_lenght_norm * np.cos(result_angle_norm+4.2))
result_green = np.absolute(result_lenght_norm * np.cos(result_angle_norm+2.1))
result_blue = np.absolute(result_lenght_norm * np.cos(result_angle_norm))
result_rgb = np.zeros((580,705,3))
result_rgb[...,0] = (result_red + (np.min(result_red)*-1) ) / (np.min(result_red)*-1 + np.max(result_red)) 
result_rgb[...,1] = (result_green + (np.min(result_green)*-1) ) / (np.min(result_green)*-1 + np.max(result_green)) 
result_rgb[...,2] = (result_blue + (np.min(result_blue)*-1) ) / (np.min(result_blue)*-1 + np.max(result_blue))
#result_rgb
plt.imshow(result_rgb)

Just by shifting angles between colours you can change image vector

So in summary I really enjoyed getting to do this algorithm as i remember seeing it and wanting to replicate it.

Its important to note that shadows will get picked up as edges.
But i was also wondering if you have enough images from diffrent direction you can generate a polygon :p

Ive uploaded the notebook to my git if you want to give it a try.
Im thinking ill do a RNN next.