Today I finally drew the 'digital/physical' word fragment. There's still the 'building' word, and I'm excited to explore how to create faces in real-time, merging the physical and the digital.
I decided to use Openframeworks for this, and its OpenCV library. Using one of their examples, I avoided reinventing the wheel when it comes to the image filtering pipeline. In this case, background subtraction was used to determine contours.
In order to track the position of the physical shapes, I had to be able to identify them, so a rectangle should only be used to place eyes, for example. This was harder than I expected.
My plan was to adapt the algorithm in this Python tutorial to C++ in Openframeworks:
https://www.opencv-srf.com/2011/09/object-detection-tracking-using-contours.html
In other words, filter and threshold the image, get the contour of object, simplify the number of points, count the number of points, and classify if it was a rectangle, triangle etc.
I had a lot of difficulty identifying what functions to use in the CV library, and how to adjust the parameters; in particular the ofPolyline simplify() function was very useful in minimising the shapes' points. I had to filter the webcam image properly using blurGaussian() and brightnessContrast() using calibrated parameters. The contours were also not completely simplified even when the simplify parameter was very high, often adding one additional point. To classify the objects then I had to redefine the number of points of a rectangle (+1 to 4).
I did not have time to implement tracking for circles as they require a different method altogether (Hough transform). In the video above you could see that they don't work, the outlines drawn are simplified to the extent that they become rectangles or triangles.
In all, I rather enjoyed this process, and it's a step towards tracking these for further projects. I had learnt how to better refine my questions and quickly discard threads that do not apply. In any case, I learnt a lot more about the openCV libraries and how to better treat the image feed before analyzing it.