Microsoft Research’s Shahram Izadi and Philip Torr of the University of Oxford have come up with an intuitive way of teaching computers the names of objects while simultaneously creating 3D models of said objects. Their SemanticPaint system learns the names of objects by simple voice commands and can then automatically identify similar objects.
SemanticPaint uses a Kinect or a similar multi-sensor. You simply go around and aim the sensor at the environment or objects you wish to scan to start the process. When you want to name an object, you can either touch that object or draw a border around it, and then say its name. On its visual interface, you’ll see that SemanticPaint marks each named object with a unique color. The example above shows chairs as green, bananas as blue, the floor as red and tables as yellow.
Once you’ve named a few objects, you can order SemanticPaint to enter a test mode, where it will attempt to scan and identify unnamed objects that it thinks are similar to the ones you’ve already named. The beauty of SemanticPaint is that you can interrupt this automated process and correct any errors in real time.
You can read Shahram and Philip’s full paper on Microsoft Research’s website. The researchers hope that SemanticPaint could someday help guide robots and visually impaired people. Then there’s the potential for new augmented reality and other interactive experiences. Perhaps someday we’ll be able to easily spot specific persons in a crowd, extrapolate the appearance of archeological finds or make accurate mockups of unexplored areas. I hope we make sure to mark humans as friends.