In many circles, the holy grail of blind mobility aides has been a system that “tells” the user what is in front of her. We are close to achieving this thanks to an application called TapTapSee made by Net Ideas LLC. TapTapSee runs on the iPhone and I tested the application this afternoon while returning from the world book fair. You point the camera in whichever direction you want and double tap the relevant button on the screen to take a picture. The picture is sent to the TapTapSee server where the image processing and object recognition takes place. A text description is then returned to the phone where it is displayed.

TapTapSee correctly identified the following things.

• The color of the car that was standing next to mine. I was photographing out of the right rare passenger window.
• My car seat.
• The name and type of my car when I photographed the rare of the car which included the name plate.

The more interesting part of this exercise was the misrecognitions.

• A gray railing was identified as a gray gate.
• A stand of trees was identified as a parking lot.
• A number of cars parked along the side of the road was also identified as a parking lot.
• The underside of my car’s roof was identified as a white wall.

These misrecognitions demonstrate the importance of context. Yes, my car’s roof is a white fabric with a metal top but taken in isolation, it could be a wall. The same applies to the cars parked at the side of the road. Therefore, our next step has to be context aware object recognition. GPS may provide some limited help but I don’t know how my phone is supposed to know if I am in a car, standing on the road or in a shop.

This emphasizes the relevance of raw sensory input so that we can integrate said input with our knowledge of context and come to a valid conclusion about our surroundings. Systems such as the the vOICe as well as the good old white cane and the many varieties of sonar based devices already provide suitable sensory extensions. As in many such cases, it is up to us to construct our own technology mix.