Skip links

techesoterica.com

A blog dealing with sensory substitution and other esoteric concepts and technologies like speech-recognition and chaos theory

Main navigation

the vOICe

Rich soundscapes and the questions they answer

by ·

I met a researcher in sensory substitution the other evening. We were discussing the vOICe and he commented that he did not realize it could give so much information. This post is to address that gap. Yes, for years, I have been telling people how the vOICe works but I have perhaps not explained the functional benefits of the schema and the kinds of questions a user can answer when she uses this program.

Feature What it does
Horizontal panning or time Allows me to see where objects are in a left to right orientation or the other way round if you prefer. E.G. I can determine if a glass is on my left or on my right.
the pitch maps to height Allows me to distinguish if I am looking at a set of pans on my dining table or at a stack of plates or find the backs of chairs and or sofas
volume is mapped to brightness Answer several questions like Are the lights on? (incredibly handy if your bedroom has two-way switches; is it sunrise or sunset? Am I going to collide with a lit stove?

If we put the above attributes together, you can gather a surprising amount of information about what is around you. Some questions you will be able to answer are:

  • Is the object I am looking rough or smooth? Yes, you can determine texture. Does the wall in front of me have texture paint? Are there pebbles at the bottom of this stream?
  • Am I looking into a well? (Estimating distance accurately is difficult)
  • Is that object with bars on it a Window?
  • Is there writing on this whiteboard?
  • What is the pattern on this piece of fabric?
  • What do those fireworks look like?
  • How does uncle Bob look in this photo? (You may not be able to detect the exact expression on uncle Bob’s face without a lot of image cleanup but will get an idea of his shape)
  • Is the person in front of me clean shaven?
  • What is the hairstyle of the cute lady on my right?
  • Do I live on a tree lined street?
  • What does that sword / insert sculpture of choice behind the glass case in that museum look like?
  • How do the sun, flowing water and the grass look when viewed from this angle?
  • Is the vegetation thinning in this endless forest?
  • Is my food about to boil over?
  • Is that a staircase on my left or in front of me?
  • How do I find my way between these filing cabinets?
  • Am I in an open space and seeing the sky perhaps?

Vision is contextual therefore you will be able to answer these questions and many more depending on how much you practice and how aware you are of your surroundings.

I am happy to add other questions you can answer to this list so add them to the comments or send them to me.

How well do you see what you hear? Michael Proulx at TEDxBathUniversity

by ·

A superb presentation by Dr. Proulx on vision, what the blind can teach us about sight and seeing with sound.

His core message of inclusivity is refreshing and is not something I had thought of before in that way. One of the key advantages of using the vOICe is that you can begin to utilize existing visual clues in your environment. No special infrastructure required. Take museum accessibility. Yes, I am happy to touch things but with the vOICe, I have the ability to see them. You could extend the same to road signs and other land marks or too anything else.

Inclusivity can also give the blind more independence. There is no need for you to rely on someone elses description in buying clothes as just one example.

In terms of biology, the most fascinating portion of the presentation was the bit where Dr. Proulx describes how congenitally blind and sighted people represent objects differently. He mentioned rows and columns when describing how sighted people organize information with respect to objects in their environment. I could not help but think of matrices. How many of you slept in statistics class?

Finally, here is the link to the presentation.

How well do you see what you hear? Michael Proulx at TEDxBathUniversity

My mainstream accessible jungle safaris

by ·

I recently went on an elephant ride and a tiger safari to the Jim Corbett tiger reserve. I used the vOICe on both occasions and was able to enjoy both of them much more. Neither of these experiences were customized for the blind in any way. I was with a group of sighted friends.

The elephant safari was much richer in visual detail since I was closer to the jungle. The tiger safari was in an open Maruti Gypsy. It had bars on the top to hold and was open on all sides. Yes, I could get excellent side views but looking sideways is difficult since my neck is not used to it.

So, what did I see?

  • I could detect changes in flora so I could tell if plants were growing close together or were spaced apart.
  • Detect the water flowing in the river due to its characteristic haziness as it flowed over the rocks.
  • See different plants that the guide was pointing out which was exactly the same thing my sighted friends were experiencing.
  • See other structures such as huts and other manmade construction sites.
  • See the face of the elephant. It was quite narrow.
  • Dodge the leaves on the side of the trail by using the rate of change of leaf density as a guide to distance.

You may well wonder what happened to the animals? Well, they are rather hard to spot even for sighted people. Take the spotted dear as an example. The only reason anyone could spot them was because they moved. The idea is to spot motion quickly. As for the rest, the animals were too far away for me to really see. I was unable to keep changing the frame rate of the vOICe since I wanted detail and changing the frame rate to a higher one would have led to my losing detail.

Tips for other users who take the vOICe or similar technology on a safari

  • Forget about taking still images. Record a video if you can. This is because animals do not wait for you. The jungle is theirs and they go at whatever speed they like. Your sighted companions may be too stunned or too busy to help you capture a still image.
  • Be careful of light levels. It can be dark in the jungle or the light can be muted thanks to the trees. It was when I went.
  • You may have to stand to see certain birds and animals so keep things portable.
  • Wear headphones that do not mask environmental noise. There is plenty of bird song to hear. It is fun trying to look in the direction of interesting bird calls and try and spot the birds.
  • Keep your nose sharp since there is much to smell.
  • Beware of looking at the inside of the vehicle you are in. This is a great temptation since if your neck is anything like mine, you will tire looking sideways. This is not a problem on the elephant safari since you are sitting sideways on the elephant’s back.
  • The usual tips about batteries apply.

Would I have still enjoyed the safaris without the vOICe?

The answer is “not as much.” I would not have been able to participate fully in the experience and would have been stuck with second hand information. Visitors are not allowed to leave the vehicle in Jim Corbett due to safety conserns so I would not have been able to touch the plants let alone see how the forest changed. Yes, some kind of echo location may have helpped but I am not good enough to detect changes in types of trees etc. so the ability to sense things remotely came in handy.

the Taj Mahal a not so wonderful wonder

by ·

On 18 December 2011, I paid a visit to the Taj Mahal in Agra. I was with my cousins and we had spent a pleasant 5 hours driving from Delhi to Agra. The first shock was the snaking security line. Good on them as far as the crowd control goes but why could they not have more ticket counters? Who were they protecting against? What were they protecting?

I was wearing the vOICe so I had my video glasses and netbook. I eventually reached the security check barrier. This was like a doorway. I was frisked and then asked to take off my backpack.

I could not do this since I was all wired up. I was lead to a table where an inspector of police heard my story. I explained the vOICe as an artificial eye.

We were then directed to an officer of the Archeological Survey of India. I had to wait near the security desk while my cousin went to meet with the official.

No argument worked. I had to remove the vOICe and leave it in the tender care of the security staff. The reasoning of the ASI officer was that there was an order by the Supreme Court forbidding laptops. There was no countering that but I am puzzled that the learned judges would allow mobile phones and not laptops? A mobile phone is a computer and can do much more than a laptop.

I did get some photographs with the camera and did use the mobile phone version of the vOICe for a bit but as of this writing, there is nothing that can give me the eye level perspective and the broad field of view like the video glasses with the fish-eye lens.

The high point of my visit came in the souvenir shop outside the Taj Mahal. The staff was welcoming and were happy to show me their tools and products. There is much use of colors so keep a color detector handy or use the color spectrum feature of the vOICe. The staff is very good about adjusting the lighting conditions in the shop to best show off things like stones that are supposed to glow at the time of the full moon. I was able to detect the bright point of light which were the glowing stones under a flashlight.

Tips for any visitors specially those wearing electronic aides.

1. Leave the aide at home or in the vehicle unless it resembles a mobile phone.

2. Do not visit on the weekend since it is very crowded.

3. Keep at least 3 hours for the entire exercise.

4. Hire a guide. Ours was invaluable since besides telling us interesting stories, he was able to minimize the time we spent in the visitor’s line. He knew the shortest routes to get from one point to another. Mind you, you do not have much scope for getting lost since given the crowd, you are almost boxed in.

5. Eat before entering the Taj Mahal since food is not permitted inside the complex.

A video of a walk in the garden

by ·

See this Youtube video made by Dr. Meijer of another walk in his garden. The video supports closed captioning. This is not read by a screen reader so here is his description along with how far in terms of time the events occur in the video.

0: I walk across the terrace towards our neighbor’s house, with our own house of white bricks on the left side. Bright sky is at the top right.

10: a white garden chair shows briefly at the bottom left, and the tonality of the bright cement of the brick wall of our neighbor’s house becomes apparent as I walk towards it until it is within arm’s reach. Then I look left and right along the brick wall such that you get the changing rhythm rates associated with visual perspective.

37: then I take a closer look at the white garden chair, and pass it to walk towards a wooden fence, with a white football lying on the ground in front of it. The football gives a low-pitched beep while the wooden fence gives a slowing rhythm as I move closer.

1.08: I turn left to face the wall with white bricks of our own house, giving a more noisy texture with a tonality from the darker cement, and I move a bit closer to the wall.

1.25: I walk around a kind of white brick pillar at the corner of our house, and approach it from the side that has a rain pipe in the middle, and move up close and put my hand on the rain pipe.

1.55: I turn a bit left and take a close-up look at a red table cloth with white dots on a garden table. The many white dots give a peculiar sound texture.

2.12 I turn further left to take a closer look at a corner of our house, with windows on the left and right right side of the corner.

2.22: I turn left and walk over the grass to another wooden fence, giving a characteristic rhythm that slows down as I move closer.

2.33: Turning a bit to the right I take a look at some nearby foliage giving a kind of random texture.

2.45: Next I take a look at the roof with its regular pattern made up of the rows and columns of stone roof tiles.

3.06: I walk across the terrace to a white door in the fence that faces the street behind it.

3.31: I briefly turn right looking towards the white brick wall, and then turn left for a close-up look at some other foliage, and turn further left to effectively make a U-turn after facing another brick wall.

3.58: Then walking back across the terrace I turn to the right to have a close-up look at an empty rabbit’s cage, which here gives a characteristic simultaneous tonality and rhythm at close range, corresponding to a rectangular metal wire pattern.

4.08: I walk further across the terrace to take a second look at the white brick pillar with the rain pipe, again putting my hand on the rain pipe at close range.

4.38: end of video clip.

A walk in the garden experience

by ·

This is a description of a walk that Dr. Peter Meijer the creator of the vOICe took on one of the first days of spring 2010. Note the description of texture. It is these kinds of descriptions that provide an insite to what can be experienced using the vOICe.

“Hi All,

Today was the first really beautiful Spring day in The Netherlands, with lovely sunshine and an ideal temperature, and I decided to go outside in my garden at home wearing The vOICe with camera glasses.
Of course I am not blind, but it was an excellent occasion to get a sense of what is possible in a realistic (safe, home) environment.

I found it very enjoyable to be able to hear all the rich textures.
The brick wall of our neighbor’s house is up made of relatively dark stones with brighter cement, and the cement pattern thereby gave the very characteristic dense tonality of many horizontal lines. The brick wall of our own house is made up of white bricks with darker cement, also giving a tonality, but now with the added “white” noise.
A curious rise in pitch in this tonal sound took a second for me to realize that it was the shadow of our neighbor’s house cast on our white brick wall. Since one cannot touch shadows, such “anomalies”
would initially confuse blind users. Looking at the roof I heard the regular pattern made up of the rows and columns of stone roof tiles, at a slanted orientation and tapered due to visual perspective.
Looking down, the grass gave a smoother sounding texture than the stone gravel tiles of my terrace. Along the white brick wall I could hear the interruptions by the windows, with their wooden lining and horizontal and vertical “spokes”. Also very nice was the “V shape”
that I heard when looking at a corner of our house, as it was caused by the effect of visual perspective on the wooden window lining of the windows on both sides of the corner: quite helpful to know that one is looking at a corner. Our wooden fence gave a characteristic strong rhythm, and the foliage gave different types of swirly sounds for the somewhat random twigs, branches and leaves. The rich variation in textures made it fairly easy to know in which direction I was looking and where I was relative to the various structures. Distance could to some extent also be judged from the appearance of the visual textures.

I know that many (sighted) people dislike the sounds of The vOICe, but here the soundscapes did not bother me at all. It is as if the rich visual feedback provided by them for me suppresses that negative aspect. I once compared this phenomenon to learning to read, where at some point you enjoy a beautiful story or poem no matter how ugly the font is with which it is printed – you just don’t really “see”
the font anymore.

Anyway, just sharing today’s personal experience, and other people would undoubtedly have other experiences and opinions.

Best wishes,

Peter Meijer”

Response to: For the Blind, Technology Does What a Guide Dog Can’t

by ·

My response to the New York Times article at
For the Blind, Technology Does What a Guide Dog Can’t

This is a good step in making Google’s Android accessible. However, using my Nokia E51 with Wayfinder Access and the vOICe, I can already do all of the things outlined in the article.

As regards touch screens, thank you. Give me a good keyboard and or speech-recognition any day. Finally, if I ever had to use a touch screen, I would rather have a single fixed point represent an item on the screen rather than an adaptive one. Just imagine having to type 5 before you type the actual number you want to dial or, am I misinterpretting the description in the article?