Skip links

techesoterica.com

A blog dealing with sensory substitution and other esoteric concepts and technologies like speech-recognition and chaos theory

Main navigation

vision

My mainstream accessible jungle safaris

by ·

I recently went on an elephant ride and a tiger safari to the Jim Corbett tiger reserve. I used the vOICe on both occasions and was able to enjoy both of them much more. Neither of these experiences were customized for the blind in any way. I was with a group of sighted friends.

The elephant safari was much richer in visual detail since I was closer to the jungle. The tiger safari was in an open Maruti Gypsy. It had bars on the top to hold and was open on all sides. Yes, I could get excellent side views but looking sideways is difficult since my neck is not used to it.

So, what did I see?

  • I could detect changes in flora so I could tell if plants were growing close together or were spaced apart.
  • Detect the water flowing in the river due to its characteristic haziness as it flowed over the rocks.
  • See different plants that the guide was pointing out which was exactly the same thing my sighted friends were experiencing.
  • See other structures such as huts and other manmade construction sites.
  • See the face of the elephant. It was quite narrow.
  • Dodge the leaves on the side of the trail by using the rate of change of leaf density as a guide to distance.

You may well wonder what happened to the animals? Well, they are rather hard to spot even for sighted people. Take the spotted dear as an example. The only reason anyone could spot them was because they moved. The idea is to spot motion quickly. As for the rest, the animals were too far away for me to really see. I was unable to keep changing the frame rate of the vOICe since I wanted detail and changing the frame rate to a higher one would have led to my losing detail.

Tips for other users who take the vOICe or similar technology on a safari

  • Forget about taking still images. Record a video if you can. This is because animals do not wait for you. The jungle is theirs and they go at whatever speed they like. Your sighted companions may be too stunned or too busy to help you capture a still image.
  • Be careful of light levels. It can be dark in the jungle or the light can be muted thanks to the trees. It was when I went.
  • You may have to stand to see certain birds and animals so keep things portable.
  • Wear headphones that do not mask environmental noise. There is plenty of bird song to hear. It is fun trying to look in the direction of interesting bird calls and try and spot the birds.
  • Keep your nose sharp since there is much to smell.
  • Beware of looking at the inside of the vehicle you are in. This is a great temptation since if your neck is anything like mine, you will tire looking sideways. This is not a problem on the elephant safari since you are sitting sideways on the elephant’s back.
  • The usual tips about batteries apply.

Would I have still enjoyed the safaris without the vOICe?

The answer is “not as much.” I would not have been able to participate fully in the experience and would have been stuck with second hand information. Visitors are not allowed to leave the vehicle in Jim Corbett due to safety conserns so I would not have been able to touch the plants let alone see how the forest changed. Yes, some kind of echo location may have helpped but I am not good enough to detect changes in types of trees etc. so the ability to sense things remotely came in handy.

TapTapSee an example of out of context vision

by ·

In many circles, the holy grail of blind mobility aides has been a system that “tells” the user what is in front of her. We are close to achieving this thanks to an application called TapTapSee made by Net Ideas LLC. TapTapSee runs on the iPhone and I tested the application this afternoon while returning from the world book fair. You point the camera in whichever direction you want and double tap the relevant button on the screen to take a picture. The picture is sent to the TapTapSee server where the image processing and object recognition takes place. A text description is then returned to the phone where it is displayed.

TapTapSee correctly identified the following things.

  • The color of the car that was standing next to mine. I was photographing out of the right rare passenger window.
  • My car seat.
  • The name and type of my car when I photographed the rare of the car which included the name plate.

The more interesting part of this exercise was the misrecognitions.

  • A gray railing was identified as a gray gate.
  • A stand of trees was identified as a parking lot.
  • A number of cars parked along the side of the road was also identified as a parking lot.
  • The underside of my car’s roof was identified as a white wall.

These misrecognitions demonstrate the importance of context. Yes, my car’s roof is a white fabric with a metal top but taken in isolation, it could be a wall. The same applies to the cars parked at the side of the road. Therefore, our next step has to be context aware object recognition. GPS may provide some limited help but I don’t know how my phone is supposed to know if I am in a car, standing on the road or in a shop.

This emphasizes the relevance of raw sensory input so that we can integrate said input with our knowledge of context and come to a valid conclusion about our surroundings. Systems such as the the vOICe as well as the good old white cane and the many varieties of sonar based devices already provide suitable sensory extensions. As in many such cases, it is up to us to construct our own technology mix.

Seeking a use case for vision

by ·

A frequent question I have been asked about the vOICe is “what can you do with it”? This is a good question. Many blind people are well adapted so can readily do most things sighted people do so this question can also be considered as finding a use case for vision. As of this writing, we cannot drive and depth detection is a challenge until we get affordable off-the-shelf video glasses with stereo cameras. I am compiling a list of tasks that need sight to do. Ideally, there should not be any other way to accomplish said task.
I asked the question on the seeing with sound mailing list and here are some of the tasks that members have suggested. Please add more to the comments section.

  • Painting and checking that the surface is fully covered without
    having to touch and thereby ruin the wet paint.
  • Street cleaning, “seeing” thrown-away garbage from a distance such
    that you can go an pick it up.
  • Cleaning up the garbage left behind in other indoor and outdoor
    places such as cinemas, conference centers, markets, etc.
  • Mowing the lawn (as a paid job), tracking the edges of parts that
    have already been mown. This can be tricky because of low contrast
    between mown and non-mown areas. Little bright flowers such as
    daisies can sometimes make the task much easier.
  • Farming, seeing non-uniformities in crop growth from a distance,
    or locating (silently standing) cattle from a distance.
  • Shorelining the pavement and familiar environments without touch.
    This may also be helpful in carrying goods around by having both
    hands free for carrying.
  • Signing your name
  • Locating objects that you have dropped or locating objects by color

It is not your vision stupid: updated thoughts on shape recognition

by ·

I’ve often bemoaned the fact that I’m not sufficiently experienced with vision to facilitate shape recognition. Over the past year or so, I have had access to a tiger embosser from View Plus Technologies which has allowed me to print images and feel them. Surprise, surprise! I was unable to identify what those images were without context. The same is the case when I look at images using the vOICe. The vOICe renders images in two dimensions just like what the tiger does. The resolution using the vOICe is higher and since I’m wearing a head mounted setup, I can use the vOICe to explore my environment immersively. There seem to be two conditions that facilitate shape and scene recognition. The first is historical experience so, if I have experienced a shape before, I’m able to identify what it is when I look at it again. Secondly, the scene has to match my mental map that I have built either using my kinesthetic awareness or some other means. This condition I experienced this condition firsthand when I was looking at a doorway using the fisheye lens with the video glasses. I was uncomfortable when I had the narrow angle view of the doorway. However, the moment I got the fisheye lens on the glasses, I was able to perceive both sides of the doorway which I knew would be there.

I am not too sure what this means for sensory substitution and learning how to use it effectively. In all probability, what this means is that experience and mental maps need to be stressed. The modality of information delivery is not as relevant as I thought it to be.

Ripples

by ·

This photograph has been taken close to sunset. The idea is to catch the ripples on the water. I was above on a jetty. This is a classic illustration of being able to capture something using vision and not touch. I could have touched the water but I would have got whet and I could not have recorded my experience in a form that could have been easily conveyed.

Negative video may help in detecting the ripples better. They are quite small in the soundscape. Watch for the higher pitched entries.

A series of ripples
ripples

Ripples

Another Select City Walk experience

by ·

I took the vOICe out to the shopping mall (The Select City Walk) once again. This time, I made two changes to the configuration of my netbook. I set the power scheme to maximum performance and I used a program switch speed for windows xp to set it to maximize the performance of the netbook’s CPU. I made these changes after a discussion with Peter as a test. What was happening was that the vOICe would stop giving me input. It almost seemed to hang. Mind you, my netbook is an Asus eee pc with 512MB of ram running Windows XP pro sp2. The problem remains as in there are times when the vOICe stops giving me any output or, sometimes when it is giving me output and I try to access it’s menus, I cannot do so. Jaws v9 works ok but the vOICe seems to slow down.

However, the maximizing of the CPU’s usage helped though the battery was exhausted after about 2 hours.

1. I found it easier to distinguish doorways.

2. The tiles on the wall were clearly visible.

3. I could tell when we were looking at columns and pillars.

4. I had an aha moment when I saw a rectangular speaker. The rectangle was like the one in exercise mode. The same applies to circular speakers that were mounted on shelves.

5. I had a look at a lot of curios and other items including toy cars. The toy cars soundscapes reminded nme of what I used to see when using the vOICe on the road. This was logical.

6. We went to a restaurant and there was a buffet lunch. I tried looking at different food items but they all seemed to look the same.

7. I found a strange glass that had a short stem and a large bowl like structure where the water was filled. Think of it like a vine glass with a short stem and a very large glass. I sensed it as a flattish circle more like a bowl so I was unable to find it visually until my dad explained the shape of the glass to me.

8. looking at people was interesting. They often blocked the light and are quite squishy.

Pranav

First run with my new MINI DV D008 USB camera sunglasses

by ·

I finally got the glasses. Remember, if you ever want to ship to India, please use a couriering service such as UPS or if you need a less expensive option registered airmail.

My first experience was positive and I could immediately feel the clarity of the image. Yet, the soundscapes did not quite give me the expected shapes. Still, the camera worked and it was now time to put the epoxy or, in my case super glue. Before I did that though the glasses stopped working. I could see a LED glowing and this was detectable using my light probe. I tried plugging and unplugging the glasses from my computer and tried them on my netbook. No go. I then remembered Peter’s point about resetting the glasses. Finding a suitable pin and that little hole with the clickable reset switch was an adventure in itself. I reset the glasses and all was well again. I now have the following setup.

1. MINI DV D008 USB camera sunglasses,

2. An Asus EEE pc. This is one of the earlier models that have come with Windows XP. See the following link for a review.
http://reviews.cnet.com/laptops/asus-eee-pc-4g/1707-3121_7-32466960.html

3. A Plantronics DSP400 headset.

I decided to try something simple for a change. Taking my key from some of Jacob’s posts I put some bottles and glasses on my dining table and tried moving them around. The bottles were taller than the glasses. I could sense the change in height but I had to bend my neck to really see the glasses. I tried positioning the bottles and glasses in different sequences. Something was still wrong since the soundscapes were not sounding the way I imagined them to sound. I then remembered that in my previous setup. I had had to tilt the view by 180 degrees to cater to the way the webcam was mounted. I now set the view rotation to 0 and all was well. I had hit shift+f1 to restore the vOICe to it’s best default settings but that had not changed the rotation. That is a very good design choice since you do not want the rotation reset every time you get the vOICe back to default settings. I repeated my old experiments and found it much easier to make sense of what was a bottle and what was a glass. Along the way, I discovered that I could increase the camera resolution to 600 by something. That was quite a lot of input and got uncomfortable fast. I now have the resolution set to a lower value which is 352 by 288 pixels.

I then moved to my driveway. My gate was clearly visible and I tried to experience perspective. This is going to take more work but things are improving. My garden was interesting. I seem to have some short plants or perhaps grass where I looked. Now, I have my Honda City parked on the left. As you go towards the gate, there is a small space between the side of the car and the wall. You need to walk through this space to reach the gate. So, I tried the experiment of finding that space using only vision and then walking without touching either the wall or the car. I then tried the experiment in reverse so I turned around such that the gate was behind me and tried going towards my house. The experiment was successful but I will need to start trusting my vision. I instinctively reached out to either side to ensure that I was on the right track. Before you ask, the blocking of all other sensory input was a conscious thing. I used no ear muffs or anything it was a matter of telling my brain to rely only on what the vOICe told it.

Next came the grand experiment. My dad asked me to get him a glass of water. He held the glass out towards me. I had seen a glass before so I got the side right but the height wrong. I then carried the filled glass of water back to him. I had toyed with the idea of looking at the water as I was pouring it but gave it up since bending down while tracking the positions of my hands and coordinating the glass and the bottle was too challenging. I filled the glass without mishap and took it back to my father. He held his fingers out for it and this time I got everything correct. However, judging height still needs more work.

A note about the startup sequence of the glasses The netbook that I have is very old and windows XP is not even at service pack 3. So, when I start the netbook, I must hit the power button of the glasses to switch them into camera mode. I can then launch the vOICe. If I do not do this, then I can keep hitting the power button and the glasses will sit there behaving like a drive. I have to reboot the netbook and switch to camera mode before doing anything else.