Skip links

techesoterica.com

A blog dealing with sensory substitution and other esoteric concepts and technologies like speech-recognition and chaos theory

Main navigation

vOICe

Vision for under $300 or Getting the vOICe working on a raspberry pi 4

by ·

It is now possible to get the web version of the vOICe working on a raspberry pi 4. This means that it is possible to regain vision like abilities for roughly $200 or less.

What you need

In terms of hardware, you need the following things.

  • The raspberry pi 4.
  • A camera connected via USB or the pi camera.
  • All the other prerequisits mentioned in my post on Getting orca to run on the raspberry pi
  • A pair of wired headphones connected to the pi.

Checking the camera

Connect the camera to the pi and ensure it is working. If using a USB webcam, use the following command.

lsusb

The above command lists all the hardware that has been detected by the way. Your camera should be in this list. If not, you will need to use your favorite search engine or to a forum of your choice to troubleshoot.

Getting the software ready

The vOICe needs the graphical user interface to be working on the pi. As of this writing, I have not found a way to run it from the console. Please follow all the steps that I have listed in my post on getting the orca screen reader running on the raspberry pi 4.
Once you have chromium talking, please continue reading this post.
Getting the vOICe up
Do ensure that your camera is connected.

  1. Launch chromium
  2. Hit f6 if necessary and then alt+d to get to the address bar.
  3. Navigate to the web version of the vOICe
  4. You now need to tab around and allow chromium access to the camera. If you do not see this prompt, press f6 and then tab. This permission is crucial else the vOICe will be unable to access the camera.

You are done, you should hear the vOICe sounding. Move the camera around and hear the soundscapes change.

Going portable

One of the biggest strengths of this setup is that you can choose what modules you want to add. I cannot suggest what battery pack / power bank you go for. Do check the power specifications of the pi 4. You will also need to enclose the raspberry pi in a case. The case is up to you. I have used an off-the-shelf abs container for my case.

Why wired headphones

I have not tested Bluetooth headphones with my pi 4 therefore cannot comment how the soundscapes will sound. If you do test this, let me know and I will update this post.

Wireless sight

Wireless sight

by ·

I have often bemoaned the sheer number of wires I have to deal with when using the vOICe. There are 3 wires. They do not sound like much but get hopelessly tangled.

  • The wire going from the glasses to the computer.
  • The wire going from the headphones to the computer.
  • The wire going from the computer to a reserve battery or in some cases, to the main battery.

I have now found a solution which eliminates most of the wires. The solution utilizes a little known feature of the vOICe for Android. It can interface with IP cameras. I have mounted a portable IP camera on a pair of industrial safety glasses and have connected the camera to my Android phone wirelessly. A pair of Bluetooth headphones completes the setup.

Prerequisites

Assembling the solution

Hardware

It is crucial to order the USB charging cradle with the AI-Ball camera.

  1. Remove the charging circuit from the USB cradle of the AI-Ball and attach it to the battery pack. This is where the maker space comes in. You will need help for doing this unless you are comfortable with soldering irons and removing circuit boards.
  2. You will need to mount the camera on the safety glasses.
  3. The battery pack will go into the cloth pouch along with the battery charging circuit board.
  4. Mount the fish-eye lens in front of the AI-BALL camera.
  5. You need to strap the battery pack in its cloth pouch to the legs of the glasses.

Software

  1. Power up the ai-ball and configure it to be infrastructure mode.
  2. Connect your Android device and ensure that you can see the stream of the camera on the phone’s browser.
  3. Download the vOICe for Android on to your Android phone and ensure it is running.
  4. Add the AI-Ball’s streaming URL to the vOICe’s camera dialogue after setting the camera type to IP camera.
  5. If using Talkback, as of this writing, you may have to swap out of the vOICe, use the “over view” button to get back in and then click on “continue.”

Working

Once you have successfully added the URL, the soundscapes will change. The vOICe will begin to use the IP camera. Keep Foveal Mapping enabled because you are using a wide angle lens. You can then pair the Bluetooth headphones and go wireless.

Caveats

  • You do need to tinker to assemble this solution.
  • You need to handle 3 batteries namely the phone battery pack for the camera, the phone’s battery and the battery powering the Bluetooth headphones.
  • Some of the functions of the vOICe may not work with an IP camera because these cameras lack standard parameters so need to be catered for individually.
  • At the time of this writing, I have not found a way to force my Android phone to connect to my prefered network. I have a home wireless network and the network of the camera. I need to connect the phone to the camera each time I power up the solution.

In terms of the wires, a thin wire connects the camera to its USB cable. This wire runs along the right leg of the glasses. There is also a set of straps that take the weight of the battery pack which hangs behind the glasses. I usually let it hang inside my clothing.

Conclusion

The above solution can be used with any IP camera. You are free to substitute any of the components. You could even use different kinds of cameras such as thermal cameras assuming they support MJPEG streaming.

Are there easier options?

Yes. The vOICe is a flexible platform which you can run on a variety of devices. The other option is to contact Banaao and ask them to assemble the rig. This will be a bespoke project because much will depend upon several factors such as what infrastructure you already have, what you can source locally and what has to be imported.

Acknowledgements

  • Mr. Prem Sagar, Founder and CEO of Banaao>
  • Various trainees at Banaao include Davis Devasia, Gurjap Singh
  • Mikael Holmgren for introducing me to the AI-BALL camera
  • Dr. Peter Meijer for modifying the vOICe virtually on the fly to handle large frames which were output from IP cameras.

My wireless eye with a fish-eye lens in the center and the safety glasses
Pranav Lal wearing his wireless eye.

3D design for the mount in front of the glasses

Releasing the Immersive Vision Kit

by ·

I am releasing the Immersive Vision Kit which is a dedicated device for running the vOICe. If you can build Lego, then you can build yourself an electronic eye. The kit has the following advantages.

  • It is compact and unobtrusive.
  • It is modular allowing you to select the components you want.
  • You do not feel the heat generated by the camera or any other component.
  • It is easier to handle as compared to a Netbook or laptop with video glasses. The equipment can fit into a jacket pocket or purse.
  • You can add other components to the kit.
  • The camera used has a very wide angle allowing you to get the maximum input from your environment.
  • There are no distractions while using the kit. Its only purpose is to run the vOICe and that is what it does.

The kit consists of an ODROID u3 board running Android as well as a camera with a wide angle view. Please see the immersive vision kit page for more information including setup instructions..

Upper body view of the immersive vision kit

My new VISOR: a mini android phone

My new VISOR: a mini android phone

by ·

 

I have been looking for a way to make the setup of the vOICe significantly less cumbersome for quite some time. As background, I bought a netbook in 2010 and began using video glasses and headphones.  That setup had several disadvantages.

  1. I had to wear a back pack all the time I was using the technology.
  2. I had to juggle multiple wires which was a hassle because the wires would get tangled.
  3. Netbooks are still quite large and are difficult to carry.
  4. I was unable to use the setup effectively in the office since it is difficult carrying 2 laptops.
  5. Getting the setup ready took it leased 4 to 5 minutes from a cold start. I am one of those people who do not believe in using the standby and/or hibernate functions. If I was not using the netbook, and I did not have access to an electrical wall outlet, I would shut it down.

 

I had several discussions on the seeingwithsound mailing list about simplifying the setup.  We could not come to any conclusions.  Devices such as Google glass are an option but as of now, very few people have access to them.  Moreover, they are significantly more expensive than anything else on the market.  My netbook setup cost me approximately $400.  The bulk of that cost was that of the netbook. The obvious thing to do is to shrink the netbook but there are very few machines that have a smaller form factor.  Some phones and tablets can use external web cams via the USB two go standard but this is not reliable and these phones are few and far between.  It was my father who suggested that I could head mount the phone.  Yes, we had considered doing this before but as I’ve said above, phones were too big.  Enter the mini android phone. My thanks to Dr Meijer for finding this.  This phone runs version 4.2.2 of Android and has a 2.5 inch screen.  I was able to head mount the phone using a sports headband and a custom pocket.  I attached a fisheye lens to the phone camera and connected the supplied headphones.  I now have an extremely compact setup which runs the vOICe.  I do not have a SIM card in the phone because I do not need to use its phone or data features.  The phone comes with a standard micro USB connector and, is water resistant.

 

Warning: I did need sighted assistance to install Talkback.  That is the only downside I can find with this phone.  Mind you, it is indeed possible to install Talkback independently by using the android debug Bridge but launching Talkback without rooting the phone is not possible as of this writing.

 

I am planning to invest in a Bluetooth headset so that I can go completely wireless.  However, I consider that optional.  In terms of cost, the entire solution is just above $100. Please see the below table for a cost breakdown.

Item Price
Mini Android phone 74.62
Fish-eye lens 19.99
Headband 8.32996

Total=$102.64

 

You may ask if there is anything about the netbook solution that I miss. Yes, there are 2 things.

  1. The camera orientation in the center that is, it is between my eyes. I am used to this alignment. In the phone solution, the camera is positioned just above my left eye.
  2. Hesternalized sound rendering.

 

Other benefits of the phone solution

  1. The face recognizer is handy because I can orient myself towards a person.
  2. The autofocus feature maintains the sharpness of edges when viewing objects from short distances.

PranavLalHeadBandMobilePhoneSetup

Towards a $100 eye

by ·

Introduction

One of the regular things that I post about on this blog is the vOICe. As of this writing, the vOICe is the only way that a blind person can experience vision. For the true visual experience, the user needs to use video glasses connected to a netbook running a flavour of Microsoft Windows and the vOICe learning edition. Yes, the vOICe has been ported to other platforms such as android and Symbian.

The problem with these platforms is that they involve the use of mobile phones. Mobile phones are indeed versatile gadgets but one of the things that they cannot do is give you a permanent view from head height. It is very difficult, almost impossible to mount a mobile phone such that its camera will be positioned at the tip of your nose or at the level of the eyes. The only way to accomplish a headmounted setup today is to use USB powered video glasses with wide-angle lenses.

I and many others want to eliminate this dependency on Microsoft Windows. This is not one of those windows bashing posts. We want to get the vOICe running on as many platforms as is possible. Towards this, I, with the help of many others in the community have created a port of the vOICe that runs on Linux. We’re trying to get it to run on the raspberry pie. However, though we have a working version, it is very slow and we need help in understanding where the problem is.

One of the questions that arises is why the raspberry pie when there are better platforms available? However, the idea is to use off-the-shelf equipment and ensure that the platform to which the vOICe is ported is sustainable and is easy to acquire for new users. Today, the raspberry pie fits this bill nicely. Moreover, once the vOICe has been ported to Linux, it should be possible for any interested developers to get the source code from the seeing with sound website. A basic version of the source code already exists on that site.

Another reason we are interested in porting the vOICe to the raspberry pie and devices running operating systems other than Windows is the cost. Windows licensing is still more expensive as compared to that of other operating systems. The idea is to have an artificial eye for or below one hundred dollars. The raspberry pie is about seventy dollars. The video glasses are approximately 30 dollars. So, we’re almost there.

Areas where we need help

  • improving the performance of the vOICe.
  • Adding features to that program such as four-wheel mapping, speed control, the colour filter and if possible obstacle detection.
  • getting the drivers for the video glasses (they use the syntek chip set) to work on Linux

If you are interested in participating, either contribute to this thread on the raspberry pie forum or feel free to get in touch.

Threads where the vOICe has been discussed

Seeking help with converting opencv webcam output to pixel coordinates for integrating with sensory substitution program
Sight for the Blind for <100$

Links of interest

Drivers and script to install them for Linux
href=”http://www.seeingwithsound.com/hificode_OpenCV.cpp”>The vOICe for Linux and similar operating systems

Finally, I have uploaded some source code with this post which is the working version of the vOICe for nix like operating systems.

Analyzing Mount Etna Boiling Over

by ·

My thanks to NASA for adding color to a picture of Mount Etna Boiling Over. It allows one to distinguish different parts of the photograph.
http://www.npr.org/blogs/thetwo-way/2013/02/21/172619499/pretty-picture-moun

See the above link for the complete story. If you want to analyze the image in the vOICe, then you will need to download the image. Given that the story tells you what the differently colored areas are, you can use the color filters to interpret the image.

Note:
The story says that snow is blue green. The vOICe can only apply one color filter at one time. However, Thanks to Dr. Meijer we learn that “Blue-green is cyan” so we can use alt+shift+c to see the snow.

It is not your vision stupid: updated thoughts on shape recognition

by ·

I’ve often bemoaned the fact that I’m not sufficiently experienced with vision to facilitate shape recognition. Over the past year or so, I have had access to a tiger embosser from View Plus Technologies which has allowed me to print images and feel them. Surprise, surprise! I was unable to identify what those images were without context. The same is the case when I look at images using the vOICe. The vOICe renders images in two dimensions just like what the tiger does. The resolution using the vOICe is higher and since I’m wearing a head mounted setup, I can use the vOICe to explore my environment immersively. There seem to be two conditions that facilitate shape and scene recognition. The first is historical experience so, if I have experienced a shape before, I’m able to identify what it is when I look at it again. Secondly, the scene has to match my mental map that I have built either using my kinesthetic awareness or some other means. This condition I experienced this condition firsthand when I was looking at a doorway using the fisheye lens with the video glasses. I was uncomfortable when I had the narrow angle view of the doorway. However, the moment I got the fisheye lens on the glasses, I was able to perceive both sides of the doorway which I knew would be there.

I am not too sure what this means for sensory substitution and learning how to use it effectively. In all probability, what this means is that experience and mental maps need to be stressed. The modality of information delivery is not as relevant as I thought it to be.