I often wear the the vOICe running on my vision 800 glasses to my office. Some of you may find this surprising. I have good blindness skills and the office is a known environment. Moreover, it is visually the same that is, not much changes at least in my office.
So why then bother with synthetic vision? Why not use something like a cane?
I can certainly use a cane and these days, this has become even easier with the smart cane made by IIT Delhi in collaboration with a number of partners. However, sonar and the cane do not give me sufficient information.
My office is on the second floor of a building. I usually take the stairs for two reasons. I like the exercise and the elevators do not talk. When I am climbing the stairs, it is very easy for me to read the floor indicator so I do not cross the second floor and end up at the cafeteria. It is also easier to walk towards doors because I can see them from a distance so can zero on them and reduce wiering. In the event that I do not see the floor indicator, I am able to see the walls. The office has windows on the landings and they are visually distinct as compared to the doors.
I accidentally discovered that the the vOICe is very handy during presentations because I can “look” at the screen of my laptop and read the text because said text is enlarged and the OCR in the vOICe picks it up without problems. This is very handy when evaluating products that contain dashboards etc.
I continue to refine my skills at detecting people, chairs and other obstacles. One experiment I plan to do is to read the names of meeting rooms.
I currently compliment synthetic vision with my blindness skills and may add other technology into the mix.
Accessibility
getting the orca screen reader working with a raspberry pi
Many of us want to run the graphical orca screen reader on a raspberry pi, the credit card sized computer for the ultimate in portable computing experiences. This has been done before. I got this working last night and by popular demand, here are my instructions. Be warned, I went down a few rabbit and may be installing a few extra packages but at the end, orca is working.
My thanks to Michaele A. Ray of the website raspberry VI website for doing this first and showing that this is indeed possible and to the members of the raspberry-vi mailing list for help and urging me to write this post.
Prerequisits
You will need to have the following items.
- A usb keyboard plugged into the pi for use on the pi console.
- A pair of headphones or speakers connected to the headphone jack.
- The raspberry pi should be powered and connected to the internet. A wired connection is the best as it is stable and easy to troubleshoot.
-
Download the full raspbian image
Download the latest image of raspbian from the below link. Get the full image as opposed to raspbian light. This is because the image has a lot of the graphical user interface things installed.
download raspbian -
Getting the image on to the sd card
Burn it on to a sd card using your program of choice. If you are using Microsoft windows, one of the most accessible programs is balenaEtchar
-
Enable ssh at boot
Once the image has been written, you will see 2 logical partitions on the sd card. There is a partition called “boot”. Create a blank file named “ssh” without the quotes on that partition. The way I do this on Microsoft windows is to go to a comma dprompt, switch to the drive that has the partition and then issue the following command.
copy con ssh
You will be in a line editor. Hit ctrl+z to save and close out of the file. -
Eject the card and load it into your pi.
-
Boot the raspberry pi and login to it over ssh.
- Update the raspberry pi
This step is crucial. Failure to do this will result in orca not speaking. Run the following commands.
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get dist-upgrade -y
Once the above commands have completed successfully, reboot the raspberry pi. -
Configure the pi to login to a text console
run the following command.
sudo raspi-config
Read the screen output very carefully. You need to go into the boot options and enable the option to login to the console without a password. - Once you have enabled the login, and are back on the main raspi-config screen, hit shift+tab once to save your settings and exit. Reboot.
-
IInstall dependencies
This is the trickiest part of the entire exercise.
Orca needs a bunch of things to run. As of this writing, speech dispatcher and espeak are not installed by default.
sudo apt-get install sox -y
Install the sox package for multimedia libraries. Some of them may be needed by speech dispatcher.
sudo apt-get install speech-dispatcher -y
Install the speech dispatcher service. Orca needs to talk to speech synthesizer. Be warned, this is an older version of speech dispatcher. There is a new one available on github but I have not tried compiling it from source on this installation.
sudo apt-get install espeak -y
The espeak speech synthesizer
sudo apt-get install gnome-orca -y
The orca screen reader and associated dependencies
Once you have carried out the above steps, reboot the raspberry pi. The text console login comes in handy here. Ensure that you have a USB keyboard plugged in. You may want to ssh into the pi to ensure that the pi has booted. If you have configured the raspberry pi to automatically login into a text console, you can enter the following at the command prompt. Be warned, at this point, you do not have any speech therefore you should know your keyboard well.
startx
You will have to wait for at least 2 minutes for the gui to load. I am giving a safety margin. Once again, you do not have speech at this point. Press alt+f2. This will place you at a prompt where you can enter a command to execute.
Type the following
orca
You will need to wait yet again but you should hear orca talking.
There is much left to do. I am still working on how to start orca automatically once the gui loads and need to find a good way for this to coexist with a console screen reader such as fenrir. I will update this post as I get feedback and learn more.
Shopping for footwear using synthetic vision
Sacheta wanted to buy footwear, so there we were, in Kala Niketan, Janpath on a Saturday evening looking for bellies and such. As usual, I was wearing my Vision 800 glasses running the vOICe. One advantage in footwear shopping is that the customer needs to sit to try it. The shop was not too crowded and we did not have a problem in finding a seat. However, before sitting, we looked around the shop, asking the staff for what she wanted. There were shelves packed with footwear. I was unable to understand the shapes until I touched a slipper. It had been stacked on its end such that the strap that comes on the top of your foot was facing the customer. This is different from how shoes are stacked because shoes are kept with their backs to the customer or at least that was the way they were kept the last time I checked.
The shapes were uniform from a far and the shelf boundaries formed hard edges in the scene. The shop had a large range of footwear in several colors. Once I was seated, I began to play with the color filter. Blue, green and red did not yield much feedback. I hit pay dirt when I chose orange. This was strange. I could hear shoppers asking for a colors like “rose gold”, whatever that is. Where was this color? I don’t think anyone wears orange shoes but then what do I know? The analyze option came in really handy. I sat back letting the scan tell me what filters were working. I then enabled the live OCR which introduced a new wrinkle. As I panned, I heard “fresh stock.” Hmm, what could they mean? Were they referring to the latest fashion? Was someone baking shoes? I could not ask Sacheta because she was engaged with a persistent sales person.
Once she had completed her shopping I was able to clear up some of the mysteries. The orange filter worked probably because the shop used a lot of yellow light. The “fresh stock” meant that there was no sale on those items. Don’t ask me why they could not say this up front.
I did have a chance to look at shoes and to distinguish details, my palm remains the best tool. However, with some more practice, I think I will be able to reliably distinguish between shelves full of slippers and shelves where shoes are stacked.
I did not buy anything so did not get a chance to try walking but I can see myself creating a visual landmark of a shelf or something else and walking independently to test shoes.
Some may ask, what is the point of it all? The simple way to buy footwear is to ask for what you want and try until you like something. Yes, this approach works well but my scene reader helped me to get a better idea of what was in the shop and thereby make a more informed choice.
Note
Photography is not allowed in the shop so I did not take any images.
Note 2
Be careful when looking and searching for a place to sit. These shops are full of humans sitting and not paying attention to their surroundings. Stay in the middle of the isle as much as you can. Let the staff guide you to a seat.
Access to travel the Planet Abled way
Planet Abled invited me to speak at the Access To Travel conference conducted by them on 27 September 2017. I went in expecting the usual conference but was pleasantly surprised. It was conducted at Park Hotel. The conference experience began just after the metal detectors. I was barely through the device when I was met by a Planet Abled team member and escorted to the venue. I checked as I entered the hall. Something was wrong. The light level dropped, and I was in an enclosed space. I am light independent therefore I continued through and entered the auditorium.It turns out that the entrance had been reconfigured as a darkness simulator.
Cameras and Planet Abled staff were everywhere. Unlike other conferences, they did not ask me to settle down. There was active focus on moving around and interaction. The staff helped. We, the tourists had a chance to meet other key leaders in the travel industry such as Mr. Subash Goyal. The food was good and was dry. I have attended more than my share of conferences where the food has gravy and is impossible to eat without 2 working hands.
it is rare to have a discussion on recreation in India. The access to travel conference was one of the few events that addressed not only the challenges when traveling with a disability but allowed me to see what everyone else was doing to have fun.
The most enjoyable part for me in the conference was the emphasis on stories. Each speaker had his own story to tell and had time to tell it in. It was also very easy to ask speakers questions and to meet those who stayed after the conference. As always, the speakers were cross disabilities. I returned home with a greater sense of unity. Our senses that we used to engage with the world were different, but the problems were the same and had the same broad solution namely making people better humans and treating each other with dignity and respect.
Artificial vision in the enterprise
I recently acquired the Vision 800 smart glasses. This gave me a compact and convenient setup allowing me to run the the vOICe. I have had several visual experiences. When I wear the glasses, I am effectively wearing an Android tablet on my head. Yes, it would be nice to do multiple things but given the specifications of the glasses, I use them as a dedicated vision device. I also use bone conduction headphones.
- I am able to read floor numbers as well as other signage. This means that when the fire martial asks me to exit from gate 3B, I know what he is talking about. In addition, I can navigate the stairs independently and do not need to count floors.
- I can lean forward and see if my laptop has an error message. It is easier talking to the help desk if I can tell the problem and many times, I can solve the problem independently.
- I am better at indoor navigation. I am able to tell when silent humans are in the way.
- The camera on the vision 800 glasses is on the extreme left. I am not used to scanning therefore the narrow field of view and the left orientation is not matching the sense of space my body has. This is taking getting used to.
- I am also still working out the right time to look down.
- I can derive more information about my environment such as detecting flower pots that have been placed on top of filing cabinets.
- Bone conduction headphones are a double edged sword. Yes, I can hear environmental sounds but on the other hand, in situations like lunch time at the office Cafeteria, they are almost useless. I cannot hear the soundscapes unless I increase the volume significant in the vOICe.
- I have run the glasses for over 8 hours. They do not heat up much.
- I can better handle situations where colleagues leave things on different parts of my table. For example, a colleague heated my lunch and his. He kept my lunch but forgot to tell me that he had done this. I scanned the table and was able to get hold of my plate.
Post processing images including describing them automatically
As most of you know, I publish plenty of images on this blog. I ensure that all of them are described. The biggest challenge I have in posting photographs on this blog is captioning them. I have to get images described manually before I put them up here. Once I take my photographs I isolate my pictures by location because of the geotagging my phone does. I then send them to people who have been on the trip who describe the images. I have been searching for solutions that describe images automatically. I was thrilled to learn that wordpress had a plugin that used the Microsoft Cognitive Services API to automatically describe images. The describer plugin however did not give me location information therefore I rolled my own code in python. I have created a utility that queries Google for location and the Microsoft Cognitive Services API for image descriptions and writes them to a text file. I had tried to embed the descriptions in EXIF tags but that did not work and I cannot tell why.
References
You will need an API key from the below link.
Microsoft Cognative Services API
The wordpress plugin that uses the Microsoft Cognitive Services API to automatically describe images when uploading
Notes
- You will need to keep your cognitive services API key alive by describing images at least once in every 90 days I think.
- Do account for Google’s usage limits for the geotagging API.
- In the code, do adjust where the image files you want described live as well where you want the log file to be stored.
- Do ensure you add your API key before you run the code.
import glob from PIL import Image from PIL.ExifTags import TAGS from PIL.ExifTags import TAGS, GPSTAGS import piexif import requests import json import geocoder def _get_if_exist(data, key): if key in data: return data[key] return None def get_exif_data(fn): """Returns a dictionary from the exif data of an PIL Image item. Also converts the GPS Tags""" image = Image.open(fn) exif_data = {} info = image._getexif() if info: for tag, value in info.items(): decoded = TAGS.get(tag, tag) if decoded == "GPSInfo": gps_data = {} for t in value: sub_decoded = GPSTAGS.get(t, t) gps_data[sub_decoded] = value[t] exif_data[decoded] = gps_data else: exif_data[decoded] = value return exif_data def _convert_to_degrees(value): """Helper function to convert the GPS coordinates stored in the EXIF to degrees in float format""" d0 = value[0][0] d1 = value[0][1] d = float(d0) / float(d1) m0 = value[1][0] m1 = value[1][1] m = float(m0) / float(m1) s0 = value[2][0] s1 = value[2][1] s = float(s0) / float(s1) return d + (m / 60.0) + (s / 3600.0) def get_lat_lon(exif_data): """Returns the latitude and longitude, if available, from the provided exif_data (obtained through get_exif_data above)""" lat = None lon = None if "GPSInfo" in exif_data: gps_info = exif_data["GPSInfo"] gps_latitude = _get_if_exist(gps_info, "GPSLatitude") gps_latitude_ref = _get_if_exist(gps_info, 'GPSLatitudeRef') gps_longitude = _get_if_exist(gps_info, 'GPSLongitude') gps_longitude_ref = _get_if_exist(gps_info, 'GPSLongitudeRef') if gps_latitude and gps_latitude_ref and gps_longitude and gps_longitude_ref: lat = _convert_to_degrees(gps_latitude) if gps_latitude_ref != "N": lat = 0 - lat lon = _convert_to_degrees(gps_longitude) if gps_longitude_ref != "E": lon = 0 - lon return lat, lon def getPlaceName(fn): lli=() lli=get_lat_lon(get_exif_data(fn)) g = geocoder.google(lli, method='reverse') return g.address def getImageDescription(fn): payload = {'visualFeatures': 'Description'} files = {'file': open(fn, 'rb')} headers={} headers={ 'Ocp-Apim-Subscription-Key': 'myKey'} r = requests.post('https://api.projectoxford.ai/vision/v1.0/describe', params=payload,files=files,headers=headers) data = json.loads(r.text) dscr=data['description'] s=dscr['captions'] s1=s[0] return s1['text'] def tagFile(fn,ds): img = Image.open(fn) exif_dict = piexif.load(img.info["exif"]) exif_dict['Description''Comment']=ds exif_bytes = piexif.dump(exif_dict) piexif.insert(exif_bytes, fn) img.save(fn, exif=exif_bytes) def createLog(dl): with open('imageDescriberLog.txt','a+') as f: f.write(dl) f.write("\n") path = "\*.jpg" for fname in glob.glob(path): print("processing:"+fname) createLog("processing:"+fname) try: imageLocation=getPlaceName(fname) except: createLog("error in getting location name for file: "+fname) pass try: imageDescription=getImageDescription(fname) except: createLog("error in getting description of file: "+fname) pass imgString="Description: "+imageDescription+"\n"+"location: "+imageLocation createLog(imgString) try: tagFile(fname,imgString) except: createLog("error in writing exif tag to file: "+fname) pass
Getting fenrir talking on a raspberry pi
Fenrir is a promising new user mode screen reader. It is primarily written in python 3. Here are my instructions to install it on a raspberry pi.
I am assuming that you are using RASPBIAN JESSIE LITE and are at a terminal prompt.
Updating your installation
You must update your installation of raspbian otherwise components like espeak will not install correctly.
sudo apt-get update
Let the update finish. The above command just fetches the listing of packages that need to be updated. Now do the actual upgrade.
sudo apt-get upgrade -y
This is going to take a while to complete.
You now have to install several dependencies for fenrir to work.
Espeak
This is the speech synthesizer fenrir will use.
sudo apt-get install libespeak-dev
Python 3
sudo apt-get install python3-pip -y
Fenrir is written in python 3.
The python daemon package
sudo apt-get install python-daemon -y
A package that allows fenrir to run as a service.
evdev
sudo pip3 install -evdev
A python package that handles keyboard input.
The speech-dispatcher package
sudo apt-get install speech-dispatcher -y
The above package is required to get fenrir to talk to a speech synthesizer such as espeak.
The ConfigParser package
sudo pip3 install configparser
You may need this dependency to parse the fenrir configuration file.
A module for checking spelling
sudo apt-get install enchant -y
sudo pip3 install pyenchant
An optional package to handle spell checking.
Git
sudo apt-get install git -y
Git is a version control system which you will use to pull down the latest source code of fenrir.
It is now time to install fenrir. Execute the following command.
git clone https://github.com/chrys87/fenrir.git
You now need to start configuring the pi for fenrir to work.
Execute the following command.
sudo spd-conf
You will be asked a series of questions. The main thing you need to change is the kind of sound output technology to use. You need to use alsa if you are using the 3.5MM headphone jack of the pi like I am doing. When you are asked about using pulseaudio type “alsa” without the quotes. Hit enter to move to the next prompt after each question. Do not forget to adjust the speed and pitch to your liking.
You now need to test your speech synthesizer configuration. Do ensure that you have your headphones or speakers ready.
sudo spd-say testing
If you hear the word “testing” you are good to start fenrir. If not, look through your logs and seek support.
To start fenrir, execute the following command.
Warning: You will need to change terminals once you execute the below command so you may want to open a new terminal and keep it handy.
Assuming you are in your home directory type the following commands.
cd fenrir/src/fenrir
sudo python3 ./fenrir
You should hear a stack of startup messages which signifies that fenrir is running.
Usage
I am still playing with fenrir therefore cannot comment much. The key bindings appear to be similar to those of the speek up screen reader. If you want to check them out, take a look at the config file in your installation directory or at the following link.
There is a tutorial mode available accessed with the fenrir key and h. The fenrir key is the insert key by default.
Acknowledgements
Thanks to the following people for helping me get fenrir talking and for raspberry pi advice.
Michael A. Ray
Storm Dragon
Other members on the IRC channel called #a11y at the Server: irc.netwirc.tk; specifically:
PrinceKyle, Jeremiah, chrys and southernprince.