Planet Abled invited me to speak at the Access To Travel conference conducted by them on 27 September 2017. I went in expecting the usual conference but was pleasantly surprised. It was conducted at Park Hotel. The conference experience began just after the metal detectors. I was barely through the device when I was met by a Planet Abled team member and escorted to the venue. I checked as I entered the hall. Something was wrong. The light level dropped, and I was in an enclosed space. I am light independent therefore I continued through and entered the auditorium.It turns out that the entrance had been reconfigured as a darkness simulator.
Cameras and Planet Abled staff were everywhere. Unlike other conferences, they did not ask me to settle down. There was active focus on moving around and interaction. The staff helped. We, the tourists had a chance to meet other key leaders in the travel industry such as Mr. Subash Goyal. The food was good and was dry. I have attended more than my share of conferences where the food has gravy and is impossible to eat without 2 working hands.
it is rare to have a discussion on recreation in India. The access to travel conference was one of the few events that addressed not only the challenges when traveling with a disability but allowed me to see what everyone else was doing to have fun.
The most enjoyable part for me in the conference was the emphasis on stories. Each speaker had his own story to tell and had time to tell it in. It was also very easy to ask speakers questions and to meet those who stayed after the conference. As always, the speakers were cross disabilities. I returned home with a greater sense of unity. Our senses that we used to engage with the world were different, but the problems were the same and had the same broad solution namely making people better humans and treating each other with dignity and respect.
The keyboarding buz
I went for the pots and Pens writing retreat planning to write the great Indian novella. The retreat was conducted at the Shilaroo Project. I did write but the retreat turned into a writing research trip which according to Aditi is still writing. I experienced farming, the forest, plenty of photo opportunities, superb people, pottery and some challenging navigation situations. I was running the vOICe on my vision 800 smart glasses and used them extensively to get a feel of the place.
The navigation
The house at The Shilaroo Project is situated upon multiple levels thanks to it being on a hill. There were many staircases. The staircases inside the house were easy to find and navigate due to the use of railings. The risers were large and visually distinct. However, things were different outside. Yes, some of the main staircases did have railings but the steps were not uniform, and vision was not the best tool to use in this situation especially when descending. I had to bend my neck and really study the staircase, for which I did not have patience. However, I had human guides with me, so I did not have a significant problem. The farm is full of interesting places such as the blue bridge over the pond. It is terrific for photography due to the contrast of light and shade. There is a variety of plants including some large apple trees. The multiple levels also make for great photography angles.
Light independent pottery
One of the attractions of pottery is its physical nature. You hold the clay in your hands to work with it. Vision does not play a significant role at the initial stage. You need to determine the right amount of water to put, the texture of the clay etc., by feel. Yes, looking at delicate clay pots is a visual activity. I experimented with a manual potter’s wheel and a variety of carving tools.
a
Clay is an excellent material for teaching blind people geometry. I could carve straight lines and other shapes into clay. The best thing is that erasing my drawing is as simple as rolling the clay into a ball and mushing it. You can then use your palms to press the clay into a new shape. The manual wheel is a lot of fun. You sit on a stool like contraption and then use one leg to push the bottom plate of the wheel. There is a ball bearing that will rotate the upper wheel. I had to be careful here. I could spin the wheel fast, but I had to first center the clay. In addition, I had to avoid touching the clay as I pushed the wheel else it would jerk and go off center. The other thing I found was that touching the spinning wheel was a mistake. I had to touch only the clay. Muscle memory was enough for this because I had placed the clay. I did splash a fair bit of clay around. None of it hit the camera. However, getting different shapes conceptually did take some knowledge of geometry and of 2d vs. 3d. For example, we were discussing the shaping of cup handles. Virangna suggested that they looked like human ears. I felt my ear and tried comparing it to a cup handle and failed. The way I understood this comparison was to think about drawing the handle and my ear on a sheet of paper and converting from 3d to 2d.
Solving the mystery of terrace farming
The high-point for me was when Virangna demonstrated how terraces are cut for farming. It was one of those things that was never fully explained in geography class and I was unable to imagine what the process would be. Yes, the mountain side would be cut but then where would the cutting machinery stand? She and Arun Cherian, the architect of the Shilaroo project created a model of a mountain in clay. On the side of the mountain facing me, they then cut a trench like path to the base of the mountain. Finally, they took a trimming tool and cut away the side of the mountain by widening the trench. That explained the cutting of the mountain. The mountain had to be re-enforced by adding metal plates and building walls usually out of Roller-compacted concrete. They then took small cubes of clay and added them to the mountain creating houses.
I go unplugged with yoga
The yoga studio is a large hall with about 22 electrical wall outlets. The irony is that I did not need to use them. The space has large glass windows that let in the sun. Wearable technology is almost forbidden in class. The idea is to listen to your body. It may be possible to watch a teacher do the steps of each exercise because Upa yoga is done very slowly. However, verbal instructions and quick corrections with some extra time worked. The highlight here was that Shubhangna, the yoga teacher was willing to have a conversation and to respect contrary points of view. Moreover, she explained the reason behind each exercise and could change it where necessary based on my needs.
The forest
The forest walk conducted on top of the Hatu peak was advertised as one of the highlights of the trip and it more than lived up to that expectation. The hike was easy. I took plenty of pictures. This is where I ran into a challenge in path finding. I had limited time therefore could not experiment extensively. Detecting obstacles on the ground is a challenge due to low light levels because of the litter of dead leaves on the ground.
I had to identify tree roots and other objects. I also tried some quick scanning through the trees to find a path but did not get very far. I need to test a wide-angle view in this situation. There were also safety concerns. There were sudden drop offs and one slip meant a fall of about 200 feet.
We had a picnic lunch and then Aditi asked us to use synesthesia to write. I had live synesthesia sounding in my ears which made for some creative work. However, this is where the bees interfered.
We were seated in a meadow like area surrounded by trees and flowers. The highway was a distant memory and there were almost no other people around. You would think that would be the perfect environment for writing. Perhaps yes, for some of you but for me, it was not. I have read in literature of the comforting droning of bees but with respect, the author of those words does not know what he is talking about. I could hear the high-pitched buzzing noise as they flew. The noise would drop in pitch as they came near me and then they would accelerate away. There were indeed flowers where we were sitting. I was using a Logitech K480 keyboard with my iPhone SE which worked like a charm. I wrote 3 paragraphs and then realized that my temptation to use the keyboard as a bee swat was too strong. I knew this was a bad idea. The bees were ignoring me, and the rest of the group did not appear to have a problem. We read our pieces and hiked back with me taking a few more photos.
I see the moon
We had a camp fire night.
The sky was clear with the stars and moon shining. I tilted my head up to see the stars. No go. I did not see any light. However, the moon was different. Thanks to Virangna and Arun for the apple pie walk and helping me find the moon. The farm had tiny lights to facilitate night time navigation. We found a clear area and then they pointed my head in the right direction to see that pinpoint of light. It was a spec in the camera view, but I did see the moon. As a child, I was told that the moon was in the sky. The sky was above me, so I expected to see the moon when I turned my head to the sky. This is almost correct. I had to look at an angle to see the moon. I also had a look at the kitchen because we had to check the status of the apple pie.
Other views
Acknowledgements
Aditi Rao and my writing group for the fantastic support and terrific stories
Vidula Sonagra for logistics and image descriptions
Brig. Anuj Kainthla, Vsm (Retd.)- for clearing my concepts on stellar navigation and the use of blankets as well as for gadget charging heaven thanks to the plethora of electrical sockets
Anita Kainthla for her hospitality, food and desserts
Virangna and Shubhangna Kainthla for art, yoga and fascinating conversation
The staff at the Shilaroo project for back end support
Artificial vision in the enterprise
I recently acquired the Vision 800 smart glasses. This gave me a compact and convenient setup allowing me to run the the vOICe. I have had several visual experiences. When I wear the glasses, I am effectively wearing an Android tablet on my head. Yes, it would be nice to do multiple things but given the specifications of the glasses, I use them as a dedicated vision device. I also use bone conduction headphones.
- I am able to read floor numbers as well as other signage. This means that when the fire martial asks me to exit from gate 3B, I know what he is talking about. In addition, I can navigate the stairs independently and do not need to count floors.
- I can lean forward and see if my laptop has an error message. It is easier talking to the help desk if I can tell the problem and many times, I can solve the problem independently.
- I am better at indoor navigation. I am able to tell when silent humans are in the way.
- The camera on the vision 800 glasses is on the extreme left. I am not used to scanning therefore the narrow field of view and the left orientation is not matching the sense of space my body has. This is taking getting used to.
- I am also still working out the right time to look down.
- I can derive more information about my environment such as detecting flower pots that have been placed on top of filing cabinets.
- Bone conduction headphones are a double edged sword. Yes, I can hear environmental sounds but on the other hand, in situations like lunch time at the office Cafeteria, they are almost useless. I cannot hear the soundscapes unless I increase the volume significant in the vOICe.
- I have run the glasses for over 8 hours. They do not heat up much.
- I can better handle situations where colleagues leave things on different parts of my table. For example, a colleague heated my lunch and his. He kept my lunch but forgot to tell me that he had done this. I scanned the table and was able to get hold of my plate.
Wireless sight
I have often bemoaned the sheer number of wires I have to deal with when using the vOICe. There are 3 wires. They do not sound like much but get hopelessly tangled.
- The wire going from the glasses to the computer.
- The wire going from the headphones to the computer.
- The wire going from the computer to a reserve battery or in some cases, to the main battery.
I have now found a solution which eliminates most of the wires. The solution utilizes a little known feature of the vOICe for Android. It can interface with IP cameras. I have mounted a portable IP camera on a pair of industrial safety glasses and have connected the camera to my Android phone wirelessly. A pair of Bluetooth headphones completes the setup.
Prerequisites
- A maker space: Banaao in my case
- The AI-BALL camera
- The BLUEZ 2S from Aftershokz
- An Android phone. I am using the Motorola Moto G4 Play
- A fish-eye lens.
- A pair of glasses.
- A phone battery pack.
- A cloth pouch for holding the battery pack.
- A set of straps to hang the battery pack from the glasses.
Assembling the solution
Hardware
It is crucial to order the USB charging cradle with the AI-Ball camera.
- Remove the charging circuit from the USB cradle of the AI-Ball and attach it to the battery pack. This is where the maker space comes in. You will need help for doing this unless you are comfortable with soldering irons and removing circuit boards.
- You will need to mount the camera on the safety glasses.
- The battery pack will go into the cloth pouch along with the battery charging circuit board.
- Mount the fish-eye lens in front of the AI-BALL camera.
- You need to strap the battery pack in its cloth pouch to the legs of the glasses.
Software
- Power up the ai-ball and configure it to be infrastructure mode.
- Connect your Android device and ensure that you can see the stream of the camera on the phone’s browser.
- Download the vOICe for Android on to your Android phone and ensure it is running.
- Add the AI-Ball’s streaming URL to the vOICe’s camera dialogue after setting the camera type to IP camera.
- If using Talkback, as of this writing, you may have to swap out of the vOICe, use the “over view” button to get back in and then click on “continue.”
Working
Once you have successfully added the URL, the soundscapes will change. The vOICe will begin to use the IP camera. Keep Foveal Mapping enabled because you are using a wide angle lens. You can then pair the Bluetooth headphones and go wireless.
Caveats
- You do need to tinker to assemble this solution.
- You need to handle 3 batteries namely the phone battery pack for the camera, the phone’s battery and the battery powering the Bluetooth headphones.
- Some of the functions of the vOICe may not work with an IP camera because these cameras lack standard parameters so need to be catered for individually.
- At the time of this writing, I have not found a way to force my Android phone to connect to my prefered network. I have a home wireless network and the network of the camera. I need to connect the phone to the camera each time I power up the solution.
In terms of the wires, a thin wire connects the camera to its USB cable. This wire runs along the right leg of the glasses. There is also a set of straps that take the weight of the battery pack which hangs behind the glasses. I usually let it hang inside my clothing.
Conclusion
The above solution can be used with any IP camera. You are free to substitute any of the components. You could even use different kinds of cameras such as thermal cameras assuming they support MJPEG streaming.
Are there easier options?
Yes. The vOICe is a flexible platform which you can run on a variety of devices. The other option is to contact Banaao and ask them to assemble the rig. This will be a bespoke project because much will depend upon several factors such as what infrastructure you already have, what you can source locally and what has to be imported.
Acknowledgements
- Mr. Prem Sagar, Founder and CEO of Banaao>
- Various trainees at Banaao include Davis Devasia, Gurjap Singh
- Mikael Holmgren for introducing me to the AI-BALL camera
- Dr. Peter Meijer for modifying the vOICe virtually on the fly to handle large frames which were output from IP cameras.
Post processing images including describing them automatically
As most of you know, I publish plenty of images on this blog. I ensure that all of them are described. The biggest challenge I have in posting photographs on this blog is captioning them. I have to get images described manually before I put them up here. Once I take my photographs I isolate my pictures by location because of the geotagging my phone does. I then send them to people who have been on the trip who describe the images. I have been searching for solutions that describe images automatically. I was thrilled to learn that wordpress had a plugin that used the Microsoft Cognitive Services API to automatically describe images. The describer plugin however did not give me location information therefore I rolled my own code in python. I have created a utility that queries Google for location and the Microsoft Cognitive Services API for image descriptions and writes them to a text file. I had tried to embed the descriptions in EXIF tags but that did not work and I cannot tell why.
References
You will need an API key from the below link.
Microsoft Cognative Services API
The wordpress plugin that uses the Microsoft Cognitive Services API to automatically describe images when uploading
Notes
- You will need to keep your cognitive services API key alive by describing images at least once in every 90 days I think.
- Do account for Google’s usage limits for the geotagging API.
- In the code, do adjust where the image files you want described live as well where you want the log file to be stored.
- Do ensure you add your API key before you run the code.
import glob from PIL import Image from PIL.ExifTags import TAGS from PIL.ExifTags import TAGS, GPSTAGS import piexif import requests import json import geocoder def _get_if_exist(data, key): if key in data: return data[key] return None def get_exif_data(fn): """Returns a dictionary from the exif data of an PIL Image item. Also converts the GPS Tags""" image = Image.open(fn) exif_data = {} info = image._getexif() if info: for tag, value in info.items(): decoded = TAGS.get(tag, tag) if decoded == "GPSInfo": gps_data = {} for t in value: sub_decoded = GPSTAGS.get(t, t) gps_data[sub_decoded] = value[t] exif_data[decoded] = gps_data else: exif_data[decoded] = value return exif_data def _convert_to_degrees(value): """Helper function to convert the GPS coordinates stored in the EXIF to degrees in float format""" d0 = value[0][0] d1 = value[0][1] d = float(d0) / float(d1) m0 = value[1][0] m1 = value[1][1] m = float(m0) / float(m1) s0 = value[2][0] s1 = value[2][1] s = float(s0) / float(s1) return d + (m / 60.0) + (s / 3600.0) def get_lat_lon(exif_data): """Returns the latitude and longitude, if available, from the provided exif_data (obtained through get_exif_data above)""" lat = None lon = None if "GPSInfo" in exif_data: gps_info = exif_data["GPSInfo"] gps_latitude = _get_if_exist(gps_info, "GPSLatitude") gps_latitude_ref = _get_if_exist(gps_info, 'GPSLatitudeRef') gps_longitude = _get_if_exist(gps_info, 'GPSLongitude') gps_longitude_ref = _get_if_exist(gps_info, 'GPSLongitudeRef') if gps_latitude and gps_latitude_ref and gps_longitude and gps_longitude_ref: lat = _convert_to_degrees(gps_latitude) if gps_latitude_ref != "N": lat = 0 - lat lon = _convert_to_degrees(gps_longitude) if gps_longitude_ref != "E": lon = 0 - lon return lat, lon def getPlaceName(fn): lli=() lli=get_lat_lon(get_exif_data(fn)) g = geocoder.google(lli, method='reverse') return g.address def getImageDescription(fn): payload = {'visualFeatures': 'Description'} files = {'file': open(fn, 'rb')} headers={} headers={ 'Ocp-Apim-Subscription-Key': 'myKey'} r = requests.post('https://api.projectoxford.ai/vision/v1.0/describe', params=payload,files=files,headers=headers) data = json.loads(r.text) dscr=data['description'] s=dscr['captions'] s1=s[0] return s1['text'] def tagFile(fn,ds): img = Image.open(fn) exif_dict = piexif.load(img.info["exif"]) exif_dict['Description''Comment']=ds exif_bytes = piexif.dump(exif_dict) piexif.insert(exif_bytes, fn) img.save(fn, exif=exif_bytes) def createLog(dl): with open('imageDescriberLog.txt','a+') as f: f.write(dl) f.write("\n") path = "\*.jpg" for fname in glob.glob(path): print("processing:"+fname) createLog("processing:"+fname) try: imageLocation=getPlaceName(fname) except: createLog("error in getting location name for file: "+fname) pass try: imageDescription=getImageDescription(fname) except: createLog("error in getting description of file: "+fname) pass imgString="Description: "+imageDescription+"\n"+"location: "+imageLocation createLog(imgString) try: tagFile(fname,imgString) except: createLog("error in writing exif tag to file: "+fname) pass
Getting fenrir talking on a raspberry pi
Fenrir is a promising new user mode screen reader. It is primarily written in python 3. Here are my instructions to install it on a raspberry pi.
I am assuming that you are using RASPBIAN JESSIE LITE and are at a terminal prompt.
Updating your installation
You must update your installation of raspbian otherwise components like espeak will not install correctly.
sudo apt-get update
Let the update finish. The above command just fetches the listing of packages that need to be updated. Now do the actual upgrade.
sudo apt-get upgrade -y
This is going to take a while to complete.
You now have to install several dependencies for fenrir to work.
Espeak
This is the speech synthesizer fenrir will use.
sudo apt-get install libespeak-dev
Python 3
sudo apt-get install python3-pip -y
Fenrir is written in python 3.
The python daemon package
sudo apt-get install python-daemon -y
A package that allows fenrir to run as a service.
evdev
sudo pip3 install -evdev
A python package that handles keyboard input.
The speech-dispatcher package
sudo apt-get install speech-dispatcher -y
The above package is required to get fenrir to talk to a speech synthesizer such as espeak.
The ConfigParser package
sudo pip3 install configparser
You may need this dependency to parse the fenrir configuration file.
A module for checking spelling
sudo apt-get install enchant -y
sudo pip3 install pyenchant
An optional package to handle spell checking.
Git
sudo apt-get install git -y
Git is a version control system which you will use to pull down the latest source code of fenrir.
It is now time to install fenrir. Execute the following command.
git clone https://github.com/chrys87/fenrir.git
You now need to start configuring the pi for fenrir to work.
Execute the following command.
sudo spd-conf
You will be asked a series of questions. The main thing you need to change is the kind of sound output technology to use. You need to use alsa if you are using the 3.5MM headphone jack of the pi like I am doing. When you are asked about using pulseaudio type “alsa” without the quotes. Hit enter to move to the next prompt after each question. Do not forget to adjust the speed and pitch to your liking.
You now need to test your speech synthesizer configuration. Do ensure that you have your headphones or speakers ready.
sudo spd-say testing
If you hear the word “testing” you are good to start fenrir. If not, look through your logs and seek support.
To start fenrir, execute the following command.
Warning: You will need to change terminals once you execute the below command so you may want to open a new terminal and keep it handy.
Assuming you are in your home directory type the following commands.
cd fenrir/src/fenrir
sudo python3 ./fenrir
You should hear a stack of startup messages which signifies that fenrir is running.
Usage
I am still playing with fenrir therefore cannot comment much. The key bindings appear to be similar to those of the speek up screen reader. If you want to check them out, take a look at the config file in your installation directory or at the following link.
There is a tutorial mode available accessed with the fenrir key and h. The fenrir key is the insert key by default.
Acknowledgements
Thanks to the following people for helping me get fenrir talking and for raspberry pi advice.
Michael A. Ray
Storm Dragon
Other members on the IRC channel called #a11y at the Server: irc.netwirc.tk; specifically:
PrinceKyle, Jeremiah, chrys and southernprince.
The golden triangle and beyond
Our tour began with the Delhi flower market at Ghazipur. Ghazipur otherwise is better known for its slotterhouse.
The flower market is a whole sale market therefore no organized shops. There are stalls with baskets of flowers and everyone is out to sell.
We then moved to Agra.
In case you are wondering, yes we did visit the Taj but it was my seventh visit to it and I did not take many photographs.
The more interesting bits with respect to the Taj lie outside it. These are the art and craft shops that sell you replicas of the taj and show you how they are made and do other marble related work. Most of the designs involve complex geometrical patterns. The carving is done by hand using a chisel. I did ask about 3d-printing and other machines. I am told that the marble is very soft and machines will break the stone. The designers are also very experienced. There is no cadcam software. The people have been doing it from generations and the skill is passed from father to son as far as I could tell.
We then moved to Jaipur.
I did try using the vOICe to determine how much of the cloth I have colored and have printed on. It is a non-trivial task because the paint / ink is very light when you apply it. It does darken after drying but this is going to take a lot of work. Moreover, because of the poor contrast, it was difficult to make out what patterns I had printed.
One of the most interesting things we did in Jaipur was to visit shops selling gems and Jewry and see how they are made. They had a little setup outside the shop where one could see the workers polishing gems etc. You need to be careful with the cutting wheel and the polishing is intricate work best done during the day because it needs natural light for the colors to stand out. Again, I tried looking at the setup. It may be possible to determine how much of a stone is polished using vision but the hand appeared to be a better tool for the job at least for now. In addition, the process cannot be completely automated because of the soft stones in use and how will a machine create a design? Heuristics of some kind coupled with a neural net with massive amounts of data are perhaps options but until that happens, it is up to human ingenuity to create designs. Again, this trade is based on generations.
It may be possible to tell semiprecious stones apart by their texture but I am uncertain if this is a reliable means of identification.
Be warned, you do not sense the wind if you are outside the palace. The monument is on a busy street and is a quick photo stop unless you want to go inside and climb the several stairs. I did not have an opportunity to do this.
We had a chance to visit fort Amer. The drive up to the fort is a lot of fun if you take the elephant. In addition, forget the term “camera shy” because there an inordinate number of photographers seeking your custom.
We then moved to Jodhpur.
We then moved to Pushkar for sun and sand but found much more.
This trip was special in many ways. For one, I had a multilingual group and our conversations were conducted in English, Hindi and German. I was mistaken for a foreign tourist which was intriguing and most of all, I had not seen my country through the eyes of a foreigner before. It showed me how much I took for granted.
Acknowledgements
Aparna Mathur leadership consultant / art, history and food lover for the image descriptions
Girdhari Singh Shekhawat
our fearless guide and leader and one of the fastest learners I have ever had the chance to meet
Anil Dhyani Shrooti Sharma, June and the other staff of Sita Travels for giving me the opportunity of co-leading the tour and for fantastic organization
Laura Kutter, CEO of Tour de sens for some fantastic conversation and brilliant organization and group management
Marcel and Raphaela Franke for being who you are and for breaking the ice
Barbara Krug for fantastic conversations and navigation in tricky places
Sigrid Gleser for breaking the ice and stimulating technical conversation
Rita Gleser for intriguing teaching opportunities and food
Claudia for the laughs
Teena, Christina, Gregor and everyone else for the fun.