Skip links

techesoterica.com

A blog dealing with sensory substitution and other esoteric concepts and technologies like speech-recognition and chaos theory

Main navigation

Pranav

Coffee shops, dirty tables and Oreo shakes

by ·

It was the weekend and I was on a family outing. We drifted into a coffee shop and found a table without difficulty. My mother remarked that the table was dirty. I tilted my head by 30 degrees and sure enough I could detect some dark objects on the table. I do not know what they were and did not care to find out. Once the waiter had cleaned the table, I took another look. I got a nice clean sound with no obstructions. I am going to develop the habit of scanning every restaurant table.
It took some thinking to figure out what I was going to drink. I do not like coffee so settled for an Oreo shake. The shake came in a jar like container which was short but had a large diameter. I tilted my head down and saw this cylindrical structure. It was astonishingly dark. It appears that Oreo shakes are brown and do not allow light to pass through them. As I drank the shake, I kept glancing at the container and was able to detect an increase in light. Visual confirmation that I was getting through the shake.

getting the orca screen reader working with a raspberry pi

by ·

Many of us want to run the graphical orca screen reader on a raspberry pi, the credit card sized computer for the ultimate in portable computing experiences. This has been done before. I got this working last night and by popular demand, here are my instructions. Be warned, I went down a few rabbit and may be installing a few extra packages but at the end, orca is working.
My thanks to Michaele A. Ray of the website raspberry VI website for doing this first and showing that this is indeed possible and to the members of the raspberry-vi mailing list for help and urging me to write this post.

Prerequisits

You will need to have the following items.

  • A usb keyboard plugged into the pi for use on the pi console.
  • A pair of headphones or speakers connected to the headphone jack.
  • The raspberry pi should be powered and connected to the internet. A wired connection is the best as it is stable and easy to troubleshoot.

  1. Download the full raspbian image

    Download the latest image of raspbian from the below link. Get the full image as opposed to raspbian light. This is because the image has a lot of the graphical user interface things installed.
    download raspbian

  2. Getting the image on to the sd card

    Burn it on to a sd card using your program of choice. If you are using Microsoft windows, one of the most accessible programs is balenaEtchar

  3. Enable ssh at boot

    Once the image has been written, you will see 2 logical partitions on the sd card. There is a partition called “boot”. Create a blank file named “ssh” without the quotes on that partition. The way I do this on Microsoft windows is to go to a comma dprompt, switch to the drive that has the partition and then issue the following command.
    copy con ssh
    You will be in a line editor. Hit ctrl+z to save and close out of the file.

  4. Eject the card and load it into your pi.

  5. Boot the raspberry pi and login to it over ssh.

  6. Update the raspberry pi
    This step is crucial. Failure to do this will result in orca not speaking. Run the following commands.
    sudo apt-get update
    sudo apt-get upgrade -y
    sudo apt-get dist-upgrade -y
    Once the above commands have completed successfully, reboot the raspberry pi.

  7. Configure the pi to login to a text console

    run the following command.
    sudo raspi-config
    Read the screen output very carefully. You need to go into the boot options and enable the option to login to the console without a password.

  8. Once you have enabled the login, and are back on the main raspi-config screen, hit shift+tab once to save your settings and exit. Reboot.

  9. IInstall dependencies

    This is the trickiest part of the entire exercise.
    Orca needs a bunch of things to run. As of this writing, speech dispatcher and espeak are not installed by default.
    sudo apt-get install sox -y
    Install the sox package for multimedia libraries. Some of them may be needed by speech dispatcher.
    sudo apt-get install speech-dispatcher -y
    Install the speech dispatcher service. Orca needs to talk to speech synthesizer. Be warned, this is an older version of speech dispatcher. There is a new one available on github but I have not tried compiling it from source on this installation.
    sudo apt-get install espeak -y
    The espeak speech synthesizer
    sudo apt-get install gnome-orca -y
    The orca screen reader and associated dependencies

Once you have carried out the above steps, reboot the raspberry pi. The text console login comes in handy here. Ensure that you have a USB keyboard plugged in. You may want to ssh into the pi to ensure that the pi has booted. If you have configured the raspberry pi to automatically login into a text console, you can enter the following at the command prompt. Be warned, at this point, you do not have any speech therefore you should know your keyboard well.
startx
You will have to wait for at least 2 minutes for the gui to load. I am giving a safety margin. Once again, you do not have speech at this point. Press alt+f2. This will place you at a prompt where you can enter a command to execute.
Type the following
orca
You will need to wait yet again but you should hear orca talking.

There is much left to do. I am still working on how to start orca automatically once the gui loads and need to find a good way for this to coexist with a console screen reader such as fenrir. I will update this post as I get feedback and learn more.

Open kitchens and the wonder of food in progress

by ·

I was at MKT. The restaurant is setup differently to most such establishments. It is in the lower ground floor of the Chanakya mall and boasts a variety of cuisines. You can read about them at the above link. Visually, it is a fascinating place thanks to the live kitchens. I have spent many hours looking at food and one of the biggest problems with food is the lack of contrast. It is usually difficult to get good light and food with a high contrast as compared to your plate such that you can see the food and eat. As usual, I was wearing the vision 800 glasses running the vOICe Sacheta and I wanted to see their 4 open kitchens. Once again, I was in a situation where touching was not appropriate because the chefs were stacking food outside, ready for the waiters to pickup. I wanted to see the activity. The kitchens were behind transparent glass but they did have large windows to allow the cooks to send the food out. I was able to see the clean rising flame of the Italian kitchen and the round pizzas that emerged from it. The nachos were lined up in baskets outside the live Mexican counter. They were cylindrical but there were some differences in their shapes in stacking.
We then had a look at the Indian section where the cook was making chapatis. I could see these as dark disks lying on the counter. This was one of those situations where I used my other senses to zero on to the object of visual interest. I localized the sound of the stacking and then pointed my head in that direction. Visual scanning would have given me the same information but I treat vision as a multi sensory process.
We then moved to the Chinese counter where there was some kind of machine and a lot of activity taking place.
I did once again try to watch the cooks at work but was unable to perceive the actual motion of the people. I did notice the rapid changes in the scenery so could tell something was happening.
Sacheta took a few videos after seeking permission from the staff. That gave me the chance to stand still and look. Panning my head from side-to-side also helped and when looking through a window, pan up and down for maximum visual coverage. You can see details without panning but the devil is in the details and a little interaction brings a lot of clarity.
My special thanks to Mr. Pankaj Mishra, one of the manager’s of MKT for being so welcoming. The service is good and, before I forget, the food was excellent.
Pro food tip: do try the house special when it comes to drinks.
Pro vision tip: when your table has a lot of items on it, scan your vicinity for that tall glass of mocktail so that you can grab it in one shot. Blindness techniques work too but scanning is so much cooler.

A review of Machine Learning is Fun! by Adam Geitgey

by ·

I bought the book with much anticipation since I am a regular reader of Mr. Geitgey’s posts. The book did not disappoint. I particularly enjoyed the introductory section on neural networks, specially, , the lucid description of forward and back propagation. I have read many references on the web and have taken the famous machine learning course by Andrew NG but none of those references explained how a neural network works as well as machine learning is fun did.
The code examples are easy-to-read and are well organized.
The VMware virtual machine is a nice touch.
I would have liked to see more discussion about adversarial neural networks and generative neural networks. In addition, more details of commonly used optimization algorithms such as gradient descent would have been welcome.
Finally, a section on how to install the several libraries mentioned would also be handy.

Standing in line at the supermarket checkout

by ·

Sacheta and I were at the supermarket. We were standing in the checkout line when she remembered something, told me to hold our place and was gone. I was wearing my glasses and had the vOICe running. My first thought was “Woo hm”, how do I keep our place? My second thought was “hmm, I am sighted so lets see what I can do.” I studied the view in greater detail and moved my head a bit to the left and then to the right. I could see a constant shape in front of me. I heard the rustle of a plastic bag ahead of me and confirmed that the constant shape was probably the back of a person.
I then turned my head to the left and checked the view. I could see more objects. I did the same on the right and the view was similar. I now knew that I was focused on the right object. I stepped a few paces in front until almost the full left to right panning was filled by that constant shape. I did not want to invaid the stranger’s personal space so also used other blindness skills like passive echo location. The individual moved away and the view opened up. It was my turn at the cash counter.
The key takeaway for me was the checking I did. In this situation, the narrower view given by the glasses helped because I was able to associate a physical action with the change of the scene when I turned my head sideways. I could have done the same thing with a wide angle view but would have had to pay more attention to perspective.

Acing the supermarket with artificial vision

by ·

It was another weekend family outing we were at the DLF Place mall.
I was sitting at the back of the car, with the back of my head resting on the headrest. I was able to look out the windscreen and was able to see the traffic outside. This is a good way to get used to the looming effects of the vOICe. Objects come into the foreground as they come closer and move away as you and or they drive away. Pay attention to the volume and the way the sound moves to determine what is in the foreground. In addition, imperfections on the object will show up as small objects.
At the mall, I spent most of my time in a shop called Food Hall which sells a variety of comestibles. Mom and dad were supervising the grinding of fresh turmeric therefore Sacheta and I had a fair bit of time to look around. Most of the food is in packets though things like bars of chocolate are kept as is on the shelves. There are also freezers that contain a variety of meat, vegetables etc.
The OCR kept reading out snippets from labels and other signage so I was able to find something with coco at the biscuit counter and look at some of the chocolate bars at the chocolate counter. The OCR was particularly handy when I was standing at one place waiting for Sacheta to finish studying something like the counter in front of us. A quick note about looking at the chocolate counter. Beware of the OCR because chocolate bars have a lot of text on them so there is a fair bit of information to absorb and the OCR will take its time. I had a similar experience with signage. There is a restaurant called the hard rock cafe next to Food Hall. It has superb signage which I can read from at least 3 to 5 feet away. The same does not apply to Food Hall. I spent some time looking at its sign but I could only read the word “food”. One more thing, words on packets and signs may be positioned one below the other therefore be ready to scroll and pan by moving your head. The panning is needed due to the large fonts that are used.
Another handy thing that I was able to do was to determine if shelves contained bottles or packets. I use the term “bottles” to include jars and other cylindrical containers. This is important information because I was extra careful near these shelves. Those bottles tend to fall easily and if I am lucky, break. Not that this has ever happened to me but this gives the bottles some extra security and my credit card a lot of peace. The way to check this is to get the shelf into the foreground and focus on what shape emerges. Practice at home with bottles so that you know what they look like. You can hear the rising circular shape or a circular shape that has a near uniform pitch indicating a jar/. Stay about half an arm length away from the shelf for best results.
I had the leisure to study packing. Each packet had labels in different fonts and with different font sizes. I don’t know how manufacturers decide the fonts and sizes of labels. However, if any of you are reading this, please use large fonts and keep them standard so that they are machine and human readable. Take the case of a packet that contained “premium almonds;” I was able to read the word “premium” but not the word “almonds.” I suspect this was a font and perhaps a camera placement issue though the word “almonds” was in a large font but Sacheta told me that its styling was different. Small fonts do not help.
If you hear a lot of hard surfaces in the soundscape and or a tropical bird like call, you can be quite certain that you are looking at shelves of packets. If there is space between the hard parts of the soundscapes and you detect a cylinder, then you are looking at shelves of small bottles. You can better experiment with this at shops that sell makeup.
Looking inside the freezers did not yield much information. There were things inside them but I was unable to read their labels and the lighting was variable.

Where is that milkshake?

Once we had finished shopping, we went to Big Chill for beverages and bites. I had ordered a Swiss chocolate milkshake. The waiter placed the glass on the table saying “Swiss chocolate” and vanished. I usually ask someone where the glass is placed. This time, I tilted my head by a few degrees, increased the volume of my bone conduction headset and studied the table to look for that cylinder. I found it and got the glass without mishap. Thanks Dr. Meijer for emphasizing grasping exercises!
Talking about the volume control, it is important hat you can regulate the volume of the soundscapes. An external control is the best for this. Set the vOICe to a volume of 7 or 8 if you are in a noisy environment like a mall and then use the hardware control on the headset to tweak the volume based on your needs.
Finally, if you detect pixelated soundscapes in a shop, it could mean that the lighting is positioned above the shelf or there is a single light source shining down on to the shelf. In this case, get closer to the shelf and do what you can, else ask the staff for help.

Shopping for footwear using synthetic vision

by ·

Sacheta wanted to buy footwear, so there we were, in Kala Niketan, Janpath on a Saturday evening looking for bellies and such. As usual, I was wearing my Vision 800 glasses running the vOICe. One advantage in footwear shopping is that the customer needs to sit to try it. The shop was not too crowded and we did not have a problem in finding a seat. However, before sitting, we looked around the shop, asking the staff for what she wanted. There were shelves packed with footwear. I was unable to understand the shapes until I touched a slipper. It had been stacked on its end such that the strap that comes on the top of your foot was facing the customer. This is different from how shoes are stacked because shoes are kept with their backs to the customer or at least that was the way they were kept the last time I checked.
The shapes were uniform from a far and the shelf boundaries formed hard edges in the scene. The shop had a large range of footwear in several colors. Once I was seated, I began to play with the color filter. Blue, green and red did not yield much feedback. I hit pay dirt when I chose orange. This was strange. I could hear shoppers asking for a colors like “rose gold”, whatever that is. Where was this color? I don’t think anyone wears orange shoes but then what do I know? The analyze option came in really handy. I sat back letting the scan tell me what filters were working. I then enabled the live OCR which introduced a new wrinkle. As I panned, I heard “fresh stock.” Hmm, what could they mean? Were they referring to the latest fashion? Was someone baking shoes? I could not ask Sacheta because she was engaged with a persistent sales person.
Once she had completed her shopping I was able to clear up some of the mysteries. The orange filter worked probably because the shop used a lot of yellow light. The “fresh stock” meant that there was no sale on those items. Don’t ask me why they could not say this up front.
I did have a chance to look at shoes and to distinguish details, my palm remains the best tool. However, with some more practice, I think I will be able to reliably distinguish between shelves full of slippers and shelves where shoes are stacked.
I did not buy anything so did not get a chance to try walking but I can see myself creating a visual landmark of a shelf or something else and walking independently to test shoes.
Some may ask, what is the point of it all? The simple way to buy footwear is to ask for what you want and try until you like something. Yes, this approach works well but my scene reader helped me to get a better idea of what was in the shop and thereby make a more informed choice.

Note

Photography is not allowed in the shop so I did not take any images.

Note 2

Be careful when looking and searching for a place to sit. These shops are full of humans sitting and not paying attention to their surroundings. Stay in the middle of the isle as much as you can. Let the staff guide you to a seat.