Skip links

techesoterica.com

A blog dealing with sensory substitution and other esoteric concepts and technologies like speech-recognition and chaos theory

Main navigation

concepts

The case for synthetic vision in the modern office

by ·

I often wear the the vOICe running on my vision 800 glasses to my office. Some of you may find this surprising. I have good blindness skills and the office is a known environment. Moreover, it is visually the same that is, not much changes at least in my office.
So why then bother with synthetic vision? Why not use something like a cane?
I can certainly use a cane and these days, this has become even easier with the smart cane made by IIT Delhi in collaboration with a number of partners. However, sonar and the cane do not give me sufficient information.
My office is on the second floor of a building. I usually take the stairs for two reasons. I like the exercise and the elevators do not talk. When I am climbing the stairs, it is very easy for me to read the floor indicator so I do not cross the second floor and end up at the cafeteria. It is also easier to walk towards doors because I can see them from a distance so can zero on them and reduce wiering. In the event that I do not see the floor indicator, I am able to see the walls. The office has windows on the landings and they are visually distinct as compared to the doors.
I accidentally discovered that the the vOICe is very handy during presentations because I can “look” at the screen of my laptop and read the text because said text is enlarged and the OCR in the vOICe picks it up without problems. This is very handy when evaluating products that contain dashboards etc.
I continue to refine my skills at detecting people, chairs and other obstacles. One experiment I plan to do is to read the names of meeting rooms.
I currently compliment synthetic vision with my blindness skills and may add other technology into the mix.

Installing opencv on Arch Linux from source

by ·

I am going to be showing you how to install the OpenCV computer vision library from source. This is the best way to install this library to ensure that you get the full set of features this library has to offer. Moreover, it allows you to customize your installation. For example, you may not want any python bindings.
This post has been inspired by the tutorials at PyImageSearch and Learn OpenCV.
The bulk of this tutorial has been taken from the superb tutorial at
Install OpenCV 4 on your Raspberry Pi

Installing dependencies

The first thing you need to do is to install dependencies. I am a newcomer to Arch Linux. I come from Ubuntu. The package names are different between Ubuntu and arch Linux. You need to use a search string like
"arch linux" in your search engine of choice to find the package names.
If you wanted to search for the package called build-essentials in Arch Linux, the search string you would use is
"Arch linux" build-essentials
The tutorial on installing OpenCV on the raspberry pi lists a series of dependencies. I have created an Arch Linux mapping to the Ubuntu dependencies and have given the equivalent commands.

Note

I am using the image created by the F123 project which as an accessible Arch Linux on the raspberry pi. I installed OpenCV on that image therefore most of my dependencies were met.
Run each of the commands and follow the prompts. If a dependency is already installed, then select not to reinstall it.

Raspbian Arch Linux
sudo apt-get install build-essential cmake unzip pkg-config yay -S build-devel
yay -S cmake
sudo apt-get install libjpeg-dev libpng-dev libtiff-dev yay -S libjpeg
yay -S libtiff
yay -S libpng
sudo apt-get install libxvidcore-dev libx264-dev yay -S xvidcore
yay -S libx264
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev yay -S ffmpeg
yay -S v4l-utils
sudo apt-get install libgtk-3-dev yay -S gtk3
sudo apt-get install libcanberra-gtk* yay -S libcanberra
sudo apt-get install libatlas-base-dev gfortran yay -S lapack
yay -S gcc-fortran
sudo apt-get install python3-dev already done because python development headers are already installed

Setting up python version 3

I am maximizing my use of python 3 which is what I will be using here. Python3 is already installed therefore, all we need to do now is to create a python virtual environment to keep things isolated.
sudo pip3 install virtualenv virtualenvwrapper
Clean the package cache.
sudo rm -rf ~/get-pip.py ~/.cache/pip
We need to enable some convenience commands to be able to redily invoke our virtual environment. Ensure that you are in your home directory.
cd ~ and then press enter.
Run the following commands to add to your .profile file.

echo -e "\n# virtualenv and virtualenvwrapper" >> ~/.profile
echo "export WORKON_HOME=$HOME/.virtualenvs" >> ~/.profile
echo "export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3" >> ~/.profile
echo "source /usr/bin/virtualenvwrapper.sh" >> ~/.profile

Once the above commands are run, you can verify that the file has been created correctly by examining its contents.
cat .profile
Activate the commands by issuing the following command.
source .profile

Download OpenCV

Ensure that you are in your home folder.
cd ~
Run the following commands to download OpenCV version 4, the latest at the time of this writing to your raspberry pi.

wget -O opencv.zip https://github.com/opencv/opencv/archive/4.0.0.zip
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.0.0.zip

Expand OpenCV and set up the folders

It is time to expand the zip archives you downloaded.

unzip opencv.zip
unzip opencv_contrib.zip

Rename the folders for easy reference.

mv opencv-4.0.0 opencv
mv opencv_contrib-4.0.0 opencv_contrib

Building OpenCV

Make the virtual environment

Run the following command to make your OpenCV virtual environment.

mkvirtualenv cv -p python3

You can name your virtual environment whatever you like. If you do name it differently, remember to account for that during the rest of this tutorial.
Switch to your virtual environment by executing the following command.

workon cv

The command prompt will change such that the name of your virtual environment will be displayed as a part of the prompt. You can exit the environment by using the following command.

deactivate

Install numpy

You need to install a dependency for OpenCV python.
Ensure that you are in your virtual environment.

workon cv


pip install numpy

The installation of numpy is going to take quite some time. Do not be alarmed.

Generate the build files

It is time to create the build files for OpenCV.
Switch to the opencv folder, create a build folder to hold the build files and then switch to it.

cd ~/opencv
mkdir build
cd build

Run cmake to generate the files. If you are copy pasting the below lines, do not manually try to delete the line breaks and create a single huge command. Copy paste each line and hit enter. The two fulstop characters at the end of the last line tell cmake that it should run.

cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
-D ENABLE_NEON=ON \
-D ENABLE_VFPV3=ON \
-D BUILD_TESTS=OFF \
-D INSTALL_PYTHON_EXAMPLES=OFF \
-D OPENCV_ENABLE_NONFREE=ON \
-D CMAKE_SIZEOF_VOID_P=4 \
-D BUILD_opencv_python3=TRUE \
-D BUILD_EXAMPLES=OFF ..

Do inspect the output and ensure that you do not see any significant errors. You may want to enable the logging facility in your terminal emulation program assuming you are running headless on your raspberry pi.

Increase the size of the swap file

The raspberry pi is a memory constrained device. You need to create a swap file to allow for the compile to complete. It is crucial that you do this else the compilation process will not complete.
In Arch Linux, you need to create a swap file first and then activate it. Issue the following commands.

sudo dd if=/dev/zero of=/swp bs=1024 count=4000000
sudo chmod 600 /swp
sudo mkswap /swp
sudo swapon /swp

See below for what these commands mean.

sudo dd if=/dev/zero of=/swp bs=1024 count=4000000

Create a file which will become our swap file. This file must be as large as the swap file we want. I have taken 4GB for safety.

sudo chmod 600 /swp

Assign the right permission to the file we have created.

sudo mkswap /swp

Convert our ordinary file to a swap file.

sudo swapon /swp

Activate the swap file.

Compile and install OpenCV

Ensure that you are in the build folder.

cd ~/opencv/build

Warning, the below command takes several hours to complete. It is the longest part of this procedure. Keep the logging in your terminal emulator active.

make

You can make things go faster by issuing the -j flag with make.

make -j

This utilizes all available CPU cores. You can specify the number of cores you want make to use.

make -j2

The above command will use only two cores. In my experience, running just make is sufficient. I have had the compilation being interrupted if I have used more than one core. Your mileage may vary.
If make completes without errors then run the following commands.

sudo make install
sudo ldconfig

OpenCV is now installed. There are however more steps to follow to ensure that the python bindings work with our virtual environment.

Turn off the swap file

It is crucial that you switch off the swap file else you risk reducing the life of the SD card in the raspberry pi due to the increased write operations that the pi will perform when it swaps memory to the SD card.

sudo swapoff /swp

You can recover space by deleting the swap file.

sudo rm /swp

Linking in the OpenCV binding into our python virtual environment

It is time to create a sim link to the OpenCV library in our python virtual environment.
Issue the following commands.

Note

I am writing this tutorial based on python 3.7. The version of python may change therefore please adjust your steps accordingly.

Run the below commands.

cd ~/.virtualenvs/cv/lib/python3.7/site-packages/
ln -s /usr/local/python/cv2/python-3.7/cv2.cpython-37m-arm-linux-gnueabihf.so cv2.so

Testing your installation

Carry out the following steps to verify if you have installed OpenCV with its python bindings correctly.

workon cv

Ensure that the virtual environment is active. If the above command fails, then do a
source .profile
to ensure that the relevant commands are loaded.
Launch python by issuing the following command
python
Then enter the following text in the interpreter that comes up.

import cv2

If the above commands without an error and your cursor moves to the next line, you can type:

cv2.__version__

You should see the version of OpenCV you have installed. Execute

quit()

to exit python.

Troubleshooting

If the make fails, ensure that your swap file is active and do not issue the -j flag.
If you get unusual errors and the compilation process goes very fast but stops in the middle, you should check the paths to your contrib folder and ensure that all other paths are right.

Coffee shops, dirty tables and Oreo shakes

by ·

It was the weekend and I was on a family outing. We drifted into a coffee shop and found a table without difficulty. My mother remarked that the table was dirty. I tilted my head by 30 degrees and sure enough I could detect some dark objects on the table. I do not know what they were and did not care to find out. Once the waiter had cleaned the table, I took another look. I got a nice clean sound with no obstructions. I am going to develop the habit of scanning every restaurant table.
It took some thinking to figure out what I was going to drink. I do not like coffee so settled for an Oreo shake. The shake came in a jar like container which was short but had a large diameter. I tilted my head down and saw this cylindrical structure. It was astonishingly dark. It appears that Oreo shakes are brown and do not allow light to pass through them. As I drank the shake, I kept glancing at the container and was able to detect an increase in light. Visual confirmation that I was getting through the shake.

getting the orca screen reader working with a raspberry pi

by ·

Many of us want to run the graphical orca screen reader on a raspberry pi, the credit card sized computer for the ultimate in portable computing experiences. This has been done before. I got this working last night and by popular demand, here are my instructions. Be warned, I went down a few rabbit and may be installing a few extra packages but at the end, orca is working.
My thanks to Michaele A. Ray of the website raspberry VI website for doing this first and showing that this is indeed possible and to the members of the raspberry-vi mailing list for help and urging me to write this post.

Prerequisits

You will need to have the following items.

  • A usb keyboard plugged into the pi for use on the pi console.
  • A pair of headphones or speakers connected to the headphone jack.
  • The raspberry pi should be powered and connected to the internet. A wired connection is the best as it is stable and easy to troubleshoot.

  1. Download the full raspbian image

    Download the latest image of raspbian from the below link. Get the full image as opposed to raspbian light. This is because the image has a lot of the graphical user interface things installed.
    download raspbian

  2. Getting the image on to the sd card

    Burn it on to a sd card using your program of choice. If you are using Microsoft windows, one of the most accessible programs is balenaEtchar

  3. Enable ssh at boot

    Once the image has been written, you will see 2 logical partitions on the sd card. There is a partition called “boot”. Create a blank file named “ssh” without the quotes on that partition. The way I do this on Microsoft windows is to go to a comma dprompt, switch to the drive that has the partition and then issue the following command.
    copy con ssh
    You will be in a line editor. Hit ctrl+z to save and close out of the file.

  4. Eject the card and load it into your pi.

  5. Boot the raspberry pi and login to it over ssh.

  6. Update the raspberry pi
    This step is crucial. Failure to do this will result in orca not speaking. Run the following commands.
    sudo apt-get update
    sudo apt-get upgrade -y
    sudo apt-get dist-upgrade -y
    Once the above commands have completed successfully, reboot the raspberry pi.

  7. Configure the pi to login to a text console

    run the following command.
    sudo raspi-config
    Read the screen output very carefully. You need to go into the boot options and enable the option to login to the console without a password.

  8. Once you have enabled the login, and are back on the main raspi-config screen, hit shift+tab once to save your settings and exit. Reboot.

  9. IInstall dependencies

    This is the trickiest part of the entire exercise.
    Orca needs a bunch of things to run. As of this writing, speech dispatcher and espeak are not installed by default.
    sudo apt-get install sox -y
    Install the sox package for multimedia libraries. Some of them may be needed by speech dispatcher.
    sudo apt-get install speech-dispatcher -y
    Install the speech dispatcher service. Orca needs to talk to speech synthesizer. Be warned, this is an older version of speech dispatcher. There is a new one available on github but I have not tried compiling it from source on this installation.
    sudo apt-get install espeak -y
    The espeak speech synthesizer
    sudo apt-get install gnome-orca -y
    The orca screen reader and associated dependencies

Once you have carried out the above steps, reboot the raspberry pi. The text console login comes in handy here. Ensure that you have a USB keyboard plugged in. You may want to ssh into the pi to ensure that the pi has booted. If you have configured the raspberry pi to automatically login into a text console, you can enter the following at the command prompt. Be warned, at this point, you do not have any speech therefore you should know your keyboard well.
startx
You will have to wait for at least 2 minutes for the gui to load. I am giving a safety margin. Once again, you do not have speech at this point. Press alt+f2. This will place you at a prompt where you can enter a command to execute.
Type the following
orca
You will need to wait yet again but you should hear orca talking.

There is much left to do. I am still working on how to start orca automatically once the gui loads and need to find a good way for this to coexist with a console screen reader such as fenrir. I will update this post as I get feedback and learn more.

Open kitchens and the wonder of food in progress

by ·

I was at MKT. The restaurant is setup differently to most such establishments. It is in the lower ground floor of the Chanakya mall and boasts a variety of cuisines. You can read about them at the above link. Visually, it is a fascinating place thanks to the live kitchens. I have spent many hours looking at food and one of the biggest problems with food is the lack of contrast. It is usually difficult to get good light and food with a high contrast as compared to your plate such that you can see the food and eat. As usual, I was wearing the vision 800 glasses running the vOICe Sacheta and I wanted to see their 4 open kitchens. Once again, I was in a situation where touching was not appropriate because the chefs were stacking food outside, ready for the waiters to pickup. I wanted to see the activity. The kitchens were behind transparent glass but they did have large windows to allow the cooks to send the food out. I was able to see the clean rising flame of the Italian kitchen and the round pizzas that emerged from it. The nachos were lined up in baskets outside the live Mexican counter. They were cylindrical but there were some differences in their shapes in stacking.
We then had a look at the Indian section where the cook was making chapatis. I could see these as dark disks lying on the counter. This was one of those situations where I used my other senses to zero on to the object of visual interest. I localized the sound of the stacking and then pointed my head in that direction. Visual scanning would have given me the same information but I treat vision as a multi sensory process.
We then moved to the Chinese counter where there was some kind of machine and a lot of activity taking place.
I did once again try to watch the cooks at work but was unable to perceive the actual motion of the people. I did notice the rapid changes in the scenery so could tell something was happening.
Sacheta took a few videos after seeking permission from the staff. That gave me the chance to stand still and look. Panning my head from side-to-side also helped and when looking through a window, pan up and down for maximum visual coverage. You can see details without panning but the devil is in the details and a little interaction brings a lot of clarity.
My special thanks to Mr. Pankaj Mishra, one of the manager’s of MKT for being so welcoming. The service is good and, before I forget, the food was excellent.
Pro food tip: do try the house special when it comes to drinks.
Pro vision tip: when your table has a lot of items on it, scan your vicinity for that tall glass of mocktail so that you can grab it in one shot. Blindness techniques work too but scanning is so much cooler.

A review of Machine Learning is Fun! by Adam Geitgey

by ·

I bought the book with much anticipation since I am a regular reader of Mr. Geitgey’s posts. The book did not disappoint. I particularly enjoyed the introductory section on neural networks, specially, , the lucid description of forward and back propagation. I have read many references on the web and have taken the famous machine learning course by Andrew NG but none of those references explained how a neural network works as well as machine learning is fun did.
The code examples are easy-to-read and are well organized.
The VMware virtual machine is a nice touch.
I would have liked to see more discussion about adversarial neural networks and generative neural networks. In addition, more details of commonly used optimization algorithms such as gradient descent would have been welcome.
Finally, a section on how to install the several libraries mentioned would also be handy.

Shopping for footwear using synthetic vision

by ·

Sacheta wanted to buy footwear, so there we were, in Kala Niketan, Janpath on a Saturday evening looking for bellies and such. As usual, I was wearing my Vision 800 glasses running the vOICe. One advantage in footwear shopping is that the customer needs to sit to try it. The shop was not too crowded and we did not have a problem in finding a seat. However, before sitting, we looked around the shop, asking the staff for what she wanted. There were shelves packed with footwear. I was unable to understand the shapes until I touched a slipper. It had been stacked on its end such that the strap that comes on the top of your foot was facing the customer. This is different from how shoes are stacked because shoes are kept with their backs to the customer or at least that was the way they were kept the last time I checked.
The shapes were uniform from a far and the shelf boundaries formed hard edges in the scene. The shop had a large range of footwear in several colors. Once I was seated, I began to play with the color filter. Blue, green and red did not yield much feedback. I hit pay dirt when I chose orange. This was strange. I could hear shoppers asking for a colors like “rose gold”, whatever that is. Where was this color? I don’t think anyone wears orange shoes but then what do I know? The analyze option came in really handy. I sat back letting the scan tell me what filters were working. I then enabled the live OCR which introduced a new wrinkle. As I panned, I heard “fresh stock.” Hmm, what could they mean? Were they referring to the latest fashion? Was someone baking shoes? I could not ask Sacheta because she was engaged with a persistent sales person.
Once she had completed her shopping I was able to clear up some of the mysteries. The orange filter worked probably because the shop used a lot of yellow light. The “fresh stock” meant that there was no sale on those items. Don’t ask me why they could not say this up front.
I did have a chance to look at shoes and to distinguish details, my palm remains the best tool. However, with some more practice, I think I will be able to reliably distinguish between shelves full of slippers and shelves where shoes are stacked.
I did not buy anything so did not get a chance to try walking but I can see myself creating a visual landmark of a shelf or something else and walking independently to test shoes.
Some may ask, what is the point of it all? The simple way to buy footwear is to ask for what you want and try until you like something. Yes, this approach works well but my scene reader helped me to get a better idea of what was in the shop and thereby make a more informed choice.

Note

Photography is not allowed in the shop so I did not take any images.

Note 2

Be careful when looking and searching for a place to sit. These shops are full of humans sitting and not paying attention to their surroundings. Stay in the middle of the isle as much as you can. Let the staff guide you to a seat.