Skip links

techesoterica.com

A blog dealing with sensory substitution and other esoteric concepts and technologies like speech-recognition and chaos theory

Main navigation

Pranav

The case for synthetic vision in the modern office

by ·

I often wear the the vOICe running on my vision 800 glasses to my office. Some of you may find this surprising. I have good blindness skills and the office is a known environment. Moreover, it is visually the same that is, not much changes at least in my office.
So why then bother with synthetic vision? Why not use something like a cane?
I can certainly use a cane and these days, this has become even easier with the smart cane made by IIT Delhi in collaboration with a number of partners. However, sonar and the cane do not give me sufficient information.
My office is on the second floor of a building. I usually take the stairs for two reasons. I like the exercise and the elevators do not talk. When I am climbing the stairs, it is very easy for me to read the floor indicator so I do not cross the second floor and end up at the cafeteria. It is also easier to walk towards doors because I can see them from a distance so can zero on them and reduce wiering. In the event that I do not see the floor indicator, I am able to see the walls. The office has windows on the landings and they are visually distinct as compared to the doors.
I accidentally discovered that the the vOICe is very handy during presentations because I can “look” at the screen of my laptop and read the text because said text is enlarged and the OCR in the vOICe picks it up without problems. This is very handy when evaluating products that contain dashboards etc.
I continue to refine my skills at detecting people, chairs and other obstacles. One experiment I plan to do is to read the names of meeting rooms.
I currently compliment synthetic vision with my blindness skills and may add other technology into the mix.

Seeing the first ever image of a black hole thanks to the vOICe

by ·

Today is a remarkable day for science. Researchers have released the first ever image of a black hole. See the below link for details.
First ever black hole image released
If you read the page, you will find that the images have alternative text and excellent captions. If you however want to perceive a black hole, textual descriptions fall far short. Leaving things to the reader’s imagination is good when reading novels but if you want accuracy you need something experiential. This is given by the The vOICe.
It is important to read the captions because they tell you what to expect in the image given that images of black holes are not routine. When I loaded the image from the story into The vOICe, I was able to make out the bright disk around the black hole due to the increase in pitch which meant that something high was in the image. The volume of the sound also increased signifying that there was something bright in the view. The panning of the sound allows me to find the black hole which is a soft agglomeration of dots towards the bottom right of the image.
If you are using NVDA, you may want to use the audio screen add-on to see the image. If however, you do not use audio screen and or NVDA then grab the image from the following direct link.
Image of black hole
You will need to use the vOICe running on windows or on the android version to import the image into the program to view to your heart’s content. When viewing the image with the vOICe, reduce the speed to quarter speed to fully enjoy the black hole and its surrounding bright ring of gasses. Zoom did not help much and negative video is not worth trying because you do not want the background.

Watching full screen youtube videos with the vOICe

by ·

The below instructions have been written by Devin Prater, a fellow user of the The vOICe
I have not edited these instructions except to activate links and to fix formatting.

Okay, here’s a guide on getting the vOICe working well with YouTube
videos, through Foobar2000 and Foo_ YouTube.

* First off, download the vOICe for Windows The first time you run it, you’ll see a license agreement, agree to it.
Running it also puts an icon for it * on your desktop, called “The vOICe.”
* After this, you should hear a repeating sound scape, which is a car
in a parking lot, I believe.

Now, let’s get Foobar2000, which is a media player which can be
extended to offer the ability to play YouTube videos.

* Go to foobar2000.org, and press enter on the download link.

* Download the version under the “latest stable version.”

* Once it’s downloaded, open the installer file, and install it as you
would a normal program. During the installation steps, you may choose
to install the “full package,” with all components, which add
functionality, or choose which components you have installed.

* After installing Foobar2000, open it from the desktop or start menu.
You’ll be given a chance to set up how Foobar2000 looks, but we’re not
worried about that, so just close that dialog box.

* You’ll land on a “playlist list.” That is where Foobar2000 collects
all the things you wish to play.

* Currently, there are no keyboard shortcuts set up for playing,
pausing, rewinding, or fast forwarding media. You can set these up by
pressing
Control+P, tabbing to the tree few, and pressing K for keyboard shortcuts.
Once there, press Tab twice to “add new, shift tab to the list and
make sure you’re on the new keyboard command, called “not set,” tab to
the tree view of actions, find the one you want, tab to the edit
field, and press the key you want to assign to the command.

* Continue setting up shortcuts until you’re happy with the functionality.
You don’t have to do all this, it just makes things a bit easier when
playing media.
Now, let’s get the Foo_Youtube component.

* You can download the Foo_Youtube component from
https://fy.3dyd.com/download/
* The download links are in the table on the page.
* Once downloaded, open the component. It’ll open in Foobar2000, and
will ask if you want to install the component.
* Click yes, then press OK in the preferences dialog that is open.
Foobar2000 will say it needs to restart, press OK again.

* After Foobar2000 restarts; or if it doesn’t restart just open it
from the desktop, you’ll be back at the playlist list.

* Now, we’ll need to set Foo-YouTube to go into full screen when a
video starts.

* Press Control+P for Preferences, Y for YouTube, right to expand that
section, down to Video, tab to Show Window Combo Box, and arrow to
“manually, full screen.” Tab to OK, and press enter.

* Now, you can start the vOICe if it’s not already started.

* You’ll hear “left, right,” in your left and right headphone, and
that car in a parking lot sound.

* Now, press Control+F9 to turn on the sonification of the “window”.

* If you want to explore other options, press Alt in The vOICe’s
window, and arrow right to the options menu, and down through the options there.
The relevant option we just turned on is in the “Sonify GUI” menu.

* Now that we have that going, we can start a YouTube video. To do
this, we’ll go back to the Foobar2000 window, and press Alt+F, then Y, then T.
This opens the file menu, the YouTube submenu, then chooses the Title
option.

* There, we have an edit box, and a combo box with options. Type
something into the edit box.

* Now, shift tab to the combo box, and arrow down to the Limits submenu.
Press right arrow to the into the menu, and you’ll probably have “1.”
This limits the amount of videos to just one, which isn’t at all how
YouTube usually works.

* Press Enter on 1, type in 50, and press the OK button.

* Now, you’ll be put back into the dialog box where you enter what
you’re searching for.

* Now, after making sure you have entered what you’re looking for,
press the OK button.

You’ll be placed on a dialog box as the videos are searched.

* After they’ve been found, you’ll be placed back on the file menu in
the
Foobar2000 window. Press either escape or alt to get back to the playlist.

* Now, press down arrow. You’ll hear the first video’s title. Press enter.
When you hear the video begin to play, press Alt+V, then V, then Enter.
Alt+V goes to the video menu, V to the visualization menu, and Enter
selects “video.”

Now, the vOICe will be sonifying the video. To exit the video and go
back to the results, press escape. To get back into the video, do
Alt+V, V, enter again.

* To search for something else, Press Control+A to select all results,
and delete to delete them from the playlist, and search for videos again.

* when you’re done, press Alt+F4 to exit Foobar2000, Alt+Tab to the
vOICe, and exit that with Alt+F4, then Enter.

Installing opencv on Arch Linux from source

by ·

I am going to be showing you how to install the OpenCV computer vision library from source. This is the best way to install this library to ensure that you get the full set of features this library has to offer. Moreover, it allows you to customize your installation. For example, you may not want any python bindings.
This post has been inspired by the tutorials at PyImageSearch and Learn OpenCV.
The bulk of this tutorial has been taken from the superb tutorial at
Install OpenCV 4 on your Raspberry Pi

Installing dependencies

The first thing you need to do is to install dependencies. I am a newcomer to Arch Linux. I come from Ubuntu. The package names are different between Ubuntu and arch Linux. You need to use a search string like
"arch linux" in your search engine of choice to find the package names.
If you wanted to search for the package called build-essentials in Arch Linux, the search string you would use is
"Arch linux" build-essentials
The tutorial on installing OpenCV on the raspberry pi lists a series of dependencies. I have created an Arch Linux mapping to the Ubuntu dependencies and have given the equivalent commands.

Note

I am using the image created by the F123 project which as an accessible Arch Linux on the raspberry pi. I installed OpenCV on that image therefore most of my dependencies were met.
Run each of the commands and follow the prompts. If a dependency is already installed, then select not to reinstall it.

Raspbian Arch Linux
sudo apt-get install build-essential cmake unzip pkg-config yay -S build-devel
yay -S cmake
sudo apt-get install libjpeg-dev libpng-dev libtiff-dev yay -S libjpeg
yay -S libtiff
yay -S libpng
sudo apt-get install libxvidcore-dev libx264-dev yay -S xvidcore
yay -S libx264
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev yay -S ffmpeg
yay -S v4l-utils
sudo apt-get install libgtk-3-dev yay -S gtk3
sudo apt-get install libcanberra-gtk* yay -S libcanberra
sudo apt-get install libatlas-base-dev gfortran yay -S lapack
yay -S gcc-fortran
sudo apt-get install python3-dev already done because python development headers are already installed

Setting up python version 3

I am maximizing my use of python 3 which is what I will be using here. Python3 is already installed therefore, all we need to do now is to create a python virtual environment to keep things isolated.
sudo pip3 install virtualenv virtualenvwrapper
Clean the package cache.
sudo rm -rf ~/get-pip.py ~/.cache/pip
We need to enable some convenience commands to be able to redily invoke our virtual environment. Ensure that you are in your home directory.
cd ~ and then press enter.
Run the following commands to add to your .profile file.

echo -e "\n# virtualenv and virtualenvwrapper" >> ~/.profile
echo "export WORKON_HOME=$HOME/.virtualenvs" >> ~/.profile
echo "export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3" >> ~/.profile
echo "source /usr/bin/virtualenvwrapper.sh" >> ~/.profile

Once the above commands are run, you can verify that the file has been created correctly by examining its contents.
cat .profile
Activate the commands by issuing the following command.
source .profile

Download OpenCV

Ensure that you are in your home folder.
cd ~
Run the following commands to download OpenCV version 4, the latest at the time of this writing to your raspberry pi.

wget -O opencv.zip https://github.com/opencv/opencv/archive/4.0.0.zip
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.0.0.zip

Expand OpenCV and set up the folders

It is time to expand the zip archives you downloaded.

unzip opencv.zip
unzip opencv_contrib.zip

Rename the folders for easy reference.

mv opencv-4.0.0 opencv
mv opencv_contrib-4.0.0 opencv_contrib

Building OpenCV

Make the virtual environment

Run the following command to make your OpenCV virtual environment.

mkvirtualenv cv -p python3

You can name your virtual environment whatever you like. If you do name it differently, remember to account for that during the rest of this tutorial.
Switch to your virtual environment by executing the following command.

workon cv

The command prompt will change such that the name of your virtual environment will be displayed as a part of the prompt. You can exit the environment by using the following command.

deactivate

Install numpy

You need to install a dependency for OpenCV python.
Ensure that you are in your virtual environment.

workon cv


pip install numpy

The installation of numpy is going to take quite some time. Do not be alarmed.

Generate the build files

It is time to create the build files for OpenCV.
Switch to the opencv folder, create a build folder to hold the build files and then switch to it.

cd ~/opencv
mkdir build
cd build

Run cmake to generate the files. If you are copy pasting the below lines, do not manually try to delete the line breaks and create a single huge command. Copy paste each line and hit enter. The two fulstop characters at the end of the last line tell cmake that it should run.

cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
-D ENABLE_NEON=ON \
-D ENABLE_VFPV3=ON \
-D BUILD_TESTS=OFF \
-D INSTALL_PYTHON_EXAMPLES=OFF \
-D OPENCV_ENABLE_NONFREE=ON \
-D CMAKE_SIZEOF_VOID_P=4 \
-D BUILD_opencv_python3=TRUE \
-D BUILD_EXAMPLES=OFF ..

Do inspect the output and ensure that you do not see any significant errors. You may want to enable the logging facility in your terminal emulation program assuming you are running headless on your raspberry pi.

Increase the size of the swap file

The raspberry pi is a memory constrained device. You need to create a swap file to allow for the compile to complete. It is crucial that you do this else the compilation process will not complete.
In Arch Linux, you need to create a swap file first and then activate it. Issue the following commands.

sudo dd if=/dev/zero of=/swp bs=1024 count=4000000
sudo chmod 600 /swp
sudo mkswap /swp
sudo swapon /swp

See below for what these commands mean.

sudo dd if=/dev/zero of=/swp bs=1024 count=4000000

Create a file which will become our swap file. This file must be as large as the swap file we want. I have taken 4GB for safety.

sudo chmod 600 /swp

Assign the right permission to the file we have created.

sudo mkswap /swp

Convert our ordinary file to a swap file.

sudo swapon /swp

Activate the swap file.

Compile and install OpenCV

Ensure that you are in the build folder.

cd ~/opencv/build

Warning, the below command takes several hours to complete. It is the longest part of this procedure. Keep the logging in your terminal emulator active.

make

You can make things go faster by issuing the -j flag with make.

make -j

This utilizes all available CPU cores. You can specify the number of cores you want make to use.

make -j2

The above command will use only two cores. In my experience, running just make is sufficient. I have had the compilation being interrupted if I have used more than one core. Your mileage may vary.
If make completes without errors then run the following commands.

sudo make install
sudo ldconfig

OpenCV is now installed. There are however more steps to follow to ensure that the python bindings work with our virtual environment.

Turn off the swap file

It is crucial that you switch off the swap file else you risk reducing the life of the SD card in the raspberry pi due to the increased write operations that the pi will perform when it swaps memory to the SD card.

sudo swapoff /swp

You can recover space by deleting the swap file.

sudo rm /swp

Linking in the OpenCV binding into our python virtual environment

It is time to create a sim link to the OpenCV library in our python virtual environment.
Issue the following commands.

Note

I am writing this tutorial based on python 3.7. The version of python may change therefore please adjust your steps accordingly.

Run the below commands.

cd ~/.virtualenvs/cv/lib/python3.7/site-packages/
ln -s /usr/local/python/cv2/python-3.7/cv2.cpython-37m-arm-linux-gnueabihf.so cv2.so

Testing your installation

Carry out the following steps to verify if you have installed OpenCV with its python bindings correctly.

workon cv

Ensure that the virtual environment is active. If the above command fails, then do a
source .profile
to ensure that the relevant commands are loaded.
Launch python by issuing the following command
python
Then enter the following text in the interpreter that comes up.

import cv2

If the above commands without an error and your cursor moves to the next line, you can type:

cv2.__version__

You should see the version of OpenCV you have installed. Execute

quit()

to exit python.

Troubleshooting

If the make fails, ensure that your swap file is active and do not issue the -j flag.
If you get unusual errors and the compilation process goes very fast but stops in the middle, you should check the paths to your contrib folder and ensure that all other paths are right.

Coffee shops, dirty tables and Oreo shakes

by ·

It was the weekend and I was on a family outing. We drifted into a coffee shop and found a table without difficulty. My mother remarked that the table was dirty. I tilted my head by 30 degrees and sure enough I could detect some dark objects on the table. I do not know what they were and did not care to find out. Once the waiter had cleaned the table, I took another look. I got a nice clean sound with no obstructions. I am going to develop the habit of scanning every restaurant table.
It took some thinking to figure out what I was going to drink. I do not like coffee so settled for an Oreo shake. The shake came in a jar like container which was short but had a large diameter. I tilted my head down and saw this cylindrical structure. It was astonishingly dark. It appears that Oreo shakes are brown and do not allow light to pass through them. As I drank the shake, I kept glancing at the container and was able to detect an increase in light. Visual confirmation that I was getting through the shake.

getting the orca screen reader working with a raspberry pi

by ·

Many of us want to run the graphical orca screen reader on a raspberry pi, the credit card sized computer for the ultimate in portable computing experiences. This has been done before. I got this working last night and by popular demand, here are my instructions. Be warned, I went down a few rabbit and may be installing a few extra packages but at the end, orca is working.
My thanks to Michaele A. Ray of the website raspberry VI website for doing this first and showing that this is indeed possible and to the members of the raspberry-vi mailing list for help and urging me to write this post.

Prerequisits

You will need to have the following items.

  • A usb keyboard plugged into the pi for use on the pi console.
  • A pair of headphones or speakers connected to the headphone jack.
  • The raspberry pi should be powered and connected to the internet. A wired connection is the best as it is stable and easy to troubleshoot.

  1. Download the full raspbian image

    Download the latest image of raspbian from the below link. Get the full image as opposed to raspbian light. This is because the image has a lot of the graphical user interface things installed.
    download raspbian

  2. Getting the image on to the sd card

    Burn it on to a sd card using your program of choice. If you are using Microsoft windows, one of the most accessible programs is balenaEtchar

  3. Enable ssh at boot

    Once the image has been written, you will see 2 logical partitions on the sd card. There is a partition called “boot”. Create a blank file named “ssh” without the quotes on that partition. The way I do this on Microsoft windows is to go to a comma dprompt, switch to the drive that has the partition and then issue the following command.
    copy con ssh
    You will be in a line editor. Hit ctrl+z to save and close out of the file.

  4. Eject the card and load it into your pi.

  5. Boot the raspberry pi and login to it over ssh.

  6. Update the raspberry pi
    This step is crucial. Failure to do this will result in orca not speaking. Run the following commands.
    sudo apt-get update
    sudo apt-get upgrade -y
    sudo apt-get dist-upgrade -y
    Once the above commands have completed successfully, reboot the raspberry pi.

  7. Configure the pi to login to a text console

    run the following command.
    sudo raspi-config
    Read the screen output very carefully. You need to go into the boot options and enable the option to login to the console without a password.

  8. Once you have enabled the login, and are back on the main raspi-config screen, hit shift+tab once to save your settings and exit. Reboot.

  9. IInstall dependencies

    This is the trickiest part of the entire exercise.
    Orca needs a bunch of things to run. As of this writing, speech dispatcher and espeak are not installed by default.
    sudo apt-get install sox -y
    Install the sox package for multimedia libraries. Some of them may be needed by speech dispatcher.
    sudo apt-get install speech-dispatcher -y
    Install the speech dispatcher service. Orca needs to talk to speech synthesizer. Be warned, this is an older version of speech dispatcher. There is a new one available on github but I have not tried compiling it from source on this installation.
    sudo apt-get install espeak -y
    The espeak speech synthesizer
    sudo apt-get install gnome-orca -y
    The orca screen reader and associated dependencies

Once you have carried out the above steps, reboot the raspberry pi. The text console login comes in handy here. Ensure that you have a USB keyboard plugged in. You may want to ssh into the pi to ensure that the pi has booted. If you have configured the raspberry pi to automatically login into a text console, you can enter the following at the command prompt. Be warned, at this point, you do not have any speech therefore you should know your keyboard well.
startx
You will have to wait for at least 2 minutes for the gui to load. I am giving a safety margin. Once again, you do not have speech at this point. Press alt+f2. This will place you at a prompt where you can enter a command to execute.
Type the following
orca
You will need to wait yet again but you should hear orca talking.

There is much left to do. I am still working on how to start orca automatically once the gui loads and need to find a good way for this to coexist with a console screen reader such as fenrir. I will update this post as I get feedback and learn more.

Open kitchens and the wonder of food in progress

by ·

I was at MKT. The restaurant is setup differently to most such establishments. It is in the lower ground floor of the Chanakya mall and boasts a variety of cuisines. You can read about them at the above link. Visually, it is a fascinating place thanks to the live kitchens. I have spent many hours looking at food and one of the biggest problems with food is the lack of contrast. It is usually difficult to get good light and food with a high contrast as compared to your plate such that you can see the food and eat. As usual, I was wearing the vision 800 glasses running the vOICe Sacheta and I wanted to see their 4 open kitchens. Once again, I was in a situation where touching was not appropriate because the chefs were stacking food outside, ready for the waiters to pickup. I wanted to see the activity. The kitchens were behind transparent glass but they did have large windows to allow the cooks to send the food out. I was able to see the clean rising flame of the Italian kitchen and the round pizzas that emerged from it. The nachos were lined up in baskets outside the live Mexican counter. They were cylindrical but there were some differences in their shapes in stacking.
We then had a look at the Indian section where the cook was making chapatis. I could see these as dark disks lying on the counter. This was one of those situations where I used my other senses to zero on to the object of visual interest. I localized the sound of the stacking and then pointed my head in that direction. Visual scanning would have given me the same information but I treat vision as a multi sensory process.
We then moved to the Chinese counter where there was some kind of machine and a lot of activity taking place.
I did once again try to watch the cooks at work but was unable to perceive the actual motion of the people. I did notice the rapid changes in the scenery so could tell something was happening.
Sacheta took a few videos after seeking permission from the staff. That gave me the chance to stand still and look. Panning my head from side-to-side also helped and when looking through a window, pan up and down for maximum visual coverage. You can see details without panning but the devil is in the details and a little interaction brings a lot of clarity.
My special thanks to Mr. Pankaj Mishra, one of the manager’s of MKT for being so welcoming. The service is good and, before I forget, the food was excellent.
Pro food tip: do try the house special when it comes to drinks.
Pro vision tip: when your table has a lot of items on it, scan your vicinity for that tall glass of mocktail so that you can grab it in one shot. Blindness techniques work too but scanning is so much cooler.