Skip links

techesoterica.com

A blog dealing with sensory substitution and other esoteric concepts and technologies like speech-recognition and chaos theory

Main navigation

Sensory substitution

I visit an airport wearing the vOICe

by ·

I had to receive my cousin at the airport. Her flight was landing at 23:45 IST. I have always been concerned about the reaction of security personnel to my video glasses. This time, I was not going inside the terminal building therefore, decided to wear my Vuzix M400 device and take a chance.
I need not have worried. No one gave me a second look or if they did, that look did not translate into any conversation or interfearance
The airport was well lit and I was able to sense people moving ahead of me. Sometimes, I saw patches of relative darkness followed by patches of light and then some kind of object like a railing or a sign. This meant that people had clustered around said obstacle and then moved away.
The OCR feature came in handy in reading the various signs.
One thing to remember at least at terminal 3 of the Indira Gandhi International Airport, is that the gate numbers are written above the gates therefore you need to raise your head to look at them unless you are tall. It is also possible to tilt the lens of the Vuzix device upwards but that did not help as much as I expected it to.
There were also times that partial signs were read. In such a case, pan your head until you get something recognizable in focus. You may also need to venture closer to the sign. It is better to frame the sign in the center of your view and then move closer.
Do watch out for people and other moving obstacles like trolleys while moving.
After reading the above pages, you may well ask, what is the point of the vOICe specially when your phone can do as good a job of text reading?
For one thing, you will need your phone to coordinate with the person and to make phone calls. who you are picking.
In addition, the visual context changes almost constantly in such an environment and in case you are moved by the crowd, you will need a mechanism to reestablish visual context. This is where an immersive setup comes in handy.
In this case, I was able to see the sign of a money exchange and was able to tell my cousin where I was standing. I did have sighted assistance with me and there were multiple money exchanges so we eventually landed up also referencing an ATM which did the trick. The sign for the ATM was harder to read probably because it was in a smaller font or may be because I was a little further from it.
So, what about those Ray-ban smart glasses? They could have helped here but as of this writing, I would need to keep queering them constantly to account for the changing context. Their things around feature would have been useful but I do not know if they would have read text that was on the objects that were all around me without sending a lot of questions to the bot. I would also not have identified the changes in landscapes as we moved between the parking and the airport terminal.
I do plan to use the vOICe inside the airport next but will see when I get the opportunity to do that.

When the unthinkable happens

by ·

My father died on 17 August in hospital. I am not going to elaborate on my emotions in this post.
I wanted to experience every minute of his last physical journey. He was placed on a stretcher which was probably made of bamboo. I rode with his body in the hearse. ‘
I do have my organic memories of that ride but wanted something tangible.

Thanks to Dr. Meijer and the vOICe, I was able to see his body and the flowers that it had been draped with as we drove to the crematoria.

I now have a physical and audio / visual memory of him.

Moreover, if I want to see a photograph, I have the ability to do so.

Once again, I am grateful to Dr., Meijer’s design decision that allows the vOICe to run on virtually any modern device. I did not have my video glasses with me, all I had was my iPhone 13 mini. However, in a matter of a minute, by entering the URL to the vOICe web app, I had sight.
My father was excited about AI and what it meant for me. I am placing a soundscape of the body and am placing an unedited description generated by ChatGPT above the audio element.

The image shows a close-up of a person who appears to be in a medical setting. The individual has a tanned complexion with visible signs of aging and is resting with their eyes closed. They are wrapped in white medical fabric, similar to gauze or a thin blanket, which envelops most of their head and neck area. There is a blue pad or cloth near the top of their head, possibly a part of a hospital bed or medical equipment.
The person has a white, cotton-like material inserted into their nostrils, commonly used to stop bleeding or to keep the nostrils clear. There are a few small red marks, possibly bruise or minor wound, on the visible side of their face. Their expression is relaxed, suggesting they are unconscious or sleeping.
The image’s surroundings are out of focus and not clearly defined, emphasizing the person and their immediate condition. The color palette consists mainly of blues, whites, and skin tones.
There is no visible text in the image providing additional details or context. The user interface from an image viewer is visible, showing options for editing and information, including the file name “IMG_1921.HEIC” and the file size of “1.9 MB”. The zoom level in the viewer

Reader’s Note

The image in the soundscape is without any of the screen elements mentioned in the above description. I had opened the image in the windows 11 photo viewer and had used the ai content describer plugin to get a description of the image from ChatGPT.

Sharing in national pride thanks to the Chandrayaan- 3 landing

by ·

The Chandrayaan- 3’s landing on the moon is old news. Many of us rushed to the television to watch the live feed from ISRO.
Like most Indians, it was a proud moment for me given that landing on the south pole of the moon had not been accomplished before.
I wanted to watch a clean video and did try watching the one on youtube.
Subsequently, ISRO which I was able to watch thanks to The vOICe.
I was able to sense the change in the image as the lander got closer and closer to the lunar surface.
For best results, do the following.

  • Launch the vOICe for windows first.
  • Activate client window sonification with ctrl+f9.
  • Apply 4-fold zoom with f4.
  • Turn off high contrast if you want to. (Thanks to Dr. Meijer for this tip.)
  • Load the video and switch to full screen view.

Enjoy the extremely short video. Note the change in perspective as the lander gets closer.

For comparison, watch and or listen to the audio described video at the below link.
Described video of the Chandrayaan-3 landing
Note:
The video is in Hindi.

Samode Bagh, two is the magic number

Samode Bagh, two is the magic number

by ·

Sacheta and I recently visited Samode Bagh, a resort about 40 KM away from Jaipur. We wanted a place that was not too far by road because we needed a quick break and are not using public transport due to COVID safety concerns.
We wanted to capture landscapes and needed someplace close to nature.
Samode Bagh, if the websites were to be believed fit that bill.
The first time we tried, we had to turn back due to the farmers protests.
We reached the resort at our second attempt. I admit I did not relax until we disembarked at the resort.
The drive is easy but the last mile is a challenge because there is a right turn into a forest like area. The resort is situated on the verge of a forest though most of the forest is man made.
We had expected a flat place with nature trails, trees and nothing else. We got the trees and well manicured gardens.
The inside is not as flat as I thought. There are multiple levels to the place and a significant amount of landscaping has been done. The property is large and used to just be a garden where the finance minister of the state of Jaipur used to come to walk. There are about 40 villages that comprise the district of Samode.
A significant amount of effort has been expended in making the garden welcoming. See the below images.

A pond in the hotel with plants in the foreground

A passage with trees and plants
A walkway with plants and trees

A fountain
A fountain

A swimming pool at Samode Bagh

The lounge is nicely done.

A lounge with sofas at Samode Bagh
Lounge at Samode Bagh

Did I mention that we were at Samode on my birthday? I was in for a huge surprise on my birthday eve. No prizes for guessing who organized it.

A bonfire, see the rising flames
My birthday bon fire

It is extremely fascinating to watch flames. I can see them dance using the vOICe.

We did visit the Samode palace. Be warned, it is not much and the Bagh is way better.

A side view of samode palace with a garden
A side view of the entrance of Samode palace

Entrance to the first courtyard of Samode Palace
Entrance to the first courtyard of Samode Palace

I did something different while in the palace. I was using the vOICe and tried to describe what I was seeing. It did not quite come off but judge for yourselves.
You will also hear talkback when I start and stop the recording. I do not know why that is. I have not edited the videos.


The glass house primarily for royal women.

My compliments to the staff at Samode Bagh. They went out of their way to make us feel welcomed and accommodated every request we made. They were keen to show us the property and volunteered to activate the fountains and other facilities to allow us to experience them.

In conclusion, you can get good landscapes at Samode Bagh and the best feature of the place is that there are no animals like dogs inside. Be warned, the palace has a lot of cats and with respect, is not worth visiting.
The staff however, at both places is superb and demonstrates what hospitality means.

Future ready vision with the vOICe and the raspberry pi 4B

by ·

I recently acquired a raspberry pi 4B with 8GB of ram along with a pi camera and other sensors. My first project is to build a robust platform on which I can run the vOICe.
It is possible to run the vOICe on the phone and on smart glasses and I do both. However, I want an inexpensive solution whose components can be easily replaced. Moreover, I want something where the software is being upgraded quickly and that is mainstream.
The raspberry pi fits the bill. It is mainstream, is being upgraded and the raspberry OS is a derivative of debian which works.
Moreover, the solution is accessible.

Prerequisites

  • A raspberry pi 4B, get either the 4GB or 8GB variants.
  • A keyboard connected to the pi.
  • A pi camera that is connected to the pi. You can use a USB webcam.
  • A consistent power source
  • Headphones or powered speakers connected to the 3.5MM jack of the pi or you need audio over HDMI
  • A 32GB or greater sd card.
  • The pi must have an internet connection via wireless or a wired LAN
  • SSH must be enabled
  • Use a distribution with a desktop such as the raspberry OS full or desktop distribution

Testing if everything is working

  1. At the console type

    sudo raspi-config

    and check if audio out is set to your preferred device.
  2. Do enable the camera interface after which you may have to reboot the pi.

Testing if the camera is working

It is important to determine if the pi camera is connected correctly. The most accessible way to do this is as follows.
run the following command from within the console.

vcgencmd get_camera

On my raspberry pi, I get the following output.

pi@glasses:~ $ vcgencmd get_camera
supported=1 detected=1
pi@glasses:~ $

If you see that the detected camera is 0, then you need to check the wiring of the camera or something else. There is no point going beyond this step because you will not be able to get any output from the vOICe.

Getting the screen reader running

Raspberry OS has accessibility built-in. The problem is that the desktop needs a monitor connected before it launches. We are running the pi headless, therefore need to take additional steps to get the desktop running.

  1. Login into the pi and run the following commands.

    sudo apt update
    sudo apt upgrade -y
  2. Run the following command to determine if you can hear through your speakers and or headphones.

    aplay /usr/share/sounds/alsa /Noise.wav

    If you can here, then do not perform the next step. If you cannot, then continue with the subsequent step.
  3. Update the firmware of the pi to use a new alsa driver. Run the following program and follow the prompts.

    raspi-update
  4. Repeat the test for the sounds. If you can hear some noise, then continue. If not, troubleshoot
  5. You need to install a dummy display driver otherwise, the graphical desktop will not launch.
  6. To install the dummy display driver, run the following command.

    sudo apt install xserver-xorg-video-dummy -y
  7. You now need to add a configuration file so that the desktop uses the driver you have just installed.

    sudo nano /etc/X11/xorg.conf

    The file should contain the below entries. If the file is not there, then create a fresh file.

    Section "Monitor"
    Identifier "Monitor0"
    HorizSync 28.0-80.0
    VertRefresh 48.0-75.0
    # https://arachnoid.com/modelines/
    # 1920x1080 @ 60.00 Hz (GTF) hsync: 67.08 kHz; pclk: 172.80 MHz
    Modeline "1920x1080_60.00" 172.80 1920 2040 2248 2576 1080 1081 1084 1118 -HSync +Vsync
    EndSection
    Section "Device"
    Identifier "Card0"
    Driver "dummy"
    VideoRam 256000
    EndSection
    Section "Screen"
    DefaultDepth 24
    Identifier "Screen0"
    Device "Card0"
    Monitor "Monitor0"
    SubSection "Display"
    Depth 24
    Modes "1920x1080_60.00"
    EndSubSection
    EndSection
  8. Save and close the file. If using nano, hit ctrl+x and follow the prompts.
  9. It is important to test sound once again. If you do not hear any sound, check that audio is set to the 3.5MM jack via raspi-config
  10. Reboot the pi by issuing the following command.

    sudo reboot
  11. The graphical desktop should load and you will hear a prompt to press control plus alt plus space to install orca.
  12. Hit the ctrl+ + alt + space keystroke on the attached keyboard attached to your pi and follow the prompts. You will need to reboot once orca is installed.

At this point, you will have a pi which will speak to you. The orca screen reader will be active.

Running the vOICe

Launch chromium from the Internet menu and navigate to the following link.
The vOICe web app
Once the page loads, you will need to approve the use of the camera. Hit f6 until you reach the address bar.
Tab until you reach the button labeled “Allow” and activate it.
You should hear the vOICe sounding.

Making the solution future ready

In Linux, it is possible to share the camera. This allows me to add other solutions to the pi such as OCR and run them when the vOICe is running. I am using a virtual loop back device via the v2l4loopback kernel module. The first step is to install this module.
On raspberry OS, this involves building the module manually. This is because as of this writing, it is not present for the very latest build of the kernel.
You must first install one dependency.

sudo apt install ncurses-dev -y

Once you have carried out the above step, you are ready to get the kernel headers. Once again, there is an installation package available but it is usually not updated to the very latest kernels which is why we are following a manual approach.
Please Follow the instructions at the below page.
RPi-Distro
We are now ready to install v4l2loopback.
Please take the following steps.

  1. Ensure you are in your home directory by issuing the following command.

    cd ~
  2. We clone the repository to our pi.

    git clone https://github.com/umlaeute/v4l2loopback.git
  3. You have to build the module and add it to the kernal. Issue the following commands.

    cd v4l2loopback
    make && sudo make install
  4. If there were no errors, then run the following command.

    sudo depmod -a
  5. Test the installation by running the following command.

    modprobe v4l2loopback

    If you do not see any errors, then the module has been loaded successfully. If there are errors, then please review the installation instructions.

We now need to make the loop back devices permanent.
I have created two loop back devices but you can create as many or as few as you like.

  • We first load the module into memory.
    edit the following file by running the below command.

    sudo nano /etc/modules

    at the end of this file, add the following entry

    v4l2loopback
  • We need to specify options for v4l2 loopback. This is done By creating a file named v4l2loopback.conf in the following folder.

    /etc/modprobe.d
  • My v4l2loopback.conf file is below.

    options v4l2loopback video_nr=25,26 card_label="vOICe","ocr" exclusive_caps=1,1
  • Once you have saved this file, run the following command to commit the changes to the pi.

    sudo update-initramfs -c -k $(uname -r)

Reboot the pi and check if the right video entries have been created in the folder
Use the following command.

ls /dev/video*

On my raspberry pi, I see video25 and video26 which tells me that the loop back devices I want have been created.
You need to supply data to the loop back devices. I have done this via ffmpeg.
You first must determine which device is which. To do this, run the following command.

v4l2-ctl --list-device

Your hardware camera should be usually at /dev/video1 or at /dev/video0
Here is the output of this command when it is run on my raspberry pi.

pi@glasses:~/scripts $ v4l2-ctl --list-device
bcm2835-codec-decode (platform:bcm2835-codec):
/dev/video10
/dev/video11
/dev/video12

bcm2835-isp (platform:bcm2835-isp):
/dev/video13
/dev/video14
/dev/video15
/dev/video16

mmal service 16.1 (platform:bcm2835-v4l2-0):
/dev/video0

vOICe" (platform:v4l2loopback-025):
/dev/video25

"ocr (platform:v4l2loopback-026):
/dev/video26


This means that /dev/video0 is my camera while /dev/video25 and /dev/video26 are my loop back devices.
I need the copying process running whenever I start the pi so I am going to put it into a script. My script is below.


#!/bin/bash
ffmpeg -re -nostdin -i /dev/video0 -f v4l2 /dev/video25 -f v4l2 /dev/video26

I have called the script copy_camera_stream.sh
I have placed it in a sub directory called scripts under the pi user.
This becomes important because I will need to create a systemD unit file to run the script as a service.
Remember, once you have created the script, you must give it executable permission. From within the scripts folder, run the following command.

chmod +x ./copy_camera_stream.sh

Do test the file to ensure that there are no errors.
One way to test it is to use the screen program and run the file from within a screen session. You can then detach from the screen session, relaunch chromium and check the video source combo box of the vOICe web app to see if the video sources are working. Select the source you want to test and check if you hear the soundscapes.

It is time to create the unit file for systemD. This should be placed in the following folder.

/etc/systemd/system

The contents of my copycamerastream.service file are below.


[Unit]
Description=Run ffmpeg to copy the pi camera stream to virtual loop back devices
After=network.target

[Service]
ExecStart=/home/pi/scripts/copy_camera_stream.sh
WorkingDirectory=/home/pi/scripts
StandardOutput=inherit
StandardError=inherit
Restart=always
User=pi

[Install]
WantedBy=multi-user.target

You can launch the service and enable it

sudo service copycamerastream enable
sudo service copycamerastream start

Reboot the pi and check if the vOICe is working.
If yes, then congratulations!

Vision for under $300 or Getting the vOICe working on a raspberry pi 4

by ·

It is now possible to get the web version of the vOICe working on a raspberry pi 4. This means that it is possible to regain vision like abilities for roughly $200 or less.

What you need

In terms of hardware, you need the following things.

  • The raspberry pi 4.
  • A camera connected via USB or the pi camera.
  • All the other prerequisits mentioned in my post on Getting orca to run on the raspberry pi
  • A pair of wired headphones connected to the pi.

Checking the camera

Connect the camera to the pi and ensure it is working. If using a USB webcam, use the following command.

lsusb

The above command lists all the hardware that has been detected by the way. Your camera should be in this list. If not, you will need to use your favorite search engine or to a forum of your choice to troubleshoot.

Getting the software ready

The vOICe needs the graphical user interface to be working on the pi. As of this writing, I have not found a way to run it from the console. Please follow all the steps that I have listed in my post on getting the orca screen reader running on the raspberry pi 4.
Once you have chromium talking, please continue reading this post.
Getting the vOICe up
Do ensure that your camera is connected.

  1. Launch chromium
  2. Hit f6 if necessary and then alt+d to get to the address bar.
  3. Navigate to the web version of the vOICe
  4. You now need to tab around and allow chromium access to the camera. If you do not see this prompt, press f6 and then tab. This permission is crucial else the vOICe will be unable to access the camera.

You are done, you should hear the vOICe sounding. Move the camera around and hear the soundscapes change.

Going portable

One of the biggest strengths of this setup is that you can choose what modules you want to add. I cannot suggest what battery pack / power bank you go for. Do check the power specifications of the pi 4. You will also need to enclose the raspberry pi in a case. The case is up to you. I have used an off-the-shelf abs container for my case.

Why wired headphones

I have not tested Bluetooth headphones with my pi 4 therefore cannot comment how the soundscapes will sound. If you do test this, let me know and I will update this post.