Skip links

techesoterica.com

A blog dealing with sensory substitution and other esoteric concepts and technologies like speech-recognition and chaos theory

Main navigation

raspberry pi

Future ready vision with the vOICe and the raspberry pi 4B

by ·

I recently acquired a raspberry pi 4B with 8GB of ram along with a pi camera and other sensors. My first project is to build a robust platform on which I can run the vOICe.
It is possible to run the vOICe on the phone and on smart glasses and I do both. However, I want an inexpensive solution whose components can be easily replaced. Moreover, I want something where the software is being upgraded quickly and that is mainstream.
The raspberry pi fits the bill. It is mainstream, is being upgraded and the raspberry OS is a derivative of debian which works.
Moreover, the solution is accessible.

Prerequisites

  • A raspberry pi 4B, get either the 4GB or 8GB variants.
  • A keyboard connected to the pi.
  • A pi camera that is connected to the pi. You can use a USB webcam.
  • A consistent power source
  • Headphones or powered speakers connected to the 3.5MM jack of the pi or you need audio over HDMI
  • A 32GB or greater sd card.
  • The pi must have an internet connection via wireless or a wired LAN
  • SSH must be enabled
  • Use a distribution with a desktop such as the raspberry OS full or desktop distribution

Testing if everything is working

  1. At the console type

    sudo raspi-config

    and check if audio out is set to your preferred device.
  2. Do enable the camera interface after which you may have to reboot the pi.

Testing if the camera is working

It is important to determine if the pi camera is connected correctly. The most accessible way to do this is as follows.
run the following command from within the console.

vcgencmd get_camera

On my raspberry pi, I get the following output.

pi@glasses:~ $ vcgencmd get_camera
supported=1 detected=1
pi@glasses:~ $

If you see that the detected camera is 0, then you need to check the wiring of the camera or something else. There is no point going beyond this step because you will not be able to get any output from the vOICe.

Getting the screen reader running

Raspberry OS has accessibility built-in. The problem is that the desktop needs a monitor connected before it launches. We are running the pi headless, therefore need to take additional steps to get the desktop running.

  1. Login into the pi and run the following commands.

    sudo apt update
    sudo apt upgrade -y
  2. Run the following command to determine if you can hear through your speakers and or headphones.

    aplay /usr/share/sounds/alsa /Noise.wav

    If you can here, then do not perform the next step. If you cannot, then continue with the subsequent step.
  3. Update the firmware of the pi to use a new alsa driver. Run the following program and follow the prompts.

    raspi-update
  4. Repeat the test for the sounds. If you can hear some noise, then continue. If not, troubleshoot
  5. You need to install a dummy display driver otherwise, the graphical desktop will not launch.
  6. To install the dummy display driver, run the following command.

    sudo apt install xserver-xorg-video-dummy -y
  7. You now need to add a configuration file so that the desktop uses the driver you have just installed.

    sudo nano /etc/X11/xorg.conf

    The file should contain the below entries. If the file is not there, then create a fresh file.

    Section "Monitor"
    Identifier "Monitor0"
    HorizSync 28.0-80.0
    VertRefresh 48.0-75.0
    # https://arachnoid.com/modelines/
    # 1920x1080 @ 60.00 Hz (GTF) hsync: 67.08 kHz; pclk: 172.80 MHz
    Modeline "1920x1080_60.00" 172.80 1920 2040 2248 2576 1080 1081 1084 1118 -HSync +Vsync
    EndSection
    Section "Device"
    Identifier "Card0"
    Driver "dummy"
    VideoRam 256000
    EndSection
    Section "Screen"
    DefaultDepth 24
    Identifier "Screen0"
    Device "Card0"
    Monitor "Monitor0"
    SubSection "Display"
    Depth 24
    Modes "1920x1080_60.00"
    EndSubSection
    EndSection
  8. Save and close the file. If using nano, hit ctrl+x and follow the prompts.
  9. It is important to test sound once again. If you do not hear any sound, check that audio is set to the 3.5MM jack via raspi-config
  10. Reboot the pi by issuing the following command.

    sudo reboot
  11. The graphical desktop should load and you will hear a prompt to press control plus alt plus space to install orca.
  12. Hit the ctrl+ + alt + space keystroke on the attached keyboard attached to your pi and follow the prompts. You will need to reboot once orca is installed.

At this point, you will have a pi which will speak to you. The orca screen reader will be active.

Running the vOICe

Launch chromium from the Internet menu and navigate to the following link.
The vOICe web app
Once the page loads, you will need to approve the use of the camera. Hit f6 until you reach the address bar.
Tab until you reach the button labeled “Allow” and activate it.
You should hear the vOICe sounding.

Making the solution future ready

In Linux, it is possible to share the camera. This allows me to add other solutions to the pi such as OCR and run them when the vOICe is running. I am using a virtual loop back device via the v2l4loopback kernel module. The first step is to install this module.
On raspberry OS, this involves building the module manually. This is because as of this writing, it is not present for the very latest build of the kernel.
You must first install one dependency.

sudo apt install ncurses-dev -y

Once you have carried out the above step, you are ready to get the kernel headers. Once again, there is an installation package available but it is usually not updated to the very latest kernels which is why we are following a manual approach.
Please Follow the instructions at the below page.
RPi-Distro
We are now ready to install v4l2loopback.
Please take the following steps.

  1. Ensure you are in your home directory by issuing the following command.

    cd ~
  2. We clone the repository to our pi.

    git clone https://github.com/umlaeute/v4l2loopback.git
  3. You have to build the module and add it to the kernal. Issue the following commands.

    cd v4l2loopback
    make && sudo make install
  4. If there were no errors, then run the following command.

    sudo depmod -a
  5. Test the installation by running the following command.

    modprobe v4l2loopback

    If you do not see any errors, then the module has been loaded successfully. If there are errors, then please review the installation instructions.

We now need to make the loop back devices permanent.
I have created two loop back devices but you can create as many or as few as you like.

  • We first load the module into memory.
    edit the following file by running the below command.

    sudo nano /etc/modules

    at the end of this file, add the following entry

    v4l2loopback
  • We need to specify options for v4l2 loopback. This is done By creating a file named v4l2loopback.conf in the following folder.

    /etc/modprobe.d
  • My v4l2loopback.conf file is below.

    options v4l2loopback video_nr=25,26 card_label="vOICe","ocr" exclusive_caps=1,1
  • Once you have saved this file, run the following command to commit the changes to the pi.

    sudo update-initramfs -c -k $(uname -r)

Reboot the pi and check if the right video entries have been created in the folder
Use the following command.

ls /dev/video*

On my raspberry pi, I see video25 and video26 which tells me that the loop back devices I want have been created.
You need to supply data to the loop back devices. I have done this via ffmpeg.
You first must determine which device is which. To do this, run the following command.

v4l2-ctl --list-device

Your hardware camera should be usually at /dev/video1 or at /dev/video0
Here is the output of this command when it is run on my raspberry pi.

pi@glasses:~/scripts $ v4l2-ctl --list-device
bcm2835-codec-decode (platform:bcm2835-codec):
/dev/video10
/dev/video11
/dev/video12

bcm2835-isp (platform:bcm2835-isp):
/dev/video13
/dev/video14
/dev/video15
/dev/video16

mmal service 16.1 (platform:bcm2835-v4l2-0):
/dev/video0

vOICe" (platform:v4l2loopback-025):
/dev/video25

"ocr (platform:v4l2loopback-026):
/dev/video26


This means that /dev/video0 is my camera while /dev/video25 and /dev/video26 are my loop back devices.
I need the copying process running whenever I start the pi so I am going to put it into a script. My script is below.


#!/bin/bash
ffmpeg -re -nostdin -i /dev/video0 -f v4l2 /dev/video25 -f v4l2 /dev/video26

I have called the script copy_camera_stream.sh
I have placed it in a sub directory called scripts under the pi user.
This becomes important because I will need to create a systemD unit file to run the script as a service.
Remember, once you have created the script, you must give it executable permission. From within the scripts folder, run the following command.

chmod +x ./copy_camera_stream.sh

Do test the file to ensure that there are no errors.
One way to test it is to use the screen program and run the file from within a screen session. You can then detach from the screen session, relaunch chromium and check the video source combo box of the vOICe web app to see if the video sources are working. Select the source you want to test and check if you hear the soundscapes.

It is time to create the unit file for systemD. This should be placed in the following folder.

/etc/systemd/system

The contents of my copycamerastream.service file are below.


[Unit]
Description=Run ffmpeg to copy the pi camera stream to virtual loop back devices
After=network.target

[Service]
ExecStart=/home/pi/scripts/copy_camera_stream.sh
WorkingDirectory=/home/pi/scripts
StandardOutput=inherit
StandardError=inherit
Restart=always
User=pi

[Install]
WantedBy=multi-user.target

You can launch the service and enable it

sudo service copycamerastream enable
sudo service copycamerastream start

Reboot the pi and check if the vOICe is working.
If yes, then congratulations!

getting the orca screen reader working with a raspberry pi

by ·

Many of us want to run the graphical orca screen reader on a raspberry pi, the credit card sized computer for the ultimate in portable computing experiences. This has been done before. I got this working last night and by popular demand, here are my instructions. Be warned, I went down a few rabbit and may be installing a few extra packages but at the end, orca is working.
My thanks to Michaele A. Ray of the website raspberry VI website for doing this first and showing that this is indeed possible and to the members of the raspberry-vi mailing list for help and urging me to write this post.

Prerequisits

You will need to have the following items.

  • A usb keyboard plugged into the pi for use on the pi console.
  • A pair of headphones or speakers connected to the headphone jack.
  • The raspberry pi should be powered and connected to the internet. A wired connection is the best as it is stable and easy to troubleshoot.

  1. Download the full raspbian image

    Download the latest image of raspbian from the below link. Get the full image as opposed to raspbian light. This is because the image has a lot of the graphical user interface things installed.
    download raspbian

  2. Getting the image on to the sd card

    Burn it on to a sd card using your program of choice. If you are using Microsoft windows, one of the most accessible programs is balenaEtchar

  3. Enable ssh at boot

    Once the image has been written, you will see 2 logical partitions on the sd card. There is a partition called “boot”. Create a blank file named “ssh” without the quotes on that partition. The way I do this on Microsoft windows is to go to a comma dprompt, switch to the drive that has the partition and then issue the following command.
    copy con ssh
    You will be in a line editor. Hit ctrl+z to save and close out of the file.

  4. Eject the card and load it into your pi.

  5. Boot the raspberry pi and login to it over ssh.

  6. Update the raspberry pi
    This step is crucial. Failure to do this will result in orca not speaking. Run the following commands.
    sudo apt-get update
    sudo apt-get upgrade -y
    sudo apt-get dist-upgrade -y
    Once the above commands have completed successfully, reboot the raspberry pi.

  7. Configure the pi to login to a text console

    run the following command.
    sudo raspi-config
    Read the screen output very carefully. You need to go into the boot options and enable the option to login to the console without a password.

  8. Once you have enabled the login, and are back on the main raspi-config screen, hit shift+tab once to save your settings and exit. Reboot.

  9. IInstall dependencies

    This is the trickiest part of the entire exercise.
    Orca needs a bunch of things to run. As of this writing, speech dispatcher and espeak are not installed by default.
    sudo apt-get install sox -y
    Install the sox package for multimedia libraries. Some of them may be needed by speech dispatcher.
    sudo apt-get install speech-dispatcher -y
    Install the speech dispatcher service. Orca needs to talk to speech synthesizer. Be warned, this is an older version of speech dispatcher. There is a new one available on github but I have not tried compiling it from source on this installation.
    sudo apt-get install espeak -y
    The espeak speech synthesizer
    sudo apt-get install gnome-orca -y
    The orca screen reader and associated dependencies

Once you have carried out the above steps, reboot the raspberry pi. The text console login comes in handy here. Ensure that you have a USB keyboard plugged in. You may want to ssh into the pi to ensure that the pi has booted. If you have configured the raspberry pi to automatically login into a text console, you can enter the following at the command prompt. Be warned, at this point, you do not have any speech therefore you should know your keyboard well.
startx
You will have to wait for at least 2 minutes for the gui to load. I am giving a safety margin. Once again, you do not have speech at this point. Press alt+f2. This will place you at a prompt where you can enter a command to execute.
Type the following
orca
You will need to wait yet again but you should hear orca talking.

There is much left to do. I am still working on how to start orca automatically once the gui loads and need to find a good way for this to coexist with a console screen reader such as fenrir. I will update this post as I get feedback and learn more.

Getting fenrir talking on a raspberry pi

by ·

Fenrir is a promising new user mode screen reader. It is primarily written in python 3. Here are my instructions to install it on a raspberry pi.
I am assuming that you are using RASPBIAN JESSIE LITE and are at a terminal prompt.

Updating your installation

You must update your installation of raspbian otherwise components like espeak will not install correctly.

sudo apt-get update

Let the update finish. The above command just fetches the listing of packages that need to be updated. Now do the actual upgrade.


sudo apt-get upgrade -y

This is going to take a while to complete.
You now have to install several dependencies for fenrir to work.

Espeak

This is the speech synthesizer fenrir will use.


sudo apt-get install libespeak-dev

Python 3


sudo apt-get install python3-pip -y

Fenrir is written in python 3.

The python daemon package


sudo apt-get install python-daemon -y

A package that allows fenrir to run as a service.

evdev


sudo pip3 install -evdev

A python package that handles keyboard input.

The speech-dispatcher package


sudo apt-get install speech-dispatcher -y

The above package is required to get fenrir to talk to a speech synthesizer such as espeak.

The ConfigParser package


sudo pip3 install configparser

You may need this dependency to parse the fenrir configuration file.

A module for checking spelling


sudo apt-get install enchant -y
sudo pip3 install pyenchant

An optional package to handle spell checking.

Git


sudo apt-get install git -y

Git is a version control system which you will use to pull down the latest source code of fenrir.

It is now time to install fenrir. Execute the following command.


git clone https://github.com/chrys87/fenrir.git

You now need to start configuring the pi for fenrir to work.

Execute the following command.


sudo spd-conf

You will be asked a series of questions. The main thing you need to change is the kind of sound output technology to use. You need to use alsa if you are using the 3.5MM headphone jack of the pi like I am doing. When you are asked about using pulseaudio type “alsa” without the quotes. Hit enter to move to the next prompt after each question. Do not forget to adjust the speed and pitch to your liking.

You now need to test your speech synthesizer configuration. Do ensure that you have your headphones or speakers ready.


sudo spd-say testing

If you hear the word “testing” you are good to start fenrir. If not, look through your logs and seek support.

To start fenrir, execute the following command.

Warning: You will need to change terminals once you execute the below command so you may want to open a new terminal and keep it handy.

Assuming you are in your home directory type the following commands.


cd fenrir/src/fenrir
sudo python3 ./fenrir

You should hear a stack of startup messages which signifies that fenrir is running.

Usage

I am still playing with fenrir therefore cannot comment much. The key bindings appear to be similar to those of the speek up screen reader. If you want to check them out, take a look at the config file in your installation directory or at the following link.

fenrir keyboard config file

There is a tutorial mode available accessed with the fenrir key and h. The fenrir key is the insert key by default.

Acknowledgements

Thanks to the following people for helping me get fenrir talking and for raspberry pi advice.

Michael A. Ray
Storm Dragon
Other members on the IRC channel called #a11y at the Server: irc.netwirc.tk; specifically:
PrinceKyle, Jeremiah, chrys and southernprince.

Towards a $100 eye

by ·

Introduction

One of the regular things that I post about on this blog is the vOICe. As of this writing, the vOICe is the only way that a blind person can experience vision. For the true visual experience, the user needs to use video glasses connected to a netbook running a flavour of Microsoft Windows and the vOICe learning edition. Yes, the vOICe has been ported to other platforms such as android and Symbian.

The problem with these platforms is that they involve the use of mobile phones. Mobile phones are indeed versatile gadgets but one of the things that they cannot do is give you a permanent view from head height. It is very difficult, almost impossible to mount a mobile phone such that its camera will be positioned at the tip of your nose or at the level of the eyes. The only way to accomplish a headmounted setup today is to use USB powered video glasses with wide-angle lenses.

I and many others want to eliminate this dependency on Microsoft Windows. This is not one of those windows bashing posts. We want to get the vOICe running on as many platforms as is possible. Towards this, I, with the help of many others in the community have created a port of the vOICe that runs on Linux. We’re trying to get it to run on the raspberry pie. However, though we have a working version, it is very slow and we need help in understanding where the problem is.

One of the questions that arises is why the raspberry pie when there are better platforms available? However, the idea is to use off-the-shelf equipment and ensure that the platform to which the vOICe is ported is sustainable and is easy to acquire for new users. Today, the raspberry pie fits this bill nicely. Moreover, once the vOICe has been ported to Linux, it should be possible for any interested developers to get the source code from the seeing with sound website. A basic version of the source code already exists on that site.

Another reason we are interested in porting the vOICe to the raspberry pie and devices running operating systems other than Windows is the cost. Windows licensing is still more expensive as compared to that of other operating systems. The idea is to have an artificial eye for or below one hundred dollars. The raspberry pie is about seventy dollars. The video glasses are approximately 30 dollars. So, we’re almost there.

Areas where we need help

  • improving the performance of the vOICe.
  • Adding features to that program such as four-wheel mapping, speed control, the colour filter and if possible obstacle detection.
  • getting the drivers for the video glasses (they use the syntek chip set) to work on Linux

If you are interested in participating, either contribute to this thread on the raspberry pie forum or feel free to get in touch.

Threads where the vOICe has been discussed

Seeking help with converting opencv webcam output to pixel coordinates for integrating with sensory substitution program
Sight for the Blind for <100$

Links of interest

Drivers and script to install them for Linux
href=”http://www.seeingwithsound.com/hificode_OpenCV.cpp”>The vOICe for Linux and similar operating systems

Finally, I have uploaded some source code with this post which is the working version of the vOICe for nix like operating systems.