Speech recognition on Raspberry Pi with Sphinx, Racket and Arduino

IMG_20131112_112628 copyIn this post I put together a number of things to control two LED from a Raspberry Pi with voice recognition (via Sphinx), Firmata and Arduino. Before you start, you may want to have a look at this other post on how to connect a Raspberry Pi and an Arduino board using Firmata and Racket: http://jura.mdx.ac.uk/mdxracket/.

First of all, we need to install PocketSphinx on Raspberry Pi to do speech recognition. I am using a standard USB camera with microphone (supported by Raspberry Pi) and I’m following the instructions available here: https://sites.google.com/site/observing/Home/speech-recognition-with-the-raspberry-pi. In essence, this is what I’ve done (as root on the Raspberry Pi), please see the link above for additional details:

apt-get install rpi-update
apt-get install git-core
rpi-update

-> Connect your USB microphone (or camera+mic) and
-> reboot the RPi at this point

vi /etc/modprobe.d/alsa-base.conf 

# change as follows:
# Comment this line
# options snd-usb-audio index=-2
# and add the following:
options snd-usb-audio index=0

-> close the file and reload alsa:

alsa force-reload

wget http://sourceforge.net/projects/cmusphinx/files/sphinxbase/\
0.8/sphinxbase-0.8.tar.gz/download
mv download sphinxbase-0.8.tar.gz
wget http://sourceforge.net/projects/cmusphinx/files/\
pocketsphinx/0.8/pocketsphinx-0.8.tar.gz/download
mv download pocketsphinx-0.8.tar.gz
tar -xzvf sphinxbase-0.8.tar.gz
tar -xzvf pocketsphinx-0.8.tar.gz

apt-get install bison
apt-get install libasound2-dev

cd sphinxbase-0.8
./configure --enable-fixed
make
make install

cd ../pocketsphinx-0.8/
./configure
make
sudo make install

Et voila’, you are now ready to test your PocketSphinx installation. Go to pocketsphinx-0.8/src/programs and run:

./pocketsphinx_continuous

If you are lucky, you should get some text back… I don’t have space here (and time) to go into the details of (pocket)sphinx. Try some simple words and see if they are recognised. I ended up building a very very simple language model using the on-line tool at this link: http://www.speech.cs.cmu.edu/tools/lmtool-new.html. I use just “green”, “red” and “off” and they seem to work fine in spite of my Italian accent.

In the next step we need to connect the output of PocketSphinx with Racket. I do this in a very primitive way: I modify the source code of pocketsphinx_continous to output just the word that is recognised. This is very simple: just modify continous.c under pocketsphinx-0.8/src/programs, comment all the printf statement and output just the recognised word (drop me an email if you don’t know how to do this). I append a “RACKET: ” string at the beginning of the printed string to make sure that this is something I have generated. You can then run pocketsphinx and redirect the output to a file with something like:

./pocketsphinx_continuous -lm /home/pi/sphinx/simple/4867.lm \
   -dict /home/pi/sphinx/simple/4867.dic > /tmp/capture.txt

(notice that I’m using the language model + dictionary generated on-line)

Time now to go back to Racket. I assume you know how to connect a Raspberry Pi with an Arduino board and talk to it in Racket using Firmata (if this is not the case, please have a look at the instructions available at http://jura.mdx.ac.uk/mdxracket/index.php/Raspberry_Pi,_Arduino_and_Firmata). In the following Racket file I simply read the file /tmp/capture.txt and send instructions to the board according to the instructions received. If a command is not recognised, I print a message on screen. The code for this is the following:

#lang racket
(require "firmata.rkt")
 
(define green 12)
(define red 13)
 
(define in (open-input-file "/tmp/capture.txt"))
 
(define (process-input str)
  (printf "processing input ~a\n" str)
  (set! str (substring str 8))
  (cond ( (string=? (string-upcase str) "RED")
          (printf "I'm setting red\n")
          (set-arduino-pin! red)
          )
        ( (string=? (string-upcase str) "GREEN")
          (printf "I'm setting green\n")
          (set-arduino-pin! green)
          )
        ( (string=? (string-upcase str) "OFF")
          (printf "I'm clearing the PINs\n")
          (clear-arduino-pin! red)
          (clear-arduino-pin! green)
          )
        (else
         (printf "Sorry I cannot understand: ~a\n" str)
         (flush-output)
         )
  )
  )
 
(define (read-loop)
  (define str (read-line in))
  (unless (eof-object? str)
               (process-input str)
    )
  (read-loop))
 
(define (start-everything)
  (open-firmata "/dev/ttyACM0")
  (set-pin-mode! green OUTPUT_MODE)
  (set-pin-mode! red OUTPUT_MODE)
  (read-loop)
  )
 
(start-everything)

Job done. Now check that:

  • The modified version of pocketsphinx_continuous is running and redirecting the output to /tmp/capture.txt
  • Launch the file above with something like racket sphinx-arduino.rkt
  • Check that your Arduino board is connected and wired up appropriately

and you should get something like this:

http://youtu.be/XGYNRHWY4Ag

Future work:

  • I don’t think there is a need to write to a file… maybe pocketsphinx can redirect to a port and Racket can listen to it?
  • Improve the language model for a domain of your choice
  • Add a couple of speakers to the Raspberry Pi so that Racket can tell you what it is doing, if something has not been recognised, etc.

18 thoughts on “Speech recognition on Raspberry Pi with Sphinx, Racket and Arduino

  1. Pingback: Raspberry Pi+PocketSphinx+Dimmer = Voice Dimmer | adriandubiel

  2. light

    I am thinking about using this program to have a voice remote start activation for my car ,,, i will start as soon as i get the parts will keep u osted thanks alt

    Reply
  3. Ulugbek

    Hi Franco,

    Could send me more details about how to connect Arduino and Raspberry, and how to have them communicate. I am new to this and the project I am working is to have a robot communicate and respond to simple commands.

    Thanks in advance!

    Reply
    1. franco Post author

      Have a look at this post: http://www.rmnd.net/middlesex-robotic-platformo-mirto/ and check the links at the end of the post.
      Essentially, we use Firmata. You need to install Firmata on the Arduino board and then use a client on the Raspberry Pi. We used a Racket client but there are other options available (Python, Java, etc.).
      We are working at a Firmata replacement but it is not ready yet.

      Reply
  4. Ulugbek

    Hi again,

    I also had trouble changing continuous.c. The compiler says it is missing sphinx_config.h. Here what it says: /usr/local/include/sphinxbase/ad.h|106|fatal error: sphinx_config.h: No such file or directory|
    ||=== Build failed: 1 error(s), 0 warning(s) (0 minute(s), 0 second(s)) ===|

    Can you suggest how to resolve this issue?

    Reply
  5. Ulugbek

    I don’t know, honestly. I have to learn everything on the fly. You mentioned that you “modify the source code of pocketsphinx_continous”. I thought of modifying continuous.c and recompiling it. I’m not CS major and may miss a point here or there. If you can suggest the steps I can try and maybe some tutorials I can use. I have no problem making a simple c++ project, but this seems a but above my head. I appreciate your help.

    Reply
    1. franco Post author

      Did you manage to compile pocketsphinx without modifications? Normally, this means downloading a .tar.gz file (I think I used pocketsphinx 0.8), copying the file to the Raspberry Pi, extracting with something like tar -xzvf filename.tar.gz. Then, cd pocketsphinx-* and type the commands:
      ./configure
      make
      make install
      If everything is OK and you can compile the original, then you can modify the source file and repeat the instructions make and make install above (from the same directory you used in the first place).
      Let me know if you manage to compile the “default” version. You can find more details here: http://cmusphinx.sourceforge.net/wiki/tutorialpocketsphinx (just check under “installation” and ignore the rest).

      Reply
  6. Ulugbek

    Hi Franco,

    Yes I did all if that. I was running I on Ubuntu on a pc not rawsberry. It works fine. Maybe I was trying to change a wrong file. What do you mean by the source file? Is continuous.c the one? I’ll continue my search. And hope your guidance will help. I am reading about how make works! It’s a whole new universe for but starring to make :-) more sense.

    Reply
    1. franco Post author

      Yes, continuous.c is the file you need to modify. Start with a simple change (for instance modify one of the print statements, like the one at line 371 of continuous.c, if you are using version 0.8) and see if it works.

      Reply
  7. Ulugbek

    Thanks. I’ll try tomorrow. Is there a special way to recompile it? I mean, will the recompiler “see” the .h and .o files? Or should I run it in some from of a project where all files are within the project. I use code blocks or gcc on ubuntu. Whisc is a better approach?

    Reply
    1. franco Post author

      I normally use gcc. This is invoked by make. So, after you have made your changes, just go back to the top-level directory where you have extracted pocketsphinx and call make from the command line.

      Reply
  8. charles

    what is the use of arduino for this program? can a raspberry bi work w/o an arduino specially controlling a servo motor via speech recognition?

    Reply
    1. franco Post author

      Yes, you can definitely control the Raspberry Pi GPIO pins directly. I’ve seen a Racket library to control GPIO pins from Racket, so you could use that one instead of Firmata and Arduino.

      Reply
  9. Andrew

    A very detailed and helpful tutorial, I’ve managed to follow your example to suit my own purposes. However I do have a question, is there any way that the pocketsphinx_continuous executable can be made to only run once, and return to the shell the moment the first input has been processed? I had hoped to modify the code to act as a one-time subprocess as part of a larger system. I had thought that the key might have been removing a particular while loop in continuous.c, which I did (the only while loop that didn’t loop on an input read), but I was unable to create the desired result. Is there something I’m missing?

    Regards.

    Reply
    1. Andrew

      I realized that adding a break statement after the last print statement did the trick. Never mind then!

      Reply
  10. Carsten

    Hello,
    first i have to say that i am new to the raspberry world.
    i am trying to get the cmu sphinx setup to build myself an automated desk with relays to turn on my screens and dim the light …
    First i tried to implement with your tutorial the cmu sphinx, but when i want to run it it gives me back:
    Error opening audio device default for capture: No such file or directory
    FATAL_ERROR: “continuous.c”, line 246: Failed to open audio device
    It seems that my hocked up usb webcam is not recognized. i tried it and i can take pictures with it. and i changed the alsa file.
    do you have an idea what i did wrong?

    thanx for your help in advance

    Reply

Leave a Reply to Carsten Cancel reply

Your email address will not be published. Required fields are marked *


9 × = seventy two

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>