{"id":214,"date":"2020-06-27T11:46:32","date_gmt":"2020-06-27T11:46:32","guid":{"rendered":"http:\/\/fadyanwar.com\/?p=214"},"modified":"2020-06-27T12:09:43","modified_gmt":"2020-06-27T12:09:43","slug":"building-a-raspberry-pi-ai-assistant-using-azure-and-ibm-cloud","status":"publish","type":"post","link":"https:\/\/fadyanwar.com\/index.php\/2020\/06\/27\/building-a-raspberry-pi-ai-assistant-using-azure-and-ibm-cloud\/","title":{"rendered":"Building a Raspberry Pi AI Assistant using Azure and IBM Cloud"},"content":{"rendered":"\n

As promised in my previous howto articles<\/a>, we are going to build an interactive AI assistant based on what we had learnt so far. In this article we are going to learn how to build an AI assistant using Raspberry Pi and make it interactive using Watson Assistant while giving it a voice using Azure Speech.<\/p>\n\n\n\n

In order to follow through this article you will need to set up your Raspberry Pi as described here.<\/a> You will also need to configure Azure Speech as detailed on this article.<\/a><\/p>\n\n\n\n

We are going to use Watson Assistant from IBM Cloud.<\/a> Azure do have a bot service<\/a>, however I found Watson more easier to learn not to mention it’s fun to use a Raspberry Pi in a multi cloud application. <\/p>\n\n\n\n

You will need to register for an IBM cloud, which you can do for free with only your mail under 5 minutes from here<\/a>. After registering you will get some decent free credit so you can create a Watson Assistant as detailed on this documenation.<\/a> It’s really exciting the kind of free stuff you can get as a developer these days.<\/p>\n\n\n\n

After setting up Watson Assistant, make sure that you added the dialog skill then create an intent as detailed here<\/a>. In my example below I created a simple intent called abilities which would be triggered by asking the assistant what can it do.<\/p>\n\n\n\n

When Watson is ready then it’s time to test it. Copy your assistant id, api key and endpoint in place of the placeholders in the below nodejs script.<\/p>\n\n\n

\nconst AssistantV2 = require('ibm-watson\/assistant\/v2');\nconst { IamAuthenticator } = require('ibm-watson\/auth');\nvar sessionid;\nvar assistantid = '<id>'\nconst assistant = new AssistantV2({\n  version: '2020-04-01',\n  authenticator: new IamAuthenticator({\n    apikey: '<apikey>', \/\/place your api key here\n  }),\n  url: '<endpoint>', \/\/place your endpoint url here\n});\n\n\/\/first create an assistant session to be used later\nassistant.createSession({\n    assistantId: assistantid\n  })\n    .then(res => {\n      console.log(JSON.stringify(res.result, null, 2));\n      sessionid = res.result["session_id"];\n      sendMessage("hello"); \/\/when session is ready send a hello message\n      \n    })\n    .catch(err => {\n      console.log(err);\n    });\n\n\nfunction sendMessage(message)\n{\n    assistant.message({\n      assistantId: assistantid,\n      sessionId: sessionid,\n      input: {\n        'message_type': 'text',\n        'text': message\n        }\n      })\n      .then(res => {\n        return res.result.output.generic[0]["text"];        \n      })\n      .catch(err => {\n        console.log(err);\n      });\n\n}\n<\/pre><\/div>\n\n\n
In order to run this script you will need first to install the required npm package as following<\/p>\n\n\n\n
npm install ibm-watson@^5.6.0<\/code><\/p>\n\n\n\n
Save the script to something like chat.js and then execute by typing the below command<\/p>\n\n\n\n
node chat.js<\/code><\/p>\n\n\n\n
You should get a json response with the session id and whatever response you had configured Watson to respond to a generic greeting. Which means Watson Assistant service is ready.<\/p>\n\n\n\n
Now it’s time put things together. You will now use Azure Speech to translate speech to text, send it over to Watson Assistant IBM cloud service, get the text response back then convert it to human like speech using Azure Speech again.<\/p>\n\n\n\n
In order to do so, use the below code snippet and replace the placeholders with Azure Speech and Watson Assistant details as per the comments in the below code.<\/p>\n\n\n
\n\/\/ Import required npm packages\nconst AudioRecorder = require('node-audiorecorder');\nconst fs = require('fs');\nvar sdk = require("microsoft-cognitiveservices-speech-sdk");\nvar player = require('play-sound')(opts = {})\n\n\nvar subscriptionKey = "<key>"; \/\/replace with your azure speech service key\nvar serviceRegion = "northeurope"; \/\/ e.g., "westus"\nvar filename = "message.wav"; \/\/ 16000 Hz, Mono\n\n\nconst AssistantV2 = require('ibm-watson\/assistant\/v2');\nconst { IamAuthenticator } = require('ibm-watson\/auth');\nvar sessionid;\nvar assistantid = '<id>'\/\/replace with assistant id\nconst assistant = new AssistantV2({\n  version: '2020-04-01',\n  authenticator: new IamAuthenticator({\n    apikey: '<key>', \/\/replace with watson api key\n  }),\n  url: '<url>', \/\/replace with watson endpoint url\n});\n\n\nassistant.createSession({\n    assistantId: assistantid\n  })\n    .then(res => {\n      console.log(JSON.stringify(res.result, null, 2));\n      sessionid = res.result["session_id"];\n      \n    })\n    .catch(err => {\n      console.log(err);\n    });\n\n\n\/\/ Create an instance.\nconst audioRecorder = new AudioRecorder({\n    program: process.platform = 'sox', \n  }, console);\n\n\/\/ Create write stream.\nconst fileStream = fs.createWriteStream(filename, { encoding: 'binary' });\n\/\/ Start and write to the file.\naudioRecorder.start().stream().pipe(fileStream);\nsetTimeout(recognize, 5000); \/\/wait for 5 seconds then call recognize function\n\nfunction recognize()\n{\n    audioRecorder.stop();\n    \n    \/\/ create the push stream we need for the speech sdk.\n    var pushStream = sdk.AudioInputStream.createPushStream();\n\n    \/\/ open the file and push it to the push stream.\n    fs.createReadStream(filename).on('data', function(arrayBuffer) {\n        pushStream.write(arrayBuffer.slice());\n    }).on('end', function() {\n        pushStream.close();\n    });\n\n    \/\/ we are done with the setup\n    console.log("Now recognizing from: " + filename);\n\n    \/\/ now create the audio-config pointing to our stream and\n    \/\/ the speech config specifying the language.\n    var audioConfig = sdk.AudioConfig.fromStreamInput(pushStream);\n    var speechConfig = sdk.SpeechConfig.fromSubscription(subscriptionKey, serviceRegion);\n\n    \/\/ setting the recognition language to English.\n    speechConfig.speechRecognitionLanguage = "en-US";\n\n    \/\/ create the speech recognizer.\n    var recognizer = new sdk.SpeechRecognizer(speechConfig, audioConfig);\n\n    \/\/ start the recognizer and wait for a result.\n    recognizer.recognizeOnceAsync(\n        function (result) {\n        console.log(result);\n\n        sendMessage(result["privText"]);\n        \n        recognizer.close();\n        recognizer = undefined;\n        },\n        function (err) {\n        console.trace("err - " + err);\n\n        recognizer.close();\n        recognizer = undefined;\n        });\n\n}\n\n\nfunction sendMessage(message)\n{\n    assistant.message({\n      assistantId: assistantid,\n      sessionId: sessionid,\n      input: {\n        'message_type': 'text',\n        'text': message\n        }\n      })\n      .then(res => {\n        say(res.result.output.generic[0]["text"]);        \n      })\n      .catch(err => {\n        console.log(err);\n      });\n\n}\n\nfunction say(text){\n\n\n  \/\/ create the pull stream we need for the speech sdk.\n  var pullStream = sdk.AudioOutputStream.createPullStream();\n\n\n  \/\/ now create the audio-config pointing to our stream and\n  \/\/ the speech config specifying the language.\n  var speechConfig = sdk.SpeechConfig.fromSubscription(subscriptionKey, serviceRegion);\n\n  \/\/ setting the recognition language to English.\n  speechConfig.speechRecognitionLanguage = "en-US";\n\n  var audioConfig = sdk.AudioConfig.fromStreamOutput(pullStream);\n\n  \/\/ create the speech synthesizer.\n  var synthesizer = new sdk.SpeechSynthesizer(speechConfig, audioConfig);\n\n\n  synthesizer.speakTextAsync(\n    text,\n    function (result) {\n      console.log(result);\n\n      \/\/when the wav file is ready, play it\n      player.play(filename, function(err){\n          if (err) throw err\n        })\n          \n      synthesizer.close();\n      synthesizer = undefined;\n    },\n    function (err) {\n      console.trace("err - " + err);\n\n      synthesizer.close();\n      synthesizer = undefined;\n    },\n    filename);\n}\n\n\n<\/pre><\/div>\n\n\n
When having both Azure Speech and Watson Assistant ready test first on your machine, you should get something similar as shown on the below video.<\/p>\n\n\n\n
\n