Introduction: Create Custom Actions for Google Assistant

By platis.solutions

Seeing the potential of Google Assistant as it was presented in Google I/O 18 as well as Volvo Cars in their infotainment system, I could not resist trying it out. I reused one of my older projects, VasttraPi and interfaced it with Google Assistant. If you want to read more about the project, I would be delighted if you take a look at the related article on my blog: Custom actions for Google Assistant.

In this Instructable we are going over the steps necessary in order to launch your own Google Assistant Action which will communicate with your server and talk back to you, listing upcoming departures. Keep in mind we are not going to use the SDK since we are running this completely within Google Assistant and not our own device. Overall it is going to be kept simple. Your action will be usable on devices that support Google Assistant such as phones, tablets, home assistants and wearables and you can even share it as you would with an app on the Play Store!

One of the main advantages of creating something like this over using services such as IFTTT is that you can create custom responses in your server which will be read by Google Assistant. In IFTTT the communication is as far as I am concerned one-way.

The following topics will be covered:

  • Creating your first Action project
  • Creating your custom Intent
  • Creating your custom entity
  • Setting a webhook to fulfil your action
  • Setting up a simple REST server, written in Python with Flask, which will provide the responses or "fulfillment"
  • Testing your Action project
  • Releasing your Action project

Here you will not find many details or theory about the various Google Assistant functions and concepts. If you are interested in them, I strongly suggest you follow the official tutorial or watch this video.

Step 1: Set Your Google Account Permissions

Before we begin we need to ensure Google Assistant has the appropriate permissions.

  1. Go to Activity Controls
  2. Make sure the following are enabled:
    1. Web & App Activity
    2. Device Information
    3. Voice & Audio Activity

Step 2: Create Your Action

  1. Go to Actions Console and click on "Add/import project"
  2. Let's call this Actions project "local-traffic-planner".
  3. Click on "Create project".
  4. On the next page, do not choose a category and click on "Skip".
  5. You should be in the main Actions Console page now.

Step 3: Action Invocation

Determine how you want to trigger Google Assistant to start your action.

  1. Click on "Decide how your Action is invoked" under "Quick setup".
  2. Call it "My local traffic planner" and click on "Save".
  3. Click on the "UPDATE SAMPLE INVOCATIONS" prompt that popped up after clicking save.
    1. You can also find this option under "Directory Information".
  4. Add "Ask My local traffic planner" as a new invocation and click on "Save".

Step 4: Add Your First Action

  1. Click on "Actions" on the left-hand bar.
  2. Click on "ADD YOUR FIRST ACTION".
  3. Select "Custom intent" and then click on "Build".
  4. You will be taken to the Dialogflow page which is where you will implement the main logic.

Step 5: Training Phrases for Your Intent

  1. Choose your time zone and click "Create".
  2. On the next page, leave the existing intents be and click on "CREATE INTENT".
  3. Give the intent a reasonable name, i.e. "departures".
  4. Go to "Training phrases" and click on "Add training phrases".
  5. Use the following phrases to train your model so it can interpret what you are telling it:

    1. I am at home right now

    2. For the time being I am home

    3. I am at our apartment at the moment

    4. I am sitting at home

    5. Currently I am near work

    6. I am at work

    7. I am at the office

    8. Work

    9. Home

Step 6: Entities

Now you have specified what should be more or less expected as an input, we need to define which parts of the input are of interest to our business logic so they can be extracted and highlighted to our server. In our case, we want to know whether the user is at home or at work, so we can respond back with the departures from the specific station. Let's see how we can do that.

  1. If you double click on one or more words of the training phrases, you will get a list of predefined entities.You can read more about each one of them here. Overall, the most suitable one would be @sys.location however I think it's best and easiest if we create our own entity which we should call @current-location.
  2. Click on the "Entities" option on the left side.
  3. Click on "CREATE ENTITY".
  4. Set the name to "current-location" and define two reference values along with their synonyms:
    1. home
      1. home, house, apartment, crib
    2. work
      1. work, office, Aptiv, code mines (lol just kidding)
  5. Click on "Save. For extra effect you can also click on "Allow automated expansion" to give Google Assistant further liberties to try and match more synonyms with your reference values.

Step 7: Action and Parameters

Time to give some meaning to the keywords inside your training phrases.

  1. Click on "Intents" and then navigate to your custom Intent, i.e. "departures" if you have been following my name suggestions.
  2. Scroll down to "Training phrases".
  3. Double click on the words that indicate your current location and choose the @current-location tag from the pop up window.
  4. Scroll down to "Actions and parameters", click on "manage"
  5. If everything was done correctly, you will see your new entity being listed there.
  6. Check the "Required" box which makes a new column, "Prompts", appear. Prompts is what the user shall hear if nothing that matches the expectation has been supplied.
  7. Click on "Define prompts" and insert something like "I did not understand your location. Where are you at the moment?".

Step 8: Fulfillment

Now it is time to hook your web service to the Google Assistant Action. Your hook will be called when this specific intent is triggered and should produce the fulfillment of this action. Before that, we also want to set our intent to conclude the action after being fulfilled.

  1. Go to "Responses" and click on "ADD RESPONSE".
  2. Do not add any responses, just enable "Set this intent as end of conversation".
  3. Scroll down to "Fulfillment" click on "ENABLE FULFILLMENT" and then turn on the "Enable webhook call for this intent".
  4. Click "Save" and then go to the "Fulfillment" option on the left-hand side.
  5. Enable the "Webhook" option and insert the URL that is being "listened" by your webserver.
    1. Whenever the intent is triggered, it will send a POST request to your website with the body of the request containing a JSON object with the current location of the user.
  6. Click Save.
  7. Now we are ready to create our web service, but before that, let's make sure that our Action welcomes us in a proper manner.

Step 9: Welcome Intent

In order to customize the user experience, we should create an appropriate greeting for us whenever we trigger our action.

  1. Go to "Intents" and then click on "Default Welcome Intent".
  2. Scroll down to "Responses", remove the existing ones and insert what you want your action how to welcome you once it is initiated.
  3. Click "Save".

Step 10: Your Python Web Service

Let's make a quick and dirty Python server using Flask. No screenshots for this step, but it should be pretty straight forward.

  1. Open a new tab and create an account on pythonanywhere.com
  2. Verify your email.
  3. Set up your web application by clicking on "Open web tab".
  4. Click on "Add a new web app" and select "Flask" as your Python web framework.
  5. Select Python 3.6 and click "Next".
  6. Choose the path your want your "flask_app.py" to reside in. I placed it directly inside of my home folder as "/home/your-username/flask_app.py".
  7. Go back to the main page by clicking on the Python logo on the upper left corner.
  8. Under files, click on "flask_app.py" to start editing it.
  9. When the web text editor opens, paste the following code and click "Save".
    The overall idea is that depending on the parsed JSON coming from Google Asssistant our server will perform an action (e.g. read or write) and report it back as a response/fulfillment that should be read to the user.
  10. Click on "Open web tab" again and then on the green "Reload button".
  11. By now you should be having your own Python web server running at "https://your-username.pythonanywhere.com/departures".

Step 11: Test Your Action

OK, you are pretty much done by now. Let's test out the whole stack now and get this "Hello world" example working!

  1. Click on "Integrations" from the left-hand side bar.
  2. Click on "Integration Settings" under the Google Assistant option.
  3. Under "Implicit invocation add the name of your intent, i.e. "departures" so it can be triggered directly by saying something like "Hey Google, talk to my local traffic planner about departures from home".
  4. Enable "Auto-preview" changes.
  5. Click on "Test" which will open a new page.
  6. Type "Talk to My local traffic planner".
  7. Your action should be invoked which should be greeting your with one of the previously set welcome intent responses.
  8. Then type "I am at work". Your Python server should be contacted and the response will be read by Google Assistant.

Cool isn't it? Now imagine what you can do interacting with sensors, actuators and other APIs through your Google Assistant.

Step 12: Release Your Action

After you are done testing your Action and it is in a good state it is time to share the love with the world or, if that does not make sense, with your friends and family.

  1. Go back to your Actions console and select your local traffic planner action.
  2. Under "Get ready for deployment" click on "Enter information required for listing your Action in the Actions directory".
  3. Enter a short description, a longer description, a small logo, your contact details, a privacy policy link (if you are not going to share this publicly just put a link to your website or something) and a category.
  4. Scroll up and click on "Save".
  5. Click on the "Release" option from the left-hand side bar.
  6. Here you can choose what state your Action is in. If you do not want to hear the response "Let's get the test version of My local traffic planner" you have to make a full-fledged public release. However that requires a review by Google and will not be covered in this tutorial.
    Instead, you can still share this with up to 20 people by choosing an Alpha release and adding them as Alpha testers.
  7. Add any alpha testers by either sending them a link or adding their emails.
  8. Click on "SUBMIT FOR ALPHA", tick the boxes, click "SUBMIT" and you are done!

Now your Action is live and can be accessed by you and your friends. Have fun!

If you are interested in the code I used, take a look at the project on GitHub.