Person Detecting Drawing Machine

The Evil Mad Scientist Axidraw has been a to go-to gantry for generative artists for years. In this tutorial you will learn how to integrate computer vision and machine learning services into this platform to create generative art with Viam's smart machine platform.

Supplies

1 MacBook that can run `viam-server` for your axidraw

1 USB dongle

1 Axidraw (plus its stock USB cable)

Python 3

A Viam Account

Step 1: Create an Account With Viam

Navigate your browser, on the computer you are planning on using to control your Axidraw to app.viam.com and create an account.

If you are having trouble creating an account, more information is available in the Viam documentation.

Step 2: Upload a Pre-trained ML Model

Before configuring the axidraw, we will need to download the pre-trained model file effdet0.tflite and the labels.txt to your computer. Next, go to https://app.viam.com/data/models. In the “Upload model” section, name the model “effdet0" and set the model type to “object detection.” Upload the two downloaded files for “Model file” and “Label file” and click “Upload model.”

Step 3: Create a New Robot

I will be creating a new robot and calling it "Axidraw". For this tutorial, it does not matter what location your robot is in.

Once you have created your robot, the app should automatically navigate you to the "Setup" tab on the robot details page. Stay there for the next step.

Step 4: Run a Viam-server Instance on Your Mac

If not installed already, install Homebrew. You can install Homebrew by running the following command in your Terminal

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

On the Setup tab, select Mac as the Architecture. Leave the RDK Type as RDK

Follow the steps shown on the Setup tab to install viam-server on your macOS computer.

Once you have followed the steps on the Setup tab, viam-server is installed and running. Return to the Setup page on the Viam app and wait for confirmation that your computer has successfully connected.

Step 5: Installing the Python SDK

You should have the viam-server installed from the previous step, now you can install the Python SDK by running the following command in your new terminal instance:

pip install viam-sdk

You will also need to install some additional dependencies since we will be using the ML model service:

pip install 'viam-sdk[mlmodel]'

More information about running the Viam Python SDK is available in the Programing a machine documentation & Python SDK documentation.

Step 6: Configure Components on Your Axidraw

Navigate to the Config > Components section of your robot

On the bottom left of the page, click the button that says "Create component" search "Axidraw" and add the result named "gantry / eggbot:axidraw". Click "Add module". Name the component "axidraw".

Next, add a camera by clicking the "Create component" button and searching "webcam" and add the option labeled "camera / webcam". Name the component "mac-cam".' In the attributes section of the webcam component card, there should be a pre-populated dropdown with your mac's webcam as an option, select it.

If this option is not appearing, please check this your Mac is running the viam-server from step 4 and is showing up as live. Alternatively, you can find the unique ID to be used as a video path by running the following in terminal. Read more about this method in the using video_path documentation.

system_profiler SPCameraDataType

Next, add a transform, this is optional but is useful to view camera detections and ensure the model is deployed properly. Click the "Create component" button, search "transform", select the "camera / transform" option, name the component "transform" and click "create". Copy the follow code into the attributes section of the transform.

Note, the "confidence_threshold" can be used to adjust how confident the model must be to accept a detection as an input. By default this value is set to .5 which functions as 50% confident.

{
  "pipeline": [
    {
      "attributes": {
        "confidence_threshold": 0.5,
        "detector_name": "detector"
      },
      "type": "detections"
    }
  ],
  "source": "mac-cam"
}

Save the configuration after pasting in this code by clicking the "Save config" button or using the "⌘S" shor-key.

Step 7: Adding Services to Your Axidraw

Next, you will need to configure two services to ensure your ML model runs effectively. Navigate to the Config > Services tab.

First, we need to reference the ML model which you previously uploaded. Click the "Create service" button in the bottom left of the screen and search "tflite_cpu" and select the option named "ML model / TFLite CPU", name the service "model" and click "Create". In the service card, switch the deployment type to "Deploy model on robot", there should be a pre-populated dropdown with a single option of "effdet0-[date-of-upload]", select it.

Next, we will be creating a vision service detector which will actually be implementing the ML model. Click the "Create service" button, search "model" and select the option named "vision / ML model", name the service "detector". On the newly created service card, there should be a ML model dropdown with a single populated model option, select that.

Step 8: Hello World: the Code Sample Tab

Navigate to the Code Sample tab.

We will start programming with the Axidraw using boiler plate code provided under the Code Sample tabin the Viam app in the robot details page.

For this sample to work properly, you must turn on the Include API Key toggle, under step two in the instructions, before copying the code sample.

Paste and save the code from step 2 in you preferred IDE and save the file into whichever directory you will be working out of.

More in-depth explanation of this workflow can be found in the Programing a machine documentation.

Step 9: Adding the Generative Drawing

For this example, I wrote some code which produces some random lines with the starting and ending points ending in different section of the page depending on how many people are in view of the camera.

There is some additional code to convert mm to inches since the module for the axidraw was originally written to operate in inches.

Paper size may also be adjusted by editing the maxX maxY values respectively.

You will need to copy the API key from your axidraw to run properly and then save this file in the directory you will be working out of.

import asyncio
import logging
import random
import argparse
from viam.robot.client import RobotClient
from viam.rpc.dial import Credentials, DialOptions
from viam.components.gantry import Gantry
from viam.services.vision import VisionClient
import signal, sys

async def connect():
    creds = Credentials(
        type='robot-location-secret',
        payload='<<Replace-with-your-API-key-from-the-code-sample>>')
    opts = RobotClient.Options(
        refresh_interval=0,
        dial_options=DialOptions(credentials=creds)
    )
    return await RobotClient.at_address('mbp-main.8fc0qlpm4c.viam.cloud', opts)

def getValue(minVal: float, maxVal: float) -> float:
    return float(minVal + (maxVal - minVal) * random.random())


def mm2inch(l):
    return l*0.0393701

async def draw_line(axidraw: Gantry, x1, y1, x2, y2):


    pos = await axidraw.get_position()
    #lift the pen
    await axidraw.move_to_position([pos[0], pos[1], 1], [])

    # go to initial point
    await axidraw.move_to_position([x1, y1, 1], [])

    #put down the pen
    await axidraw.move_to_position([x1, y1, 0], [])

    #draw the line
    await axidraw.move_to_position([x2, y2, 0], [])

    #lift the pen
    await axidraw.move_to_position([x2, y2, 1], [])

async def main():
    robot = await connect()
    logger = logging.getLogger("axidraw")

    axidraw = Gantry.from_robot(robot, "axidraw")
    await axidraw.home()

    vision = VisionClient.from_robot(robot, "detector")

    n = 50  ##number of random lines
    X1 = []
    Y1 = []
    X2 = []
    Y2 = []
    maxX = 200 #in mm
    maxY = 120 # in mm 

    async def get_multiplier() -> int:
        detections = await vision.get_detections_from_camera("transform")
        number_of_people = 0
        for detection in detections:
            # print(detection)
            if detection.class_name == "Person" and detection.confidence > 0.7:
                number_of_people += 1
        multiplier = .1 * number_of_people
        # print(multiplier, number_of_people)
        return number_of_people

    MAX_PPL = 4
    while True:
        try:
            multiplier = await get_multiplier()
            if multiplier > MAX_PPL:
                multiplier = MAX_PPL

            X1 = random.randint(0, maxX*(multiplier+1)//MAX_PPL)
            y_min = 0 if X1 > (maxX*(multiplier)//MAX_PPL) else maxY*(multiplier)//MAX_PPL
            Y1 = random.randint(y_min, maxY*(multiplier+1)//MAX_PPL)

            X2 = random.randint(0, maxX*(multiplier+1)//MAX_PPL)
            y_min = 0 if X2 > (maxX*(multiplier)//MAX_PPL) else maxY*(multiplier)//MAX_PPL
            Y2 = random.randint(y_min, maxY*(multiplier+1)//MAX_PPL)

            print(X1, Y1, X2, Y2)
            await draw_line(axidraw, mm2inch(X1), mm2inch(Y1), mm2inch(X2), mm2inch(Y2))
        except Exception as e:
            print(e)
            await axidraw.home()
            break

    await robot.close()

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    main_task = asyncio.ensure_future(main())
    for signal in [signal.SIGINT, signal.SIGTERM]:
        loop.add_signal_handler(signal, main_task.cancel)
    try:
        loop.run_until_complete(main_task)
    finally:
        loop.close()

Step 10: Plug in Your Axidraw

Plug in the power and usb cable into the Axidraw as you would normally. The board on your Axidraw should have a solid green LED shining when powered and connected correctly

Image courtesy of the Axidraw documentation

Step 11: Connect to Your Axidraw

From your terminal, you should be able to start your viam-server with the command

viam-server -config ~/Downloads/viam-axidraw-main.json

If this is not working for you, trying navigating to the setup tab of your robot on app.viam.com and copy whatever command is in step 3.

The notification in step 4 should indicate if you have been able to connect successfully.

Step 12: Start Drawing!

From your terminal run

Make sure your terminal is in the same directory as the file you have created.

python3 "your-file-name"

Your Axidraw should be making generative art!

Step 13: Output

Depending on how many people your ML model is recognizing and how long you let the program run for, your output should look something like the above.

Step 14: Iterating on Your Approach

Note, if you change the name of any components or services when adding or removing from the configuration, you will also need to update the variable names you use in the code running on the Axidraw.

If you would prefer to train you own model you can learn how in this documentation. If you decide to create your own model, you will have to eventually change the configuration of your Axidraw the match.

Step 15: Join the Community

Share you creations and feel free to reach out to me about suggestions for this tutorial on the Discord!