How to Create Digital Art With AI and Draw Robotically

This project is created as a course assignment at the California State University, Long Beach; taught by Behnaz Farahi: DESN 551: Materials, Tools, and Techniques of Prototype. This project was made in collaboration with Alison Yanacek and Zhenming Qiao.

In this project, we created images through a text-to-image-generating artificial intelligence software called Midjourney and learned to control a Universal Robots UR5 robotic arm.

Supplies

The Necessary Parts

Universal Robots UR5E Robotic Arm

The Universal Robots robotic arm is the tool we will be using to create these drawings. It is an incredibly powerful, versatile tool for multiple functions and was provided to us by the Design Department at Cal State Long Beach for the purposes of our class.

Rhino 7

Rhino 7 is powerful CAD software capable of making architectural models, as well as has a powerful node-based programming extension called Grasshopper. Grasshopper can be used to operate a Universal Robots. Grasshopper requires the Robots plugin to operate. Creating Grasshopper definitions is a specific skillset. There are plenty of tutorials out there, however. One source we recommend is Food4Rhino.

Midjourney

Midjourney is an image-generating AI. It is a relatively low-cost software embedded within Discord, with a free trial that allows other Midjourney users to see your creations publicly. (The paid version, however, is fully private).

Adobe Creative Suite

The Adobe Creative Suite is a great toolkit to have on hand to make necessary edits to images so that they play nicely with Grasshopper and the robot. There are plenty of free or low-cost tools available on the internet as well, such as GIMP and Affinity Photo.

Drawing Materials

We chose Tombo Brush Pens for this project for the variety of marks it is capable of making, as well as the color options available. However, the possibilities are limitless as long as the drawing material is modeled properly for the robot to understand its center of gravity.

We also used large-size sketching paper for testing and watercolor paper for our final piece.

Step 1: Create the Digital Version of Your Image

The process we are using can create drawn versions of any image, including ones generated from an AI like Midjourney. If you've never used Midjourney, it operates through text-to-image prompts written by the user. Essentially, you can tell the AI to imagine something (almost anything!) and it will generate images based on that text prompt.

Additionally, it has powerful revision tools. By selecting an initial intriguing composition, you can have Midjourney generate many revised iterations based on the original image. With each one, the AI learns more about what your end goal is -- and sometimes suggests its own quirky take on your request along the way. I strongly recommend using the Upscale and Remaster features several times to get a very polished composition before moving onward.

Step 2: Refine and Revise Your Image

Additionally, it has powerful revision tools. By selecting an initial intriguing composition, you can have Midjourney generate many revised iterations based on the original image. With each one, the AI learns more about what your end goal is -- and sometimes suggests its own quirky take on your request along the way. I strongly recommend using the Upscale and Remaster features several times to get a very polished composition before moving onward. In the video above, we detail our process of how one of our images developed over time through the use of Midjourney’s iterative model.

Step 3: Prepare Your Image for Grasshopper and Rhino

Grasshopper generates a topographical model of the image input into this definition based on the light/dark values of the image. Lighter areas will cause the robot to lift itself upward, away from the page. With this in mind, consider the areas that you would like the robot to draw. You can use Adobe Creative Cloud’s Illustrator and Photoshop tools to adjust the outcomes. Consider reversing the image values, or converting the image to grayscale, if the robot is not drawing what you intend for it to draw.

Step 4: Build and Test Your Grasshopper Definitions

Your Grasshopper definition will vary depending on what exactly you want to accomplish in your drawing. Ours was very straightforward – drawn on the X axis, with considerations for the height of our table and the size of our drawing surface.

Step 5: Initial Tests

We learned to control the height of the pen, as well as learn the capabilities in terms of line quantities. We recommend sending a small number of lines (around 5-20) at a time so that the robot does not get overloaded.

Step 6: Revising Your Grasshopper Definition

Grasshopper definitions for this robot are very flexible but require precision. Controlling the height of the pen, as well as the direction of the robot, will take time to adjust to get the exact results that you are aiming for. (Sometimes, millimeters at a time). Don't be afraid to take your time and run several disposable tests before using your best-quality materials. Analyze what is working in your definition and what is not, and then problem-solve and refine. This will take time! So be patient, and remain curious.

Step 7: Explore!

You can explore different pen pressures, as well as make brush strokes in X, Y, diagonal, and circular directions.

Step 8: Explore Multi-Layering

In this exercise we continued to push what was possible by channel-splitting an image created in Midjourney. We traced and modified parts of it in Illustrator, and created 5 color-coded images that we used the robot to draw on top of one another. We labeled them C.jpg, M.jpg, Y.jpg, K.jpg, and K-Type.jpg so that they would be discernable when we input them into the Grasshopper definition.