Introduction: AruControls: Reimagining Personalised Interactions on Your Digital Devices

What is AruControls?

AruControls is a set of physical modules that can be personalised for digital interactions using the AruControls app. It consists of 6 core modules that are made using AruCo markers, which enables the detection of a physical interaction using computer vision. When detected, the computer will respond according to keyboard presses or a mouse click, depending on what you have indicated on our app.

Keeping DIY in mind, these modules are also designed with affordance, tactile feedback, and ease of construction. More interestingly, we’ve opened up the possibilities for what physical interactions can become with the way we have coded AruControls. By coding according to marker detection methods rather than the physical module in itself, AruControls is able to support a wide variety of physical interactions. Hence, you can play with different forms and interfaces to interact with your digital device via mapping to keyboard presses.

You could use AruControls to make your own...

  • Productivity board of keyboard shortcuts for workflow efficiency
  • Simplified board to learn/teach unfamiliar computer software or programmes
  • Tangible game controllers that are typically played with computer presses
  • Embodied learning tools to complement online learning simulations
  • Interesting way of sending messages using AruCo markers
  • ... the list could go on!

In this Instructable, we share how we constructed our main board, as depicted in the picture above. Afterwhich, we also share a quick tutorial on how to use our app to match your desired keyboard presses to these physical modules.

Access the app here: arucontrols.github.io.

Step 1: An Introduction to Constructing Your Modules

As mentioned, AruControls is coded according to marker detection method rather than the physical modules themselves. The following instructions explain how to construct the 6 core modules we have designed, but can be changed to suit your own needs so long as you've selected the relevant marker detection module on our app. You could make these modules bigger, change its shape, or paste the Aruco makers on existing objects to enable enhanced interactions. We also used plywood and a laser cutter machine, but it is completely possible to use any material.

For each module, we will explain how we constructed it, as well as elaborate on the underlying marker detection method.

We have also attached the PDF and Ai file we used to print these modules, as well as a PDF of all construction instructions.

Attachments

Step 2: Button Construction

Marker Detection Method
For the button, we've utilised the presence and absence of a complete marker as a detection method. Other ways to use this marker detection method include splitting the marker into different pieces and using a gesture to piece them together, or manipulating whether the white border around the Aruco marker is present or absent.

When a marker is detected, the computer will respond according to the keyboard press you have indicated on our app.

The Button (Hold) makes use of the same detection method, but holds the keyboard press instead of just tapping it once. This is more likely to be used in the context of games.


Additional Construction Details
In our example, we used yupo paper to conceal the marker, and corrugated paper to enable the button to be pressed for detection. The corrugated paper makes the button tactile and retract on its own after a press.

Step 3: Toggle Construction

Marker Detection Method
Similarly, the toggle also utilises the presence and absence of a complete marker as a detection method. However, this marker detection method is constrained by only allowing one marker to be detected at a time.

When a marker is detected, the computer will respond according to the keyboard press you have indicated on our app.


Additional Construction Details
The way we have constructed this module is similar to the button, where we used yupo paper and corrugated paper to enable detection. The corrugated paper makes the button tactile and the form of the toggle ensures that it retracts on its own after it is pressed.

Step 4: Slider Construction

Marker Detection Method
The slider makes use of the distance travelled between two markers as a detection method. Hence, it requires 2 Aruco markers - a reference marker, as well as one that is able to move. For now, the slider only works when it is mapped to the adjustment of volume.

When the distance between the two markers are increasing, the volume will be adjusted accordingly.


Additional Construction Details
Our example makes use of corugated paper to create a sense of tactility in the slider. To do so, we include a strip of corugated paper against the plywood in the intersection of the slider.

Step 5: Dial Construction

Marker Detection Method
The dial makes use of the direction of a sequence of markers as a detection method. For example, it checks if a series of markers is detected in a clockwise or anti-clockwise direction, or if the markers are moving towards the left or right. You can map a keyboard press when the dial is moving in an anti-clockwise direction, and another keyboard press when the dial is moving in a clockwise direction - the computer will respond according to the markers' directionality. Hence, it is usually most appropriate when the action to detect these markers can be repeated for consecutive detection.

Additional Construction Details
Our example makes use of corugated paper to create a sense of tactility in the dial. To do so, we include a strip of corugated paper against the plywood in the intersection of the dial.

Step 6: Scroll Construction

Marker Detection Method
The scroll also makes use of the direction of a sequence of markers as a detection method. For example, it checks if a series of markers is detected in a clockwise or anti-clockwise direction, or if the markers are moving towards the left or right. You can map a keyboard press when the scroll is moving in an anti-clockwise direction, and another keyboard press when the scroll is moving in a clockwise direction - the computer will respond according to the markers' directionality. Hence, it is usually most appropriate when the action to detect these markers can be repeated for consecutive detection.

Scroll vs Dial
The main difference between the scroll and the dial is in the way it is constructed. The dial enables detection when the marker is placed vertically upright, while the scroll enables detection if a marker were to be placed flat horizontally.

Additional Construction Details
Our example makes use of a cardboard flap to create a sense of tactility in the scroll. The scroll also takes the form of a 'gear' so that each 'tooth' can hit the cardboard correspondingly to indicate a keypress.

Step 7: Knob Construction

Marker Detection Method
The knob makes use of the rotation of a marker as a detection method. You can map a keyboard press when the knob turns more than 45 degrees to the left, and another keyboard press when the knob turns more than 45 degrees to the right.

Additional Construction Details
Our example makes use of a rubber band to retract the knob back to its original position, as well as add tactility to the knob.

Step 8: The Board Base

All of the modules can fit on our board base, which can be found on our Ai file.

To ensure the placement of each module, the board base is made of two layers that should be placed on top of one another.

After you have constructed your board, you would need to elevate it to allow the markers to be detected. In our example, we used a stand made with acrylic and plywood that is angled at around 30 degrees. We heated the acrylic, and bent it according to the curvature of the base board. However, feel free to use any object to prop the base at an elevated level.

Step 9: The AruControls App

The AruControls app lets you pick a module with a corresponding marker detection method, and then select any desired computer keypress you wish to map to. We've suggested some relevant keyboard shortcuts in a drop-down menu, but it is possible to customise your own. After you've selected all the relevant modules for your board or interface, click on 'Confirm Modules'. You'll be be brought to a page where the computer vision is activated and you can see/test if your AruControl can be detected. There are also some settings that you can change to optimise the AruControl marker detection.

More specifically:

  • Size After Perspective Removal lets you adjust the size of the marker based on the perspective of the marker in the camera view

  • Brightness and Contrast allows u to adjust the lighting of the camera view

  • Grayscale allows you to clearly see the marker detection when the blue border appears

The app is currently in BETA and there are still additional functionality we would like to add. For now, the app is only able to support 6 physical modules.