Final Project Report

What does it do?

ColloGraphy is creates an illusory presence of a calligrapher through an actuated brush and the shadow of the calligrapher's hand and arm. The ColloGraphy set is also a tool that mediates human-human interaction. It consists of a pair of networked calligraphy brushes where the main brush's orientation is translated to the secondary brush's brush tip. The follower feels the master's wrist movement in the brush, yet they still have agency over the brush. The master's movement is conveyed through the follower's arm movement.

Who's done what beforehand?

Related work consists of robotic arms that can replicate the exact motion as well as plotter machines that writes pre-determined letters. None consists of characters generated through a combination of human - and brush - actuated motion.

What did you design? What parts and systems were made?

Mechanical: Using two servos for 2-DOF movement. 3D-printed design of the brush body, casing for the hardware. Electrical: Custom PCB that (1) controls the servo motors, (2) receives orientation data from an IMU, and (3) connects to WiFi. Software: UDP communication between two microcontrollers; One of the Wifi modules is set up as an access point/server. Misc: photography, videography and the overall concept.

What materials and components were used? Where did they come from? How much did they cost?

Microservo - Amazon - $5,
Adafruit BNO055 - Amazon - $30,
ESP32-Wroom-32 modules - FabLab - ~$10?,
FTDI-USB converter - Amazon - ~$11.49 each,
Calligraphy brush set - Amazon - $15.

What processes were used?

CAD, 3D printing, PCB design and milling, Input and Output, Networking. Overall iterative design and modularize everything.

What questions were answered?

Create a brush that emulated the wrist rotation. Wireless communication between two devices.

What worked? What didn't?

Overall the wireless communication works really well and is fairly robust. Surprisingly the plastic servos held up pretty well. The prototype accomplished what it intended to do.
What did not work as well is the granularity of the brush movement. The servo is quite jitterly. Without a proper method of PID control, the servo angle could be quite jumpy, which creates more chaos.

How was it evaluated?

I have invited others to try out the networked brush. It turns out that the experience is more playful rather than for skill transfer. People feel less agency over the brush as the overall calligraphy writing process requires full attention and zero tolerance for disturbance. Nevertheless, the prototype is a tangible way to explore the kinesthetic, abstract representation of one's movement during a brush-based art creation process.

What are the implications?

I am quite happy with the prototype as it satisfied my curiosity of the feeling and process of creating such system. It exposed the pros and flaws of such design, namely the servo movement jitterness and the tradeoff between the user's agency and creating a precise replica of the brush movement. That said, I am going to continue to explore the version two prototype under this concept with a magnetically driven brush.

Original ColloGraphy Concept (Week 0)

ColloGraphy (colloquy+calligraphy) gives tangible means to interact and communicate with another person, whether a loved one or another artist, through the art of calligraphy.

While a static piece of calligraphy work itself shows great artistry, it detaches from the "embodied" process during its creation: the motion of someone's arm and hand, the pressure applied by the fingers on the brush, and the subtle change in their breathing, are expressive yet missing currently.

Collography fills the missing pieces in two ways:
(1) It renders the ghost artist's motion with a shadow that matches the characters' stroke.
(2) The brush held by the audience provides haptic feedback via the body of the brush.
The goal is not to create a surrogate or deception that the ghost artist is sentient. Rather, it is an illusory feeling of back-and-forth exchange through the loosely coupled motion of the audience and the shadow, as well as characters/message generated by the system.

For the scope of this course, I will be focusing on the hardware portion. For now, I am thinking of a way to both capture and provide feedback to the participant with a modified brush. The sensing version would capture the rotation of the brush during the movement, maybe I will also set up a global tracking system to track the global movement. For feedback, I am thinking that the brush body will provide some sort of visual or heat to the hand, and the brush tip and end will be motorized to actuate but only slightly to preserve participant's agency.

Iterative Design Process

In week 8 input week, I read out orientation data from the BNO055 IMU.
In week 9 output week, I was able to control two servos.
Just to be safe before moving everything to wireless communication, I made a wired-communication version using an Arduino to map the IMU data to the servo horn position.
In week 10 networking week, I made a wireless version of the sensor and actuator boards and got the two to communicate wirelessly.
After that I can safely work on final packaging.

Final Packaging

I spent the last two days before the final demo to polish my product. First things first, everything needs to be matte black. I used a black sharpie to meticulously color every single tiny wire including the little metal screws on the servos and the servo horn. I then spray painted the brush body and a large chunck of exposed wire black. I 3D printed a hallow brush body for the wires to go through so that they wires are out of the way. I also made a case for the BNO055 and a yin-yang inspired cover for my twin PCB boards. In this process, I accidentally shorted one set of servos so I had to make a duplicate. I also almost burned my board by reversing power and ground but thankfully everything survived and I was not subjected to the night before the final demo curse.

Video shooting and Demo prep

I really love making videos but this time it was more difficult than usual because I need to operate two brushes at once. Shout out to Ken for lending me his hand. One last final touch to the demo. This was purely experimental. Initially, I wanted to make sure the participants who are supposed to hold the "follower" brush can see the "master" hand and brush. So I setup a webcam that points at the master hand+brush, hooked it up to Quicktime, and projected my computer screen using a projector. This turned out to be a very important part that completed the magical telepresence feeling. People really saw the demo as a co-creation process of an art piece in addition to the haptic feedback, which I am super happy about. WOO!

Pretty Photos

Here are some additional photos and snap shot of the final demo :)