L5: PaintIO Example
Table of Contents
- PaintIO Design Requirements
- PaintIO 1: Initial p5.js Application
- PaintIO 2: Adding in Arduino
- Updating PaintIO on the p5.js side to support serial
- Building the paintbrush controller
- Test and play with the initial end-to-end app!
- Upgrading our Arduino controller with OLED
- Video demonstration of initial PaintIO app
- PaintIO 3: Adding in bidirectionality
- PaintIO 4: Final PaintIO App
- Some example images
- Next Lesson
In the past few lessons, we’ve learned about serial communication, web serial, and using serial to create p5.js + Arduino applications (first lesson, second lesson). In this lesson, we’re going to build on our growing knowledge and existing code to create a full end-to-end p5.js + Arduino application, which we’ll call PaintIO. PaintIO includes a custom “paintbrush” controller with OLED display that controls and bidirectionally communicates with a custom painting application in p5.js.
Video. A quick demonstration of one PaintIO controller using the LIS3DH 3-axis accelerometer to set the paintbrush location, the paintbrush’s speed to set the color, the force-sensitive resistor to set the brush size, and three buttons for changing the brush shape, the brush fill vs. outline, and for clearing the drawing. The controller also displays current brush properties like size, shape, and location on the OLED. The Arduino code is in our GitHub as PaintIOAccel.ino. The p5.js app is here: live page, code.
A painting app is a wonderfully fertile Physical Computing example and helps culminate our learning thus far because:
- There are many different paint properties to control such as paint brush size, speed, color, shape
- We can explore and play with different types of sensors and hardware input to control these properties
- Painting is an openly creative and rich practice—there are very few rules! And our custom hardware-based paintbrush” can directly influence how we paint and what we paint. From an interaction design perspective, this is exciting and fun!
- Finally, as we’ve already experienced, making a painting application in p5.js is fairly easy (and fun)! But how can we make it even more interesting with Arduino and custom input?
Previously, we created this simple painting application in only ~20 lines of code (Impressive! Demonstrates the power of p5.js). In this app, the brush size is changed proportionally via mouse speed, the color is mapped to the mouse’s x location, and you can mouse click to switch between fill vs. outline. Play with it below!
Code. A simple p5.js painting application in ~20 lines of code. You can view, edit, and play with it here via the p5.js online editor. In this lesson, we’ll extend this example to include web serial and a custom “paintbrush” controller.
In this lesson, we will build on this example but with a custom “paintbrush” controller and different interaction mappings to set brush properties. You will learn how to modularize and slowly build up a p5.js + Arduino application, how to draw using offscreen buffers, how to use the keyboard for interaction, and how to think about and design application-level communication protocols between p5.js and Arduino.
PaintIO Design Requirements
First, let’s establish some design requirements for PaintIO. The app should:
Have bidirectional communication between the Arduino and p5.js app. While the Arduino should serve as the primary painting input, we should be able to also change settings in p5.js, which should be immediately reflected on the controller
At a minimum, the Arduino-based paintbrush controller should control paintbrush’s x,y location, size, shape, and fill mode (fill vs. outline). These properties should sound familiar—we’ll build directly on our previous lessons!
The paintbrush controller must also include the OLED to provide feedback to our painter about the paintbrush. We’re interested in dual-screen interaction and primary/secondary displays like Nintendo experimented with the Wii U.
Let’s specify how we will meet these requirements using serial communication.
From Arduino to p5.js
From the Arduino to p5.js, we will transmit a comma-separated string as:
xPosFrac, yPosFrac, sizeFrac, brushType, brushFillMode where:
xPosFracis a float between [0, 1] inclusive representing the brush’s x position
yPosFracis a float between [0, 1] inclusive representing the brush’s y position
sizeFracis a float between [0, 1] inclusive representing the brush’s size
brushTypeis either 0, 1, 2 corresponding to CIRCLE, SQUARE, TRIANGLE
brushFillModeis either 0, 1 corresponding to FILL, OUTLINE
We will also “echo back” received data for debugging purposes. We will prefix these echo backs with
# as we did in our previous lesson to indicate to the p5.js app that these lines are for debugging.
From p5.js to Arduino
As our app will be bidirectional, we will also communicate information from p5.js to Arduino. For this, we’ll build off the DisplayShapeBidirectionl example from the previous lesson and transmit a comma-separated string of
brushType, brushFillMode where, again:
brushTypeis either 0, 1, 2 corresponding to CIRCLE, SQUARE, TRIANGLE
brushFillModeis either 0, 1 corresponding to FILL, OUTLINE
PaintIO 1: Initial p5.js Application
As usual, let’s start with a basic prototype and build outwardly. First, let’s make a basic drawing p5.js application without any dependency on Arduino.
Copy SerialTemplate and create initial PaintIO structure
Begin by copying
SerialTemplate. If you’re using VSCode, copy
SerialTemplate and rename the folder to
PaintIO. If you’re using the p5.js online editor, simply open this project, Serial Template, and rename your project to
sketch.js, scroll down and remove the following. We will use a different approach to connect to serial.
Also comment out this line of code in
setup(), which attempts to automatically connect with previously approved serial devices:
We want to ignore anything serial related for now.
Add in and initialize painting variables
For the painting code, we will use similar variables and drawing code from DisplayShapeBidirectional in our previous lesson. But we’ll build this anew.
Add in the following global variables, which include the current
brushColor, and brush location (
brushY). Additionally, rather than paint directly to the canvas, we’ll use an off-screen graphics buffer called
offscreenGfxBuffer—so declare that too. We’ll talk more about that next.
Because we cannot use any p5.js constructs or functions until
setup() is called, we need to initialize
setup(). If we try to initialize them at declaration, the p5.js online editor is smart enough to catch this and hint at the problem:
Figure. You cannot use any p5.js functions or classes before
setup() has been called. If you do, you’ll likely receive an error like the above where we tried to use
color() during a global variable declaration. The specific error says: “Did you just try to use p5.js’s color() function? If so, you may want to move it into your sketch’s setup() function. For more details, see the p5.js wiki”
So, instead, initialize them in
createGraphics() function let’s us create a new offscreen graphics buffer. The function returns a new p5.Renderer object, which has the same drawing API as core p5.js. So, if we want to set the background of the offscreen buffer, we would write
offscreenGfxBuffer.background(100);. If we want to draw a red circle at pixel coordinate
10, 10 with a diameter of 50 on the offscreen buffer, we would write:
And so on. We can make the offscreen buffer any size but, in this case, we want it the same size as our canvas, so we pass the canvas
height in the
Add in painting code
draw() method, we will draw brush strokes to the offscreen buffer and then draw this buffer to canvas.
Obviously, we also need to add the
drawBrushStroke() method, which should feel familiar and understandable from previous lessons. The only difference is that we are drawing to the offscreen buffer object
Why use an offscreen buffer?
But why are we using an offscreen buffer? For simplicity and rendering speed!
In short, using an offscreen buffer let’s us render individual paint strokes once and only once and still draw stuff over it, like onscreen instructions, a crosshair cursor, etc.
It’s common to use offscreen graphics (or frame) buffers in gaming and visualization code because it let’s us draw computationally complex objects once—to a buffer—and then simply render this buffer whenever we need that object drawn again. As one example, this sound visualizer computes real-time sound processing on incoming microphone data and draws a variety of real-time visualizations, including scrolling waveforms and spectrograms—both which render sound data for a given sound sample buffer once and only once to an offscreen graphics buffer and then simply add new graphics to that buffer over time.
In our case, we could create a data structure—say a
PaintStroke class that takes in a x,y position, brush color, and all of the other brush-related properties—to store individual paint strokes in an array. For each new paint operation (i.e., each new circle, square, or triangle drawn), we would create a corresponding
PaintStroke object and store it in this array. And then, on each new
draw() call, we would iterate through these
PaintStroke objects and perform the appropriate p5.js draw operations. However, as the number of paint strokes grow, our rendering speed would decrease! And it’s inefficient to keep redrawing the same paint stroke over-and-over. So, instead, we draw each paint operation once and only once to the offscreen buffer!
Note that tracking and storing
PaintStroke objects is not mutually exclusive to using an offscreen buffer for rendering. We could still do this to support operations like undo/redo, change previously drawn strokes, etc. (and if we did this, then an undo/redo operation would cause us to loop through all
PaintStrokes and redraw them).
Add in on-screen instructions
Because we’re using this offscreen graphics buffer, it’s easy to draw a “layer” on top of the user’s painting with other graphics—in this case, user instructions. Crucially, unlike the paint strokes, we are not drawing these instructions to the offscreen buffer but rather directly onto the canvas.
Let’s go back to our
draw() function and add in the call to
drawInstructions() but only if
showInstructions is enabled. And note how this
drawInstructions() call must come after drawing the offscreen buffer to the screen. That way, it will be “layered” on top.
Hook up keyboard commands
If you carefully read the instruction code above, you may have noticed that we are going to listen for particular keyboard keys to control different properties and behaviors of our PaintIO program. We’ll use the following key mappings:
- The i key will show/hide instructions
- The l key (lowercase L) will clear the screen
- The b key will iterate through brush shapes (CIRCLE, SQUARE, TRIANGLE)
- The f key will iterate through fill mode types (FILL, OUTLINE)
We’ll implement keyboard support via the
keyPressed() method, which is called once every time a key is pressed.
To clear the screen, notice how we simply overwrite the current graphics buffer with a solid background of a given color (grayscale 100 in this case): the
A fully functional grayscale drawing app
Here’s what we have so far. A grayscale drawing app that has a fixed brush size but multiple brush types (circle, square, triangle)—selectable by the
b key—and fill types (fill, outline)—selectable by the
f key. You can also clear the screen with the
l key and show/hide the instructions with
i. Try it out below or open the code in the p5.js online editor here to view, edit, and play with the code yourself!
Code. You need to click on the gray canvas to give it “focus” in order for the keyboard commands to work. You can view, edit, and play with the code in the p5.js online editor.
PaintIO 2: Adding in Arduino
We made an initial mouse-based painting application in p5.js (woohoo!) but it lacks support for our custom “paintbrush” controller and, in fact, we haven’t even built or discussed this controller (boo!).
Continuing our simplicity and step-by-step construction themes: let’s build an initial “paintbrush” controller that provides only x, y brush location data (normalized as floats between [0, 1]). Initially, the controller will be input-only—that is, it communicates unidirectionally over serial from the Arduino to the p5.js app.
We need to update our p5.js application to support serial and to parse incoming data from our “paintbrush” controller and design and build said “paintbrush” controller in Arduino. Let’s do it!
Video. Here’s a sneak peak of the initial p5.js + Arduino app, which we will further develop in this lesson. Yes, I’m trying to write “Hi!”. It’s like an etch-a-sketch. :) This p5.js code version is available as Paint I/O 2 - Web Serial and the Arduino code is XYAnalogOut.ino on GitHub.
Updating PaintIO on the p5.js side to support serial
To update our p5.js PaintIO application to support web serial, we need to accomplish four steps:
- Add in a serial connect and open sequence
- Update our on-screen instructions to explain how to connect to serial (by hitting
- Add in **parsing code **that parses incoming paintbrush controller data
- **Update drawing code **to utilize parsed x,y brush location from serial data
Add serial connect and open sequence
In previous lessons we were capturing mouse clicks to initiate serial connections. Here, we will use the keyboard—specifically, the
o key. So, let’s update the
keyPressed() function to look for the
o key and then call
Update on-screen instructions
And add in the corresponding instructions:
Parse serial data
onSerialDataReceived() function to parse incoming data and set the variables
brushY, which hold the brush’s x,y coordinates in pixels as well as
lastBrushY, which track the previous x,y locations (similar to p5.js’s
Update drawing code to use brush x,y
Currently, we are only drawing brush strokes at the current
mouseY position. Let’s also now draw brush strokes at the current
brushY position, which are set by the Arduino-based paintbrush controller.
A keen reader will note that we are now calling
drawBrushStroke() twice: once for mouse input and once for our Arduino-based controller input. Yes, that’s true. But bimanuel interaction with two controllers has a long history in HCI (see Engelbart’s Mother of All Demos from 1968) and opens up many fruitful interaction possibilities! Later, we’ll make the mouse as a paintbrush toggleable.
And that’s it. The full code is available in the p5.js online editor as Paint I/O 2 - Web Serial.
Building the paintbrush controller
Now that we completed an initial PaintIO app with serial input support, it’s time to build the custom Arduino-based paintbrush controller. Recall that initially, we will simply transmit x,y brush location information from Arduino to p5.js. We could really use any analog input sensor we want for this but for simplicity, we will start with our handy and reliable potentiometer.
The initial paintbrush controller circuit
Figure. The initial paint brush controller with two potentiometers controlling the x- and y-location of the brush, respectively.
Initial paintbrush controller code
The Arduino code is simple: read from the two analog input pins, normalize these readings between [0, 1] (inclusive), and transmit them over serial as a comma-separated string.
Code. This code is available in our GitHub as XYAnalogOut.ino. You can also see the OLED-based variant as XYAnalogOutOLED.ino.
Test and play with the initial end-to-end app!
We did it! Now, it’s time to test and play with it.
Video. An initial PaintIO p5.js + Arduino app. We are using the two potentiometers to set the brush’s x,y location (the brush size is fixed) and the laptop keyboard to switch between brush types (
b key) and fill modes (
f key). The p5.js code is available as Paint I/O 2 - Web Serial and the Arduino code is XYAnalogOut.ino on GitHub.
Upgrading our Arduino controller with OLED
As we’re interested in exploring dual-screen interaction, let’s add in an OLED to our paintbrush controller—this way, the painter can get real-time, at-a-glance information about the paintbrush on the controller itself. For now, we will only show the paintbrush location. But we’ll add more information as our app progresses.
Basic paintbrush controller with OLED
Figure. Same two-pot circuit as before but with an OLED display.
Update code to show x,y location of brush on OLED
Now, let’s update our Arduino code to show the x,y location of the brush on the OLED. This will prove useful when we add in dynamic brush sizes and our brush is small in the p5.js app. The full code is in GitHub as XYAnalogOutOLED.ino; however, the relevant bit is simply:
Code. The full code is in our GitHub as XYAnalogOutOLED.ino.
Video demonstration of initial PaintIO app
Video. Here, we’re using the same p5.js code as before (Paint I/O 2 - Web Serial) but with an updated Arduino circuit (with OLED) and code to show brush position. Notice how as we move the brush via the two potentiometers, the brush’s position also shows on the OLED. We are also printing out the normalized x,y position on the OLED for debugging. The Arduino code is on GitHub as XYAnalogOutOLED.ino.
PaintIO 3: Adding in bidirectionality
Now that we have a basic end-to-end painting application with a custom input controller, let’s add some additional creative paint features, which will require updating both the p5.js and the Arduino apps:
- Bidirectional communication: Allow the painter to set the brush type and fill mode both in p5.js and on the paint controller.
- Support four incoming brush properties: From the Arduino to p5.js, we will receive a comma-separated string for the brush location, size, type, and fill mode:
xPosFrac, yPosFrac, sizeFrac, brushType, brushFillMode.
- Support clear screen signal: We want the painter to be able to trigger a clear screen using the paintbrush.
- Display brush properties on OLED: Currently, we only show a representation of the brush’s location on the OLED. Let’s improve this to show other brush information such as the brush type, size, and fill mode.
- Color: We could also receive explicit color information from the Arduino. For now, however, let’s simply set the color based on the brush’s size.
You could certainly add in more features like custom input hardware to control the brush color, brush opacity, outline thickness, etc. And feel free to do so! But we’ll focus on the above points for now.
Here’s a sneak peek!
Video. A sneak peek of the p5.js app Paint I/O 3 - Bidirectional with Color with the Arduino sketch PaintIO.ino.
Updating the p5.js application
As noted above, our p5.js PaintIO app should now support four incoming brush properties:
xPosFrac, yPosFrac, sizeFrac, brushType, brushFillMode, which are described in detail here. Additionally, when the user hits the
b key (to change the brush type) or the
f key (to change the fill mode), we want to communicate that information back to the Arduino so our paintbrush controller and OLED screen stays in sync.
Parsing additional brush properties and clear screen
Let’s begin by updating our parsing code to support the four incoming brush properties and the “clear screen” command. For the latter, let’s say that incoming serial data text that starts with “cls” triggers a clear screen. We need to check for this in addition to lines that start with “#”, which indicate debugging lines.
And then add in the
parseBrushData() function to parse
xPosFrac as a float between [0, 1],
yPosFrac as a float between [0, 1],
sizeFrac as a float between [0, 1],
brushType as a 0, 1, 2 corresponding to CIRCLE, SQUARE, TRIANGLE, and
brushFillMode corresponding to FILL vs. OUTLINE.
Transmit brush type and fill mode
We support changing the brush type and fill mode both through keyboard commands as well as the paintbrush controller. Consequently, we have to keep the two apps (p5.js and Arduino) in sync. Thus, when we use the keyboard to change the brush type or fill mode, we need to transmit that info over serial to the paintbrush controller.
Add in a method called
serialWriteShapeData, which is similar to what we had in our previous lesson on bidirectional serial communication.
And then call this function from
keyPressed(), when necessary:
Now, let’s add some color. We’ll explore a few different color mappings in a bit. For now, let’s simply map the hue to the brush size. The easiest way to control hue is to switch the
colorMode from the default, which is RGB, to HSB (or sometimes called HSB, for hue saturation, value). We can do this via the
colorMode(HSB) function, which also let’s us specify a max value for hue (H), saturation (S), brightness (B), and alpha (A). By default, this range is 360, 100, 100, 1, respectively, for HSB and 255, 255, 255, 255 for RGBA. For simplicity, we’ll make the max value 1 for HSBA. For more on HSB and its benefits, read this Wikipedia article.
But, in short, we’re using HSB to more easily control hue.
setup(), add the following:
Then, in draw, dynamically set the hue based on brush size:
That’s it. You can view, edit, and play with this new version of our PaintIO app here.
Update paintbrush controller
Now, we need to update the paintbrush controller to:
- Control the brush location, size, type, and fill mode
- Display brush information on the OLED
Update paintbrush circuit
For the hardware controls themselves, we again can use anything we want! To keep things simple for this example, we will use:
- As before, potentiometers on A0 and A1 to control the brush’s x and y location, respectively
- A potentiometer on A2 to control the brush’s size
- Three buttons for changing the brush shape, fill mode, and to clear the screen
Figure. A wiring diagram for the full paintbrush controller with three potentiometers on A0, A1, and A2 to control the brush’s location (x,y) and size, respectively and three buttons for changing the brush shape, fill mode, and to clear the screen. Image made in Fritzing and PowerPoint.
Update paintbrush code
For the code, we need to update our paintbrush controller to:
- Read A0, A1, and A2 for the brush’s x, y location and size and normalize the values to [0, 1]
- Read GPIO pins 4, 5, 6 to iterate brush shape and fill mode and to clear the screen
- Transmit these values over serial
- Read the serial input stream and parse out the brush shape and fill mode
- Draw brush-related info toe the OLED, including the brush’s location and size
Rather than walk through this code piece-by-piece, we will simply link to it on GitHub as PaintIO.ino. The code itself is essentially a culmination of the last few serial lessons—so it should be relatively straightforward (even if a bit lengthy). Feel free to ask questions!
Video demonstration of bidirectional PaintIO
And a quick video demonstration of everything working together!
Video. A video demonstration of the p5.js app Paint I/O 3 - Bidirectional with Color with the Arduino sketch PaintIO.ino.
PaintIO 4: Final PaintIO App
We made a few small updates to PaintIO to create a final prototype application, which includes support for:
- Turning on and off the mouse as a brush by hitting the
- Toggling the Arduino-based brush by hitting the
- And setting various color modes by hitting the
ckey, including coloring by the brush size, brush speed, brush location, and mouse location.
- Adding in an on-screen cursor in p5.js about where the paintbrush location is
The final application is on GitHub (p5.js live page, code) and Arduino sketch (PaintIOAccel.ino).
Accelerometer-based paintbrush controller
We also designed a far more fluid and interesting paintbrush controller using a 3-axis accelerometer to control the brush’s x,y location and a force-sensitive resistor to control brush size. We also switched to the ESP32 because the OLED + LIS3DH accelerometer libraries took up more memory than the Leonardo had available.
The pictorial and schematic wiring diagrams are below.
Figure. A pictorial diagram of the accelerometer-based paintbrush controller with ESP32. Made with Fritzing.
And the schematic:
Figure. A schematic diagram of the accelerometer-based paintbrush controller with ESP32. Made with Fritzing.
Video demonstration of PaintIO 4
Here’s a sneak peek of me using this new controller followed by a YouTube video overview of the whole PaintIO application and controller experience.
Video. A quick video demonstration of our new accelerometer-based Arduino controller (called PaintIOAccel.ino) and the PaintIO p5.js app (live page, code).
In the YouTube video below, we provide a full demonstration of PaintIO with the accelerometer-based paintbrush controller:
Video. A full video demonstration on YouTube.
You could (and should) design your own paintbrush controller too! Think about how to map different brush properties to sensors:
- Set the brush size or color based on microphone input—louder sounds correspond to larger brush sizes or different hues. Now, you can whistle, shout, and sing to paint!
- Set the brush color using a color sensor like this one from Adafruit
- Allow your painter to “paint over” an existing picture or video input stream (sort of like this)
- How to create a new paintbrush form that supports the artist, complements the onboard esnsors, better fits within their hands
Some example images
In the next lesson, we’ll introduce machine learning (ML) frameworks and use one in particular, called ml5.js, to create interactive ML-based applications with Arduino.
Previous: p5.js Serial I/O Next: ml5.js Serial