OK, now we’re really rolling! We learned about serial communication, then how to use serial in our browsers (web serial!), and then how to do this with p5.js. And we’ve already made some cool proof-of-concept demos.
Let’s take this growing knowledge and momentum to create slightly more sophisticated programs. First, we’ll cover the case of using p5.js to control something on our Arduino (Computer → Arduino) then we’ll introduce bidirectional communication (Computer ↔ Arduino) where the computer + Arduino work together to create a holistic interactive experience.
DisplayShapeOut: p5.js to Arduino
To begin, we’ll build a simple p5.js demo app that draws and resizes a selected shape (a circle, triangle, or rectangle) based on the mouse’s x position and sends this shape data as a text-encoded comma-separated string over web serial: ("shapeType, shapeSize"). On the Arduino side, we’ll parse this string and draw the current shape and size on the OLED. Because the p5.js canvas size and the OLED screen size do not match, we’ll use a normalized size value between [0,1] where 0 is the smallest size and 1 is the maximum size. Shape type is encoded as 0 for circle, 1 for square, and 2 for rectangle.
Here’s a small sneak preview of what the final interactive experience will look like.
Video. A demonstration of the p5.js app DisplayShapeOut and Arduino sketch DisplayShapeIn.ino. The p5.js app sends a shapeType and a shapeSize as a comma-separated text string to Arduino via web serial. The DisplayShapeIn.ino program parses this text and draws a shape with the appropriate size to the OLED. I did not use my regular recording setup for this because OBS Studio + my document camera have a noticeable lag. You can view, edit, play with the DisplayShapeOut code in the p5.js online editor or from our GitHub (live page, code)
Let’s go over some of DisplayShapeOut’s primary functionality. We want the user to:
Select a current shape (circle, triangle, or rectangle). We’ll do this by enumerating through them in order via a mouse press action
Change the size of the shape. We’ll do this by tracking the mouse’s x position and mapping it to size
Send shape data over serial. Each time either the current shape or size changes, we need to send an update over serial. We’ll do this using our web serial class
Draw and dynamically resize shape
We’ll build this up piece-by-piece. First, we’ll focus on the p5.js shape drawing code and then add in web serial. Let’s start by supporting a single shape type and adding in the resizing via x mouse position (similar to this part of our previous lesson).
Add in the following top-level variables:
We use the prefix const to indicate read-only variables and let for block-scoped mutable variables.
Now initialize the maxShapeSize based on the canvas width/height in setup():
Update the draw() function to actually draw our shape (a circle, for now).
Finally, we need to change curShapeSize based on the x position of the mouse:
That’s it! We made an initial interactive shape app. Save your work and try it out with VSCode’s Live Server or simply hit the play button in the p5.js editor.
Here’s a live demo from the p5.js online editor. Move your mouse over the canvas below to watch the circle change in size proportionall to the mouse’s x position.
Code. The initial code skeleton for interactively sizing a shape based on the mouse’s x position. You can view, edit, and play with the code here.
Add in support for multiple shapes
Now, let’s add in support for rendering more shapes: the square and triangle. We need a variable to track the current shape type and a method for the user to switch between shapes:
So, mapShapeTypeToShapeName defines the three shapes and their key/value relationship and curShapeType tracks the current shape as 0 (for circle), 1 (for square), and 2 (for triangle).
For selecting the shape type, there are many possibilities—we could draw small iconic representations of a circle, square, and triangle and switch shape types when these are clicked (like buttons). But we’ll do something even simpler: increment curShapeType on each mouse click.
Finally, we need to update our draw() function to draw the three shape types:
For user friendliness, let’s drop in some instructions as well. At the end of the draw() function, display some text that says “Mouse click to change the shape”:
Alright, we did it! Now check out your work by loading it with Live Server or in the p5.js online editor. Here’s a live demo:
Code. Changing shapes by mouse clicking. Code here.
Add in web serial output
Finally, the last piece is to output shape type and shape size via web serial. To limit needless serial writes, we’ll track the last shape type and size and only send out new data when these values have changed.
First, let’s add a serial write function called serialWriteShapeData(shapeType, shapeSize), which takes in a shape type and shape size and outputs them over web serial as text-encoded data.
Notably, we convert the shape size, which is in pixels, to a normalized value between [0, 1] called shapeSizeFraction—this is what we’ll transmit over serial and interpret on the Arduino side.
Now, let’s update the mouseClicked() function to handle opening and connecting with web serial or, if a connection has been made, to increment curShapeType and send that new data over serial by calling our new serialWriteShapeData() function.
Let’s also update the instructions to the user so they know that mouse clicking is state dependent:
Finally, we need to update the mouseMoved() method to call serialWriteShapeData() on a new shape size:
We could design many different types of Arduino apps that read in "shapeType, shapeSize" off serial and do something interesting. For example, we could use this info to set the paddle size and ball type (circle, square, triangle) in an OLED-based Breakout game. For this Arduino app, however, let’s simply replicate the p5.js app visual experience. This might sound hard but you’re OLED experts by now—you got this!
But how should we begin?
The key is to start simply and build up your app step-by-step, testing incrementally each step of the way.
A simple beginning and debugging strategies
Let’s begin our Arduino app simply by echo’ing the incoming data back on serial. Remember, you cannot use the Arduino IDE’s Serial Monitor once your p5.js app connects with your Arduino over serial. See error message in the figure below.
Figure. This figures shows the p5.js app DisplayShapeOut running and connected to the Arduino via web serial. Consequently, we cannot open and use the Arduino IDE’s Serial Monitor tool (Tools -> Serial Monitor) because only one program can connect to a serial port at a time. When we try, we get an error printed in the Arduino IDE console (right image) that says “Error opening serial port ‘COM5’. (Port busy)”
So, instead, let’s program our p5.js app to read incoming serial data and print it out—a web-based Serial Monitor! Luckily, our p5.js SerialTemplate code already does that. In the template, we simply have:
Which prints incoming data sent by Arduino to console and also updates the handy pHtmlMsg HTML element so you can see the info on your webpage (you could comment this out, of course).
So, the simplest Arduino sketch to start with could be an “echo back” program like:
This echo technique is a crucial debugging tool. So, make sure you understand it! We can also use the OLED display to show debugging output, which we need for this app anyway. So, let’s do that next!
A simple OLED circuit
We’ll wire up the OLED using I2C as we did in our OLED lesson. For our lesson, we’ll use the Arduino Leonardo but some of you may choose to use the Adafruit Huzzah32 (ESP32). We provide both I2C wirings below.
The Arduino Leonardo Wiring
Figure Wiring the Adafruit OLED display with I2C requires only four wires. For my wire colors, I used the standard STEMMA QT color coding: blue for data (SDA), yellow for clock (SCL), black for ground (GND), and red for the voltage supply (5V). Note that the I2C pins will differ depending on your board. For example, on the Arduino Uno, they are A4 (SDA) and A5 (SCL) rather than digital pins 2 (SDA) and 3 (SCL) as they are for the Leonardo.
The ESP32 Wiring
Figure. Wiring diagram for the Adafruit Huzzah32 ESP32 board with OLED. Note that the ESP32 has silk-printed SCL and SDA pins at the top-right corner.
Add in OLED and debug printlns
Now, let’s program the OLED to print out some debugging information. Add the following OLED-required declarations at the top:
In setup(), initialize the OLED and print out a “Waiting for serial…” message. We’ll also show the baud rate, which is a useful reminder in case you set a different value on the p5.js side.
Now, in loop(), add in the OLED-based debug printouts:
Here’s a video demonstration of what we have so far: the full DisplayShapeOut p5.js app (live page, code) running with an intermediate version of DisplayShapeSerialIn.ino, which simply echos back received data and displays some debugging output to the OLED screen.
Video. Testing the C++ echo-back code for Arduino with the the p5.js app DisplayShapeOut (live page, code). You can view this intermediate version of DisplayShapeSerialIn.ino here.
Parse serial data and update OLED debug output
So far, so good!
But now we actually need to parse the incoming serial text data into useful typed variables. Let’s do that and update our OLED-based debug output. Again, it’s useful to construct our program step-by-step testing along the way.
Update the code inside of if(Serial.available() > 0) in loop() to include parsing. There are many possible parsing approaches; however, we are going to take advantage of Arduino’s String object and functions like indexOf() and substring() to look for commas and parse out our data. We showed a similar technique in our Intro to Serial lesson.
For now, we’ll display both the raw data received over serial as well as the parsed data. Once we’re confident we have this working, we’ll remove this debug output.
Great, now let’s upload this to the Arduino and test our two apps thus far. Does the parsing work?
Video. Testing the C++ parsing code for Arduino with the the p5.js app DisplayShapeOut (live page, code). You can view this intermediate version of DisplayShapeSerialIn.ino here.
Test parsing code via Serial Monitor
Our Arduino program does not know where the incoming data from its serial port is coming from. It could be coming from any application. We can use this to our advantage for testing!
Let’s close our p5.js tab in our web browser to ensure it’s disconnected from the Arduino. Now, open up the Serial Monitor and input data into the Serial Monitor. When testing, it’s a good idea to enter both properly formed and malformed data. Make sure to test edge cases too! See video below.
Video. Using the Arduino IDE’s Serial Monitor to test our parsing code. Using the Serial Monitor is an easy, convenient way to test your serial input and parsing code on the Arduino.
Write drawing code
Awesome! We’re almost done.
Let’s pivot from reading and parsing serial input to writing our OLED-based drawing code. As we’ve previously mentioned the Adafruit GFX drawing API is not significantly different from the p5js drawing API. Note the similarities below!
First, let’s introduce some shape related types and variables:
We also need to initialize _maxShapeSize in setup():
Update our relevant parsing code to use our new global variables _curShapeType and _curShapeSizeFraction. Let’s also add in some boundary checking to ensure that the shape size fraction is between [0, 1].
Now, let’s add our drawShape(ShapeType shapeType, float fractionSize) function, which draws a circle, square, or triangle depending on the passed in shapeType at the appropriate size (fractionSize).
Finally, we need to call drawShape(), which we’ll do so at the end of loop():
DisplayShapeBidirectional: p5.js to Arduino and Arduino to p5.js
The above example demonstrated how to transmit data from p5.js to Arduino via text-encoded serial communication but did not send any commands from Arduino to p5.js. So, let’s extend our code to communicate information bidirectionally (from p5.js to Arduino and Arduino to p5.js)!
Again, there are many possibilities but let’s keep things simple. We’ll add two buttons on the Arduino side to select the shape type and a new drawing mode (fill vs. outline). We can also change these variables on the p5.js side via mouse clicks: left click to change the shape type (same as before) and right click to change the drawing mode.
Here’s a quick sneak peek at what the two apps will look like:
Notably, we are using momentary buttons for the Arduino input rather than input devices or sensors that maintain physical state like a potentiometer because fixed physical states could get out of sync with p5.js.
Updating our p5.js code
To begin, make a copy of the DisplayShapeOut p5.js folder and rename it to something like DisplayShapeBidirectional. Now, let’s add in support for the draw mode, update our parsing, and modify our instructions to the user.
Adding the fill/outline draw mode
For the fill vs. outline draw mode, we’ll add in an additional state tracking variable called curShapeDrawMode:
The draw mode can be set either by right clicking the mouse or from incoming Arduino data (from web serial). Let’s handle the former (right-mouse clicking) first.
According to the p5.js docs, the mouseClicked() function is only guaranteed to be called when the left mouse button is pressed and released. Thus, we cannot rely on this mouseClicked() for changing the draw mode. Instead, we’ll add our state tracking into mousePressed().
Notice that we also moved the shapeType tracking here too.
Add new instructions to the user
In draw(), update the instructions to the user to include info about both left-clicking and right-clicking:
Update the serialWriteShapeData function and callers
We also need to update our serialWriteShapeData() function to receive and write out three variables: shapeType, shapeSize, and shapeDrawMode instead of two as before:
And make sure to also update the serialWriteShapeData() call in mouseMoved() to use three parameters as well:
Add onSerialDataReceived parsing code
Finally, we need to add code that parses incoming serial data into shapeType, shapeSize, and shapeDrawMode. For this, we’ll add a new aspect to our Arduino → Computer communication protocol. Let’s say that any line of text transmitted with a # prefix will be ignored and considered debugging output. This way, we can continue to use our p5.js web app for output debugging while still parsing useful information.
Recall that our web serial library has an event called SerialEvents.DATA_RECEIVED, which we subscribe to in setup() and hook up a method called onSerialDataReceived(newData):
Let’s now update that onSerialDataReceived method!
Shifting now to the Arduino side. Let’s add in two buttons to our Arduino circuit: one button to iterate through shape type and another to iterate through draw modes. We’ll hook them up to GPIO pins 4 and 5 respectively with internal pull-up resistors.
Figure. The Arduino Leonardo circuit with two buttons hooked up to pins 4 and 5 using the Arduino’s internal pull-up resistors. So, by default, they are in a HIGH state and will be pulled LOW upon button press.
Add draw mode support
For the code, let’s begin by adding draw mode support:
And update our drawShape() function to accept three variables and draw the shapes accordingly (either filled or as outlines):
Add in support for buttons
Add in a new method called checkButtonPresses() that reads the two buttons, sets the global variables _curShapeType and _curDrawMode accordingly, and sends them over serial.
We can test our new button and drawing code regardless of serial input. So, let’s do that now:
Video. Testing an intermediate version of our Arduino code (in GitHub here).
Update parsing code to support draw mode
Finally, we need to update the serial parsing code to parse three comma separated values rather than just two: shapeType, shapeSizeFraction, and drawMode. Let’s move all of this serial code into its own function called checkAndParseSerial():
And now the full loop() looks like:
We did it! Below, we provide the full code links and a video demonstration.
For your prototyping journals, create a simple bidirectional app in p5.js and Arduino. Ideally, this app would correspond to an idea you have for MP3 allowing you to rapidly prototype a concept. In your journal, describe the app, link to the code (for both p5.js and Arduino), and include a brief video.
In the next lesson, we’ll bring everything together and build a fully functional paint application.