PianoReader is a web tool that watches piano tutorial videos and spits out the notes and chords. It does it all in the browser without extra server compute.

There are a lot of piano “tutorial” videos out there, but most of them aren’t step-by-step walkthroughs. They mostly look like falling raindrops that light up piano keys. These tutorials are often made with a tool called Synthesia.

I’ve struggled with these videos because I can’t keep up with the falling notes or figure out the chords I need to play. It’s up to you to keep track of the light bars.

For example, I wanted to learn a Telugu song called Samayama, but it only came in video tutorial format. I couldn’t read or remember the notes fast enough to re-create it from the tutorial.

What I really needed was some kind of piano tablature that looks like this:

|Left Hand | Right Hand |
| -------- | ---------- |
| A Maj    | A D        |
| A Maj    | A          |
| D min    | D          |

How it works

I knew this problem was going to involve some level of computer vision. I roughly knew I needed some kind of classifier or bounding box detection to make this work.

Alternatively, I considered training a model. However, I didn’t want to sit and label hundreds of Synthesia videos to predict notes. I mean, if I did the work to label just one video, I wouldn’t even have to build this :)

HTML Canvas

HTML canvas is a cheap and client-first option to do low-cost video processing. No server needed, just browser.

Canvas ships with a convenient way to pull in video frames. On every frame of the video, you can hook into it with a callback:

const canvas = document.getElementById("myCanvas");
const ctx = canvas.getContext("2d");
const video = document.getElementById("video1"); // Get your video element

const drawVideoToCanvas = () => {
  // do some work here with the frame
  ctx.getImageData([]);
  // Re-register the callback to be notified about the next frame
  video.requestVideoFrameCallback(drawVideoToCanvas);
};

// Start the drawing loop when the video starts
video.requestVideoFrameCallback(drawVideoToCanvas);

Detecting Piano Keys

Before I could detect the notes pressed, I had to determine the piano key placement in a video.

Every piano in a video has different spacing, colors, and start positions. The least ambiguous way to determine the piano location was by asking the user to label two keys of the piano. I could then generate the rest of them.

I was also surprised to know that every piano key is not spaced equally. This drove me bonkers trying to generate the rest of the piano because evenly spacing the rest of the piano keys didn’t always align perfectly.

The tutorial I wanted to run only used the white keys. For simplicity, I skipped all of the sharp keys for this project. I’m sure my technique can be expanded to support those in the future.

To make the video canvas interactive, I used Fabric.js. Fabric.js is an easy way to make a drag-drop canvas. In my case, I had users position two fabric rectangles over the first keys and then generated the rest of the piano.

Detecting Notes

If the piano was a physical machine, and every key had a light underneath it, I would have used something like a photoresistor to detect if a key was pressed.

I could then place this “tracker” on every single key of the piano to get all the keys pressed at a certain frame. I used this methodology for sampling every single key.

// iterate through the visible piano keys on the video
const keysPressedInFrame = [];

for (const key of pianoKeys) {
  // get the hex code at a key location
  const hex = ctx.getImageData(x, y, w, h);
  // do some work to check if the key is lit up or not
  const isKeyPressed = determineIfKeyIsPressed(hex);
  if (isKeyPressed) keysPressedInFrame.push(key);
}

I had to figure out if a key was ON or OFF. I was able to do this by applying a grayscale filter to the canvas and checking the shade of the pressed area.

I used tonal.js to pass in the keys pressed and extract the chords that were being pressed.

const chords = tonal.getChords(keysPressedinFrame);

Final Thoughts

The good news is that I did learn the song I wanted as a result of this tool.

Honestly, I wanted the tool to do a lot more, but some of the sub-problems were pretty heavy.

Load tutorials from Youtube

Ideally, I would have liked to drop a Youtube link and have it be analyzed. However, CORS restrictions don’t allow you to do that. One limitation of HTML Canvas is the inability to pull in cross-origin videos. This causes the canvas to be tainted, so you have to download the video before you can process it.

Running the entire video

Because I have to process every frame in the video with a callback, the video has to run in its entirety to get all of the notes.

If you speed up the video so the processing can happen faster, frames get dropped, the seconds are messed up, and the song sounds weird. The best way to get the notes detected client-side is to run the video in real-time. This can mean slow processing times for longer songs.

So for now, the tool isn’t without its limitations. It only works with the white keys. It may be off by a note. Or it may need some extra tuning. But hey, it works, and I learned a song from it, at least for that one tutorial video I wanted it to.

The project is open sourced on Github for your learning or you can try it out at pianoreader.app.