ml5.js Sound Classifier Bubble Popper
A live example of the code: https://dmarcisovska.github.io/ml5-bubble-popper/
Github code: https://github.com/dmarcisovska/ml5-bubble-popper
As I am exploring the ml5.js library, I wanted to introduce myself to the ml5.js sound classifier by creating a simple project. I created a JavaScript bubble popper a while ago, which creates a bubble upon pressing the spacebar, and deletes a bubble upon clicking it. I decided to modify the bubble popper project to use the ml5.js sound classifier because it would be a relatively easy way to play with the ml5.js sound classifier.
The ml5.js sound classifier is a pre-trained model and only recognizes these words: “the ten digits from “zero” to “nine”, “up”, “down”, “left”, “right”, “go”, “stop”, “yes”, “no”. I decided to use the words go and stop for bubble generation and popping, because these were the closest words I could think of in the set to create and pop the bubbles
HTML
To create the structure of the ml5.js sound classifier I added in some external files for p5.js, jquery, ml5.js, google fonts, bootstrap.
Within the html body, I added in some instructional text, and a card that will display words the end user is saying. I created two empty divs with ids of word and wordConfidence. These will be later used to display the word the user is saying, and the confidence level the ml5.js sound classifier returns with that word. I also added in two audio tags, to hold the two audio files that will be played when the bubble is being created or popped.
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="icon" type="image/png" href="assets/denisa-dev-purple-favicon.png">
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/p5.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/addons/p5.dom.min.js"></script>
<script src="https://unpkg.com/ml5@latest/dist/ml5.min.js" type="text/javascript"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"> </script>
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<link href="https://fonts.googleapis.com/css2?family=Permanent+Marker&display=swap" rel="stylesheet">
<link href="https://fonts.googleapis.com/css2?family=Lato:ital,wght@0,300;1,900&display=swap" rel="stylesheet">
<link rel="stylesheet" href="styles.css">
<title>Bubble Popper</title>
</head>
<body><div class="container">
<h1 class="text-center mt-5">Bubble Popper</h1>
<hr>
<p class="text-center"> Say "go" to create as many bubbles as you like. Say "stop" to pop the bubbles.</p>
<div class="card">
<div class="card-header">
<h2 class="text-center"> Sound Results</h2>
</div>
<div class="card-body text-center" id="cardBody">
You are currently saying...
<div id="word" class="text-center"> </div>
<div id="wordConfidence" class="text-center"> </div>
</div>
</div>
</div>
<audio id="bubbleSound" src="assets/bubble.wav">
<p class="text-center">If you are reading this, it is because your browser does not support the audio element. </p>
</audio>
<audio id="audio" src="assets/pop.wav">
<p class="text-center">If you are reading this, it is because your browser does not support the audio element. </p>
</audio><script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js" integrity="sha384-UO2eT0CpHqdSJQ6hJty5KVphtPhzWj9WO1clHTMGa3JDZwrnQq4sF86dIHNDz0W1" crossorigin="anonymous"></script>
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js" integrity="sha384-JjSmVgyd0p3pXB1rRibZUAYoIIy6OrQ6VrjIEaFf/nJGzIxFDsf4x0xIM+B07jRM" crossorigin="anonymous"></script>
<script src="sketch.js"></script>
</body>
</html>
CSS
In the CSS file I have added in a background image of a jellyfish. I added in some font, and hr styling. I created a bubble class which will be added to the bubble element I generate in the JavaScript file. I added in some transitions to the bubble class, so it will appear to be growing upon creation of the element. This is done with the animations and keyframes added into the css file.
body {
background: url("assets/jelly-fish.jpg") no-repeat fixed 0 0 / cover;
font-family: 'Lato', sans-serif;
color: white;
font-size: 24px;
}h1 {
font-size: 60px;
font-weight: 900;
}hr {
width: 50px;
height: .5em;
border: 0; border-top: 1px solid #ffffff;
}.bubble {
left: 0;
top: 0;
position: absolute;
height: 50px;
width: 50px;
border-radius: 50%;
border: 1px solid white;
background-color: rgba(255,255,255,0.2);
animation: bounce .5s cubic-bezier(.5, -.9, .6, 1.9) forwards;
}@keyframes bounce {
0%{
height: 50px;
}
100% {
height: 100px;
width: 100px;
}
}.card{
font-family: 'Permanent Marker', cursive;
max-width: 400px;
margin-left: auto;
margin-right: auto;
margin-top: 40px;
color: #1e043b;
font-size: 24px;
}
.card h2{
font-size: 40px;
}
JavaScript
In the JavaScript file, I created a getRandomPosition function. This function assigns a random x and y coordinate to an element within the screen view. It does this by grabbing the user’s screen height and width (minus 100). I then use the random math function to generate the x and y coordinates. For the x and y values I set the highest random value I want returned to me is the user’s height / width of the user’s screen — 100.
I then created a goResults function. If there is an error the function returns an error message. If no error is present the results are returned in an array I set as the variable results. I grab the first element in the array because that is the result with the highest confidence level. I then grab the first result label and confidence level and set those as inner text to my two blank divs I created earlier with ids of word and wordConfidence.
In order to generate a bubble if the user says “go”, I create an if statement asking if the results are equal to go. If the results are equal to go I play my bubble wave sound file. I then create a div and assign the class bubble to it. I then assign a random position to this bubble by using the getRandomPosition function. I then append the bubble to the body of my document.
In order to delete a bubble if the user says “stop”, I create an if statement asking if results are equal to stop. If they are, I grab all the elements with the class name of bubble and set that as a variable of the bubble. I then find the last bubble. If there is more than one bubble in the document, I play the popping audio and I remove the last bubble from the html.
let classifier;
const options = { probabilityThreshold: 0.7 };
let label;
let confidence;let audio = document.getElementById("audio");
let bubbleSound = document.getElementById("bubbleSound");
let word = document.getElementById("word");
let wordConfidence = document.getElementById("wordConfidence");function getRandomPosition() {
const height = document.documentElement.clientHeight - 100;
const width = document.documentElement.clientWidth - 100;
let randomHeight = Math.floor(Math.random()*height);
let randomWidth = Math.floor(Math.random()*width);
return [randomHeight,randomWidth];
}function preload() {
classifier = ml5.soundClassifier('SpeechCommands18w', options);
}function setup() {
noCanvas();
classifier.classify(gotResult);
}function gotResult(error, results) {
if (error) {
console.error(error);
}
let myResults = results[0].label;
word.innerText = "Word - " + results[0].label;
wordConfidence.innerText = 'Confidence - ' + nf(results[0].confidence, 0, 2);if (myResults === "go"){
bubbleSound.play();
let bubble = document.createElement('div');
bubble.setAttribute('class', 'bubble');
let randomBubbles = getRandomPosition(bubble);
bubble.style.top = randomBubbles[0] + 'px';
bubble.style.left = randomBubbles[1] + 'px';
document.body.appendChild(bubble);}
if (myResults === "stop"){
let bubble = document.querySelectorAll(".bubble");
let lastBubble = bubble[ bubble.length-1 ];
if (bubble.length> 0) {
audio.play();
lastBubble.parentNode.removeChild(lastBubble);
}
}
}
