TensorFlow.js is a library for machine learning in JavaScript. Using TensorFlow.js one can,

• Run existing model
• Develop ML models in JavaScript
• Use existing model and retrain it

Here in this blog post we will see first use case which is to run existing model. We will use MobileNet model to predict object through webcam.

TensorFlow.js Image Classification HTML

Let’s start it with basic HTML and add necessary scripts in it. First of all create one index.html file. Add following code in that file. Also create css folder and tfjs folder. Here we will out our style.css and our JS file to start image classification

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Image Classification</title>
    <link rel="stylesheet" href="css/style.css">
  </head>
  <body>
    <button id="webCam" class="btn webCam-button" onclick="openWebCam()">Start Web Cam</button>
    <div class="webcam-popup" id="webCamDisplay">
      <video autoplay playsinline muted id="webcam" width="224" height="224"></video>
      <button type="button" class="btn" id="predict" onclick="predictObject()">Predict</button>
      <button type="button" class="btn cancel" onclick="closeWebCamDisplay()">Close</button>
    </div>
    <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs/dist/tf.min.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/mobilenet"></script>
    <script src="tfjs/imageClassification.js"></script>
  </body>
</html>

As you can see in above HTML, we have very simple HTML here. We have button to start web cam. There is video element to show the preview of web cam. Then we have couple of buttons to start prediction. Now let’s add CSS. Create style.css in css folder and add following styles to it.

CSS Part

html {
  font-size: 60%;
  overflow-x: hidden;
  scroll-padding-top: 5rem;
  scroll-behavior: smooth;
}

header {
  margin-bottom: 2rem;
  position: relative;
  margin-left: 70px;
  margin-top: 30px;
  font-weight: bold;
}
header h1 {
  font-size: 30px;
  margin-left: 20px;
  font-weight: bold;
}

.btn {
  margin-top: 1rem;
  display: inline-block;
  padding: 1rem 2rem !important;
  border-radius: 1rem;
  color: #fff;
  background: #189ab4;
  font-size: 1.5rem !important;
  cursor: pointer;
  font-weight: 600 !important;
  margin-right: 10px;
}

.webCam-button {
  color: white;
  padding: 16px 20px;
  border: none;
  cursor: pointer;
  position: fixed;
  bottom: 140px;
  right: 28px;
}

.webcam-popup {
  display: none;
  position: fixed;
  bottom: 0;
  right: 15px;
  border: 3px solid #f1f1f1;
  z-index: 9;
  background-color: black;
}

JavaScript Part

Now we have html and css ready. It’s time for some JavaScript logic. We will create file with name imageClassification,js in tfjs folder and add following code to it.

const webcamElement = document.getElementById('webcam');
let startPrediction =  false;

async function app() {
  console.log('Loading mobilenet..');

  net = await mobilenet.load();
  console.log('Successfully loaded model');
  
  const webcam = await tf.data.webcam(webcamElement);
  while (true) {
    const img = await webcam.capture();
    const result = await net.classify(img);

    for(var i = 0 ; i < result.length ; i++ ) {
      res = result[i];
      if(res.probability > 0.5) {
        alert(res.className);
        location.reload();
        break;
      }
    }
    img.dispose();
    await tf.nextFrame();
  }
}

function openWebCam() {
  document.getElementById("webCamDisplay").style.display = "block";
}

function closeWebCamDisplay() {
  document.getElementById("webCamDisplay").style.display = "none";
}

function predictObject() {
  app();
}

Here we have functions to open & close pop up. There is predict button. Where we will start web cam preview and load the MobileNet Model. We are using async await syntax to wait till MobileNet model is loaded and then start the prediction. Following line loads the model.

net = await mobilenet.load();

Then we set the data source which is our webcam preview with following line.

const webcam = await tf.data.webcam(webcamElement);

Then we have while loop added where we capture the image frame and try to classify it with MobileNet model.

const img = await webcam.capture();
const result = await net.classify(img);

We are checking the results with accuracy over 50% to check if the classification is correct or not. If probability is good we show the classname in alert. This class name contains the name of object. It’s coming from MobileNet trained data.

When you run the code it may take a while to predict the object. You can check it out by placing multiple objects in front of web cam and check. Following are few examples.

As you can see in few cases the prediction is correct for example, mobile phone and mouse. But in the first case it’s wrong. I am not wearing bullet proof vest.

You can read more about TenorFlow.js on Official Website

Here is the link to MobileNet Model GitHub