How to Run Tensorflow Object Detection in Real Time with Raspberry V2 CSI Camera on NVIDIA Jetson Nano - Forecr.io

How to Run Tensorflow Object Detection in Real Time with Raspberry V2 CSI Camera on NVIDIA Jetson Nano

Jetson Nano

09 August 2021
WHAT YOU WILL LEARN?

1- How to use Tensorflow libraries and Raspberry V2 Camera with docker.

2- How to run Tensorflow Object Detection in Python.

3- How to detect objects in real time with Tensorflow.

ENVIRONMENT

Hardware: DSBOX-N2

OS: Ubuntu 18.04 LTS, JetPack 4.5

How to use Tensorflow libraries and CSI camera with docker


 In this blog post, we will learn how to run Tensorflow Object Detection in real time with a CSI camera. The GitHub repo has been taken as a reference for the whole process.

 Before powering up the NVIDIA® Jetson™ Nano™, we should connect our CSI camera, for example Raspberry V2 camera like we are going to use for this tutorial.

 First, we must install Tensorflow Object Detection models. Let us go into the Documents folder and create folder for all required files named Tensorflow, so that we can work methodically:


cd ./Documents 
mkdir Tensorflow
cd Tensorflow


Now let us install the required files while inside Tensorflow folder:

git clone https://github.com/tensorflow/models.git 


Now we need protobuf for serializing structured data, let us install it to the same folder:

git clone
https://github.com/protocolbuffers/protobuf/releases/download/v3.17.3/protoc-3.17.3-linux-aarch_64.zip


Let us install unzip to extract our folder:

sudo apt-get update 
sudo apt-get install unzip


Now let us extract our zip file and rename it, while deleting the old zip file afterwards:

unzip protoc-3.17.3-linux-aarch_64.zip 
mv protoc-3.17.3-linux-aarch_64 protobuf
rm protoc-3.17.3-linux-aarch_64.zip


Now we should go inside the research folder inside Tensorflow and run protobuf:

cd models/research/ 
sudo protoc object_detection/protos/*.proto –python_out=.


If it worked correctly, you should see python files in the folder path Tensorflow/ models/research/object_detection/protos:



And the final requirement for our setup, installing Gstreamer, let us install it:

sudo apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio 
sudo apt-get install gstreamer1.0-tools

How to run Tensorflow Object Detection in Python


 Since our preparations outside the docker are done, let us install our container and run it:

cd 
docker pull nvcr.io/nvidia/l4t-ml:r32.5.0-py3
sudo docker run -it --rm --gpus all -e DISPLAY=:0 -v /tmp/.X11-unix:/tmp/.X11-unix -v /home/nvidia/Documents:/home/nvidia/Documents -v /tmp/argus_socket:/tmp/argus_socket --device /dev/video0:/dev/video0 --network host nvcr.io/nvidia/l4t-ml:r32.5.0-py3



Now, we should be inside our container. We need to create, write and run a python file, and in order to do that, we need to install nano and Tkinter:


apt-get update 
apt-get install nano
apt-get install python3-tk


After we got these installations, we need to create a python file inside Tensorflow folder, and write our code:

cd ./home/nvidia/Documents/Tensorflow/models/research/object_detection/ 
touch Real_time_object_detection.py
nano Real_time_object_detection.py


Now you should see an empty file like below:



Copy the below code and paste it with Ctrl + Shift + C to this empty file:

import numpy as np 
import os
import six.moves.urllib as urllib
import sys
import tarfile
import tensorflow as tf
import zipfile
import tkinter
from collections import defaultdict
from io import StringIO
from matplotlib import pyplot as plt
from PIL import Image

import cv2

sys.path.append("..")

def gstreamer_pipeline(
     capture_width=800,
     capture_height=600,
     display_width=800,
     display_height=600,
     framerate=30,
     flip_method=2,
):
     return (
         "nvarguscamerasrc ! "
         "video/x-raw(memory:NVMM), "
         "width=(int)%d, height=(int)%d, "
         "format=(string)NV12, framerate=(fraction)%d/1 ! "
         "nvvidconv flip-method=%d ! "
         "video/x-raw, width=(int)%d, height=(int)%d, format=(string)BGRx ! "
         "videoconvert ! "
         "video/x-raw, format=(string)BGR ! appsink"
         % (
             capture_width,
             capture_height,
             framerate,
             flip_method,
             display_width,
             display_height,
         )
     )

from utils import label_map_util

from utils import visualization_utils as vis_util

cap = cv2.VideoCapture(gstreamer_pipeline(flip_method=2), cv2.CAP_GSTREAMER)

MODEL_NAME = 'ssd_mobilenet_v1_coco_11_06_2017'
MODEL_FILE = MODEL_NAME + '.tar.gz'
DOWNLOAD_BASE = 'http://download.tensorflow.org/models/object_detection/'

# Path to frozen detection graph. This is the actual model that is used for the object detection.
PATH_TO_CKPT = MODEL_NAME + '/frozen_inference_graph.pb'

# List of the strings that is used to add correct label for each box.
PATH_TO_LABELS = os.path.join('data', 'mscoco_label_map.pbtxt')

NUM_CLASSES = 90

opener = urllib.request.URLopener()
opener.retrieve(DOWNLOAD_BASE + MODEL_FILE, MODEL_FILE)
tar_file = tarfile.open(MODEL_FILE)
for file in tar_file.getmembers():
  file_name = os.path.basename(file.name)
  if 'frozen_inference_graph.pb' in file_name:
    tar_file.extract(file, os.getcwd())

detection_graph = tf.Graph()
with detection_graph.as_default():
  od_graph_def = tf.GraphDef()
  with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
    serialized_graph = fid.read()
    od_graph_def.ParseFromString(serialized_graph)
    tf.import_graph_def(od_graph_def, name='')

label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
category_index = label_map_util.create_category_index(categories)

with detection_graph.as_default():
  with tf.Session(graph=detection_graph) as sess:
    while True:
      ret, image_np = cap.read()
      image_np_expanded = np.expand_dims(image_np, axis=0)
      image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
      boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
      # Score is shown on the result image, together with class label.
      scores = detection_graph.get_tensor_by_name('detection_scores:0')
      classes = detection_graph.get_tensor_by_name('detection_classes:0')
      num_detections = detection_graph.get_tensor_by_name('num_detections:0')

      # Actual detection.
      (boxes, scores, classes, num_detections) = sess.run(
        [boxes, scores, classes, num_detections],
        feed_dict={image_tensor: image_np_expanded})
      # Visualization of the results of a detection.
      vis_util.visualize_boxes_and_labels_on_image_array(
          image_np,
          np.squeeze(boxes),
          np.squeeze(classes).astype(np.int32),
          np.squeeze(scores),
          category_index,
          use_normalized_coordinates=True,
          line_thickness=8)

      cv2.imshow('object detection', image_np)
      if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
       break


Then press Ctrl + X to quit, then press Y to save changes, and finally press ‘Enter’ to confirm the file’s name. 

How to detect objects in real time with Tensorflow


The name of the models we are using are ‘ssd_mobilenet_v1_coco_11_06_2017’. Now, we need to gain access to our camera from docker. Open a new terminal using Ctrl + Alt + T, and write the following:

xhost + 


We should see the following output from the terminal. Then let’s switch back to our first terminal and run our code, since we are ready:

python3 Real_time_object_detection.py 

Running our file may take a couple of minutes. You can see how it works below:



Press ‘q’ to stop the program.

Thank you for reading our blog post.