OpenVINO 2020.4から今、AI業界で話題のBERTのデモが入るようになりましたので、早速試してみます

その前にBERTによるquestion answering というものはなんでしょうか?
を参考にOpenVINO 2020.4をインストールしてください


$ cd opt/intel/openvino/deployment_tools/open_model_zoo/demos/python_demos/bert_question_answering_demo

$ ls  models.lst  requirements.txt

$ python3 /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/ --list models.lst -o ~/openvino_models/


$ ./ -h
 usage: [-h] -v VOCAB -m MODEL -i INPUT
                                        [--input_names INPUT_NAMES]
                                        [--output_names OUTPUT_NAMES]
                                        [--model_squad_ver MODEL_SQUAD_VER]
                                        [-q MAX_QUESTION_TOKEN_NUM]
                                        [-a MAX_ANSWER_TOKEN_NUM] [-d DEVICE]
                                        [-r] [-c]
   -h, --help            Show this help message and exit.
   -v VOCAB, --vocab VOCAB
                         Required. path to the vocabulary file with tokens
   -m MODEL, --model MODEL
                         Required. Path to an .xml file with a trained model
   -i INPUT, --input INPUT
                         Required. URL to a page with context
   --input_names INPUT_NAMES
                         Optional. Inputs names for the network. Default values
                         are "input_ids,attention_mask,token_type_ids"
   --output_names OUTPUT_NAMES
                         Optional. Outputs names for the network. Default
                         values are "output_s,output_e"
   --model_squad_ver MODEL_SQUAD_VER
                         Optional. SQUAD version used for model fine tuning
   -q MAX_QUESTION_TOKEN_NUM, --max_question_token_num MAX_QUESTION_TOKEN_NUM
                         Optional. Maximum number of tokens in question
   -a MAX_ANSWER_TOKEN_NUM, --max_answer_token_num MAX_ANSWER_TOKEN_NUM
                         Optional. Maximum number of tokens in answer
   -d DEVICE, --device DEVICE
                         Optional. Target device to perform inference
                         on.Default value is CPU
   -r, --reshape         Optional. Auto reshape sequence length to the input
                         context + max question len (to improve the speed)
   -c, --colors          Optional. Nice coloring of the questions/answers.
                         Might not work on some terminals (like Windows* cmd

VOCAB ファイルとモデルファイルが必須になります
vocabulary fileはopen model zooフォルダの中に格納されています

$ ./ -v ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt -m ~/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml -c --input=""


[ ERROR ]     Actual network input->output names: ['result.1', 'result.2', 'result.3']->['5211', '5212']


$ ./ -v ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt -m ~/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml --input="" --input_names="result.1,result.2,result.3" --output_names="5211,5212" -c
$ python3 ./ -v ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt -m ~/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml --input="" --input_names="result.1,result.2,result.3" --output_names="5211,5212" -c 
 [ INFO ] Loading vocab file:    ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt
 [ INFO ] 30522 tokens loaded
 [ INFO ] Get context from
 [ INFO ] Size: 1265 chars
 [ INFO ] Context: Bert is a golden yellow Muppet character on the long running PBS and HBO children's television show Sesame Street. Bert was originally performed by Frank Oz.
 Since 1997, Muppeteer Eric Jacobson has been phased in as Bert's primary performer. Bert has also made cameo appearances within The Muppets franchise, including The Muppet Show, The Muppet Movie, and The Muppets Take Manhattan, in addition to an appearance with Ernie on The Flip Wilson Show on September 14, 1972, on which he sang "Clink, Clank", a song about noises.[2]
 Bert's age is unclear, but performer on Sesame Street Live, Taylor Morgan, has said that "I just kind of try to think like a six year old or a seven year old, because that's how old Bert is."[3] He and his roommate Ernie form a comic duo that is one of the program's centerpieces, with Bert acting as the world weary foil to Ernie's naive troublemaking.
 As the duo often sing in their skits, several albums were released containing studio recorded versions of their songs. Bert's best known song is "Doin' the Pigeon". He and Ernie both had their own video, The Best of Ernie and Bert, and their own album, Bert and Ernie's Greatest Hits. Only Ernie, however, has hit the U.S. Billboard Hot 100 Chart with his song "Rubber Duckie".
 [ INFO ] Size: 300 tokens
 [ INFO ] Initializing Inference Engine
 [ INFO ] Plugin version is 2.1.2020.4.0-359-21e092122f4-releases/2020/4
 [ INFO ] Loading network files:
 ./ DeprecationWarning: 'inputs' property of IENetwork class is deprecated. To access DataPtrs user need to use 'input_data' property of InputInfoPtr objects which can be accessed by 'input_info' property.
   if ie_encoder.inputs.keys() != set(input_names) or ie_encoder.outputs.keys() != set(output_names):
 [ INFO ] Loading model to the CPU
Type question (empty string to exit):

そこで、who is bertと入力してみると、下記のように回答が返ります

Type question (empty string to exit):who is bert

[ INFO ] Size: 3 tokens
[ INFO ] Sequence of length 384 is processed with 1.74 requests/sec (0.58 sec per request)
[ INFO ] The performance below is reported only for reference purposes, please use the benchmark_app tool (part of the OpenVINO samples) for any actual measurements.
[ INFO ] 1 requests of 384 length were processed in 0.58sec (0.58sec per request)
[ INFO ] ---answer: 0.44 Bert
[ INFO ]    Bert is a golden yellow Muppet character on the long running PBS and HBO children's television show Sesame Street
Type question (empty string to exit):




$ ./ -v ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt -m ~/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml  --input_names="result.1,result.2,result.3" --output_names="5211,5212" -c --input=""

サトウヤシが何かはよくわかりませんが、Arenga Pinnataというみたいです

Type question (empty string to exit):how high is arenga pinnata
 [ INFO ] Size: 8 tokens
 [ INFO ] Sequence of length 384 is processed with 1.64 requests/sec (0.61 sec per request)
 [ INFO ] Sequence of length 384 is processed with 1.76 requests/sec (0.57 sec per request)
 [ INFO ] Sequence of length 384 is processed with 1.77 requests/sec (0.57 sec per request)
 [ INFO ] Sequence of length 384 is processed with 1.70 requests/sec (0.59 sec per request)
 [ INFO ] The performance below is reported only for reference purposes, please use the benchmark_app tool (part of the OpenVINO samples) for any actual measurements.
 [ INFO ] 4 requests of 384 length were processed in 2.34sec (0.58sec per request)
 [ INFO ] ---answer: 0.52 20 m tall
 [ INFO ]    It is a medium-sized palm, growing to 20 m tall, with the trunk remaining covered by the rough old leaf bases
 [ INFO ] ---answer: 0.00 Arenga pinnata in the northwestern Philippines
 [ INFO ]     Irok is a local name for Arenga pinnata in the northwestern Philippines
 [ INFO ] ---answer: 0.00 Philippines
 [ INFO ]    Sugar (jaggery) is also commonly derived from the fresh sap in Indonesia and the Philippines
 Type question (empty string to exit):



$ wget -O /home/foo/Arenga_pinnata

$ ./ -v ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt -m ~/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml  --input_names="result.1,result.2,result.3" --output_names="5211,5212" -c --input="file:/home/foo/Arenga_pinnata"


OpenVINO 2020.4 Release

OpenVINO 2020.4がリリースされました


Executive Summary

  • Improves performance while maintaining accuracy close to full precision (for example, FP32 data type) by introducing support for the Bfloat16 data type for inferencing using the 3rd generation Intel® Xeon® Scalable processor (formerly code-named Cooper Lake).
  • Increases accuracy when layers have varying bit-widths by extending the Post-Training Optimization Tool to support mixed-precision quantization.
  • Allows greater compatibility of models by supporting directly reading Open Neural Network Exchange (ONNX*) model format to the Inference Engine.For users looking to take full advantage of Intel® Distribution of OpenVINO™ toolkit, it is recommended to follow the native workflow of using the Intermediate Representation from the Model Optimizer as input to the Inference Engine.
  • For users looking to more easily take a converted model in ONNX model format (for example, PyTorch to ONNX using torch.onnx), they are now able to input the ONNX format directly to the Inference Engine to run models on Intel architecture.
  • Enables initial support for TensorFlow* 2.2.0 for computer vision use cases.
  • Enables users to connect to and profile multiple remote hosts; collect and store data in one place for further analysis by extending the Deep Learning Workbench with remote profiling capability.

CooperLakeでの演算が追加、ONNXモデルのサポート、TensorFlow 2.2.0 がサポートされたようです




  • インターネット環境

※2020.06.04追記:OpenVINO 2020.3 LTS Release も同様の手順で入る事を確認済です
※2020.07.21追記:OpenVINO 2020.4 Release も同様の手順で入る事を確認済です


Model OptimizerThis tool imports, converts, and optimizes models, which were trained in popular frameworks, to a format usable by Intel tools, especially the Inference Engine.
Popular frameworks include Caffe*, TensorFlow*, MXNet*, and ONNX*.
Inference Engine
Drivers and runtimes for OpenCL™ version 2.1Enables OpenCL on the GPU/CPU for Intel® processors
Intel® Media SDKOffers access to hardware accelerated video codecs and frame processing
OpenCVIntel Hardware用にコンパイルされたOpenCV
Sample Applications推論エンジンを使用するためのコンソールデモアプリケーション
DemosA set of console applications that demonstrate how you can use the Inference Engine in your applications to solve specific use-cases
Additional ToolsA set of tools to work with your models
Documentation for PreTrained ModelsDocumentation for the pre-trained models available in the Open Model Zoo repo



  • 6th-10th Generation Intel® Core™
  • Intel® Xeon® v5 family
  • Intel® Xeon® v6 family
  • Intel® Neural Compute Stick 2


  • Ubuntu 18.04.x long-term support (LTS), 64-bit
  • CentOS 7.4, 64-bit (for target only)
  • Yocto Project v3.0, 64-bit (for target only and requires modifications)


今回はUbuntu18.04にOpenVINO toolkit 2020.2をセットアップします
CPUスペックは、Intel(R) Atom(TM) Processor E3950 @ 1.60GHzとなります


  • CMake 3.4 or higher
  • Python 3.5 or higher

OpenVINO ToolKIT のダウンロード


OpenVINO Toolkit のインストール

openvino@ubuntu$ ls l_openvino_toolkit_p_2020.2.120.tgz

openvino@ubuntu$ tar -xvzf l_openvino_toolkit_p_2020.2.120.tgz

openvino@ubuntu$ cd l_openvino_toolkit_p_2020.2.120/

openvino@ubuntu$ sudo ./
 Welcome to the Intel® Distribution of OpenVINO™ toolkit 2020.2 for Linux*
 The Intel installation wizard will install the Intel® Distribution of OpenVINO™
 toolkit 2020.2 for Linux* to your system.

 The Intel® Distribution of OpenVINO™ toolkit quickly deploys applications and
 solutions that emulate human vision. Based on Convolutional Neural Networks
 (CNN), the toolkit extends computer vision (CV) workloads across Intel®
 hardware, maximizing performance. The Intel Distribution of OpenVINO toolkit
 includes the Intel® Deep Learning Deployment Toolkit (Intel® DLDT).

 Before installation please check system requirements:
 and run following script to install external software dependencies:

 sudo -E ./

 Please note that after the installation is complete, additional configuration
 steps are still required.

 For the complete installation procedure, refer to the Installation guide:

 You will complete the following steps:
    1.  Welcome
    2.  End User License Agreement
    3.  Prerequisites
    4.  Configuration
    5.  Installation
    6.  First Part of Installation is Complete

 Press "Enter" key to continue or "q" to quit:  


openvino@ubuntu$ sudo apt-get install cmake

 First Part of Installation is Complete
 The first part of Intel® Distribution of OpenVINO™ toolkit 2020.2 for Linux*
 has been successfully installed in 


 Open the Installation guide at:
 and follow the guide instructions to complete the remaining tasks listed below:

  • Set Environment variables 
  • Configure Model Optimizer 
  • Run the Verification Scripts to Verify Installation and Compile Samples

 Press "Enter" key to quit:  



openvino@ubuntu$ cd /opt/intel/openvino/install_dependencies/
openvino@ubuntu$ sudo -E ./ 

 This script installs the following OpenVINO 3rd-party dependencies:
   1. GTK+, FFmpeg and GStreamer libraries used by OpenCV
   2. libusb library required for Myriad plugin for Inference Engine
   3. build dependencies for OpenVINO samples
   4. build dependencies for GStreamer Plugins


openvino@ubuntu$ source /opt/intel/openvino/bin/

vi <user_directory>/.bashrc
source /opt/intel/openvino/bin/ 

Model Optimizer の設定

openvino@ubuntu$ cd /opt/intel/openvino/deployment_tools/model_optimizer/install_prerequisites
 openvino@ubuntu$ sudo ./


openvino@ubuntu$ cd /opt/intel/openvino/deployment_tools/demo
openvino@ubuntu$ ./ 

Demo check

openvino@ubuntu$ cd /opt/intel/openvino/deployment_tools/demo
openvino@ubuntu$ ./



openvino@ubuntu$ cd /opt/intel/openvino/install_dependencies/
openvino@ubuntu$ sudo -E su
root@ubuntu# ./


 dpkg: dependency problems prevent removal of intel-igc-core:
  intel-igc-opencl depends on intel-igc-core (= 1.0.2597).

 dpkg: error processing package intel-igc-core (--purge):
  dependency problems - not removing
 Errors were encountered while processing:
 ERROR: unable to remove intel-igc-core
        please resolve it manually and try to launch the script again. 


dpkg -r intel-igc-opencl

Demo check

openvino@ubuntu$ cd /opt/intel/openvino/deployment_tools/demo
openvino@ubuntu$ ./ -d GPU


Neural Compute Stick2 の使用

$ sudo usermod -a -G users "$(whoami)"

$ sudo cp /opt/intel/openvino/inference_engine/external/97-myriad-usbboot.rules /etc/udev/rules.d/
$ sudo udevadm control --reload-rules
$ sudo udevadm trigger
$ sudo ldconfig