bert_question_answering_demo

OpenVINO 2020.4から今、AI業界で話題のBERTのデモが入るようになりましたので、早速試してみます

その前にBERTによるquestion answering というものはなんでしょうか?
分かりやすいというか、そのもののページがこちらです。
と書いてしまうと簡単なのですが、ごくごく簡単に説明すると対象となるWebサイトの情報を元に質問すると答えを見つけてくれるデモと書いた方が良いでしょうか
まだまだ私の理解もそれほどではないので、これくらいしか説明できませんが…

https://openvino.jp/configuration-ubuntu1804/
を参考にOpenVINO 2020.4をインストールしてください

bert_question_answering_demoは
/opt/intel/openvino/deployment_tools/open_model_zoo/demos/python_demos/bert_question_answering_demo
に格納されています

$ cd opt/intel/openvino/deployment_tools/open_model_zoo/demos/python_demos/bert_question_answering_demo

$ ls
README.md  bert_question_answering_demo.py  models.lst  requirements.txt

$ python3 /opt/intel/openvino/deployment_tools/open_model_zoo/tools/downloader/downloader.py --list models.lst -o ~/openvino_models/

ここまででモデルがダウンロードできると思います
ただし、私の環境では、bert-large-uncased-whole-word-masking-squad-fp32-0001でFAILEDが出てしまいました
そのため、他のモデルで試すことにします

$ ./bert_question_answering_demo.py -h
 usage: bert_question_answering_demo.py [-h] -v VOCAB -m MODEL -i INPUT
                                        [--input_names INPUT_NAMES]
                                        [--output_names OUTPUT_NAMES]
                                        [--model_squad_ver MODEL_SQUAD_VER]
                                        [-q MAX_QUESTION_TOKEN_NUM]
                                        [-a MAX_ANSWER_TOKEN_NUM] [-d DEVICE]
                                        [-r] [-c]
 Options:
   -h, --help            Show this help message and exit.
   -v VOCAB, --vocab VOCAB
                         Required. path to the vocabulary file with tokens
   -m MODEL, --model MODEL
                         Required. Path to an .xml file with a trained model
   -i INPUT, --input INPUT
                         Required. URL to a page with context
   --input_names INPUT_NAMES
                         Optional. Inputs names for the network. Default values
                         are "input_ids,attention_mask,token_type_ids"
   --output_names OUTPUT_NAMES
                         Optional. Outputs names for the network. Default
                         values are "output_s,output_e"
   --model_squad_ver MODEL_SQUAD_VER
                         Optional. SQUAD version used for model fine tuning
   -q MAX_QUESTION_TOKEN_NUM, --max_question_token_num MAX_QUESTION_TOKEN_NUM
                         Optional. Maximum number of tokens in question
   -a MAX_ANSWER_TOKEN_NUM, --max_answer_token_num MAX_ANSWER_TOKEN_NUM
                         Optional. Maximum number of tokens in answer
   -d DEVICE, --device DEVICE
                         Optional. Target device to perform inference
                         on.Default value is CPU
   -r, --reshape         Optional. Auto reshape sequence length to the input
                         context + max question len (to improve the speed)
   -c, --colors          Optional. Nice coloring of the questions/answers.
                         Might not work on some terminals (like Windows* cmd
                         console)

VOCAB ファイルとモデルファイルが必須になります
vocabulary fileはopen model zooフォルダの中に格納されています
モデルは先ほどダウンロードしたものを使用します

$ ./bert_question_answering_demo.py -v ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt -m ~/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml -c --input="https://en.wikipedia.org/wiki/Bert_(Sesame_Street)"

上記コマンドでセサミストリートのBertにアクセスします
ただし残念ながらこのままだと下記のようなエラーが出ます

[ ERROR ]     Actual network input->output names: ['result.1', 'result.2', 'result.3']->['5211', '5212']

InputネームとOutputネームを設定し、下記のようなコマンドで再実行してみましょう

$ ./bert_question_answering_demo.py -v ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt -m ~/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml --input="https://en.wikipedia.org/wiki/Bert_(Sesame_Street)" --input_names="result.1,result.2,result.3" --output_names="5211,5212" -c
$ python3 ./bert_question_answering_demo.py -v ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt -m ~/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml --input="https://en.wikipedia.org/wiki/Bert_(Sesame_Street)" --input_names="result.1,result.2,result.3" --output_names="5211,5212" -c 
 [ INFO ] Loading vocab file:    ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt
 [ INFO ] 30522 tokens loaded
 [ INFO ] Get context from https://en.wikipedia.org/wiki/Bert_(Sesame_Street)
 [ INFO ] Size: 1265 chars
 [ INFO ] Context: Bert is a golden yellow Muppet character on the long running PBS and HBO children's television show Sesame Street. Bert was originally performed by Frank Oz.
 Since 1997, Muppeteer Eric Jacobson has been phased in as Bert's primary performer. Bert has also made cameo appearances within The Muppets franchise, including The Muppet Show, The Muppet Movie, and The Muppets Take Manhattan, in addition to an appearance with Ernie on The Flip Wilson Show on September 14, 1972, on which he sang "Clink, Clank", a song about noises.[2]
 Bert's age is unclear, but performer on Sesame Street Live, Taylor Morgan, has said that "I just kind of try to think like a six year old or a seven year old, because that's how old Bert is."[3] He and his roommate Ernie form a comic duo that is one of the program's centerpieces, with Bert acting as the world weary foil to Ernie's naive troublemaking.
 As the duo often sing in their skits, several albums were released containing studio recorded versions of their songs. Bert's best known song is "Doin' the Pigeon". He and Ernie both had their own video, The Best of Ernie and Bert, and their own album, Bert and Ernie's Greatest Hits. Only Ernie, however, has hit the U.S. Billboard Hot 100 Chart with his song "Rubber Duckie".
 [ INFO ] Size: 300 tokens
 [ INFO ] Initializing Inference Engine
 [ INFO ] Plugin version is 2.1.2020.4.0-359-21e092122f4-releases/2020/4
 [ INFO ] Loading network files:
     /home/klf/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml
     /home/klf/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.bin
 ./bert_question_answering_demo.py:216: DeprecationWarning: 'inputs' property of IENetwork class is deprecated. To access DataPtrs user need to use 'input_data' property of InputInfoPtr objects which can be accessed by 'input_info' property.
   if ie_encoder.inputs.keys() != set(input_names) or ie_encoder.outputs.keys() != set(output_names):
 [ INFO ] Loading model to the CPU
Type question (empty string to exit):

WikipediaのBertのページを読み込んでいます
この状態で、プログラムは入力待機状態になります
そこで、who is bertと入力してみると、下記のように回答が返ります

Type question (empty string to exit):who is bert

[ INFO ] Size: 3 tokens
[ INFO ] Sequence of length 384 is processed with 1.74 requests/sec (0.58 sec per request)
[ INFO ] The performance below is reported only for reference purposes, please use the benchmark_app tool (part of the OpenVINO samples) for any actual measurements.
[ INFO ] 1 requests of 384 length were processed in 0.58sec (0.58sec per request)
[ INFO ] ---answer: 0.44 Bert
[ INFO ]    Bert is a golden yellow Muppet character on the long running PBS and HBO children's television show Sesame Street
Type question (empty string to exit):

ということで、Bertはセサミストリートに出ているゴールデンイエローのぬいぐるみと表示されました

ここまでは、ReadMe通りに行ったので、次は別のものを使用してみます

wikipediaのサトウヤシを読み込む

$ ./bert_question_answering_demo.py -v ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt -m ~/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml  --input_names="result.1,result.2,result.3" --output_names="5211,5212" -c --input="https://en.wikipedia.org/wiki/Arenga_pinnata"

サトウヤシが何かはよくわかりませんが、Arenga Pinnataというみたいです
こちらを読み込ませてみます

Type question (empty string to exit):how high is arenga pinnata
 [ INFO ] Size: 8 tokens
 [ INFO ] Sequence of length 384 is processed with 1.64 requests/sec (0.61 sec per request)
 [ INFO ] Sequence of length 384 is processed with 1.76 requests/sec (0.57 sec per request)
 [ INFO ] Sequence of length 384 is processed with 1.77 requests/sec (0.57 sec per request)
 [ INFO ] Sequence of length 384 is processed with 1.70 requests/sec (0.59 sec per request)
 [ INFO ] The performance below is reported only for reference purposes, please use the benchmark_app tool (part of the OpenVINO samples) for any actual measurements.
 [ INFO ] 4 requests of 384 length were processed in 2.34sec (0.58sec per request)
 [ INFO ] ---answer: 0.52 20 m tall
 [ INFO ]    It is a medium-sized palm, growing to 20 m tall, with the trunk remaining covered by the rough old leaf bases
 [ INFO ] ---answer: 0.00 Arenga pinnata in the northwestern Philippines
 [ INFO ]     Irok is a local name for Arenga pinnata in the northwestern Philippines
 [ INFO ] ---answer: 0.00 Philippines
 [ INFO ]    Sugar (jaggery) is also commonly derived from the fresh sap in Indonesia and the Philippines
 Type question (empty string to exit):

いくつか回答が出ていますが、一番上位にあるものは20mの高さがあると表示されています
うまくいっているようですね

InputデータはURLで指定してますが、こちらは下記のようにwgetなどで得たデータを入れることも可能です

$ wget https://en.wikipedia.org/wiki/Arenga_pinnata -O /home/foo/Arenga_pinnata

$ ./bert_question_answering_demo.py -v ../../../models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/vocab.txt -m ~/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-int8-0001/FP16-INT8/bert-large-uncased-whole-word-masking-squad-int8-0001.xml  --input_names="result.1,result.2,result.3" --output_names="5211,5212" -c --input="file:/home/foo/Arenga_pinnata"

次は日本語の処理をやってみましょう

OpenVINO 2020.4 Release

OpenVINO 2020.4がリリースされました
https://software.intel.com/en-us/openvino-toolkit
https://software.intel.com/content/www/us/en/develop/articles/openvino-relnotes.html

Intelからのリリースを転記します

Executive Summary

  • Improves performance while maintaining accuracy close to full precision (for example, FP32 data type) by introducing support for the Bfloat16 data type for inferencing using the 3rd generation Intel® Xeon® Scalable processor (formerly code-named Cooper Lake).
  • Increases accuracy when layers have varying bit-widths by extending the Post-Training Optimization Tool to support mixed-precision quantization.
  • Allows greater compatibility of models by supporting directly reading Open Neural Network Exchange (ONNX*) model format to the Inference Engine.For users looking to take full advantage of Intel® Distribution of OpenVINO™ toolkit, it is recommended to follow the native workflow of using the Intermediate Representation from the Model Optimizer as input to the Inference Engine.
  • For users looking to more easily take a converted model in ONNX model format (for example, PyTorch to ONNX using torch.onnx), they are now able to input the ONNX format directly to the Inference Engine to run models on Intel architecture.
  • Enables initial support for TensorFlow* 2.2.0 for computer vision use cases.
  • Enables users to connect to and profile multiple remote hosts; collect and store data in one place for further analysis by extending the Deep Learning Workbench with remote profiling capability.

CooperLakeでの演算が追加、ONNXモデルのサポート、TensorFlow 2.2.0 がサポートされたようです

OpenVINO.jpでは引き続きベンチマークなどを行っていきたいと思います

環境構築(ubuntu18.04編)

インストールの前に

  • インターネット環境

※2020.06.04追記:OpenVINO 2020.3 LTS Release も同様の手順で入る事を確認済です
※2020.07.21追記:OpenVINO 2020.4 Release も同様の手順で入る事を確認済です

インストールされるコンポーネント

コンポーネント説明
Model OptimizerThis tool imports, converts, and optimizes models, which were trained in popular frameworks, to a format usable by Intel tools, especially the Inference Engine.
Popular frameworks include Caffe*, TensorFlow*, MXNet*, and ONNX*.
Inference Engine
ディープラーニングエンジン
アプリケーション組み込みライブラリ
Drivers and runtimes for OpenCL™ version 2.1Enables OpenCL on the GPU/CPU for Intel® processors
Intel® Media SDKOffers access to hardware accelerated video codecs and frame processing
OpenCVIntel Hardware用にコンパイルされたOpenCV
Sample Applications推論エンジンを使用するためのコンソールデモアプリケーション
DemosA set of console applications that demonstrate how you can use the Inference Engine in your applications to solve specific use-cases
Additional ToolsA set of tools to work with your models
Documentation for PreTrained ModelsDocumentation for the pre-trained models available in the Open Model Zoo repo

プラットフォーム

ハードウェアの制限

  • 6th-10th Generation Intel® Core™
  • Intel® Xeon® v5 family
  • Intel® Xeon® v6 family
  • Intel® Neural Compute Stick 2

OS

  • Ubuntu 18.04.x long-term support (LTS), 64-bit
  • CentOS 7.4, 64-bit (for target only)
  • Yocto Project v3.0, 64-bit (for target only and requires modifications)

ソフトウェアのインストール

今回はUbuntu18.04にOpenVINO toolkit 2020.2をセットアップします
CPUスペックは、Intel(R) Atom(TM) Processor E3950 @ 1.60GHzとなります

必要なソフトウェア

  • CMake 3.4 or higher
  • Python 3.5 or higher

OpenVINO ToolKIT のダウンロード

こちらから登録を行い、ダウンロードして下さい

OpenVINO Toolkit のインストール

openvino@ubuntu$ ls l_openvino_toolkit_p_2020.2.120.tgz

openvino@ubuntu$ tar -xvzf l_openvino_toolkit_p_2020.2.120.tgz

openvino@ubuntu$ cd l_openvino_toolkit_p_2020.2.120/

#コマンドラインインストール
openvino@ubuntu$ sudo ./install.sh
 Welcome
 --------------------------------------------------------------------------------
 Welcome to the Intel® Distribution of OpenVINO™ toolkit 2020.2 for Linux*
 --------------------------------------------------------------------------------
 The Intel installation wizard will install the Intel® Distribution of OpenVINO™
 toolkit 2020.2 for Linux* to your system.
 

 The Intel® Distribution of OpenVINO™ toolkit quickly deploys applications and
 solutions that emulate human vision. Based on Convolutional Neural Networks
 (CNN), the toolkit extends computer vision (CV) workloads across Intel®
 hardware, maximizing performance. The Intel Distribution of OpenVINO toolkit
 includes the Intel® Deep Learning Deployment Toolkit (Intel® DLDT).
 

 Before installation please check system requirements:
 https://docs.openvinotoolkit.org/2020.2/_docs_install_guides_installing_openvino
 _linux.html#system_requirements
 and run following script to install external software dependencies:
 

 sudo -E ./install_openvino_dependencies.sh
 

 Please note that after the installation is complete, additional configuration
 steps are still required.
 

 For the complete installation procedure, refer to the Installation guide:
 https://docs.openvinotoolkit.org/2020.2/_docs_install_guides_installing_openvino
 _linux.html.
 

 You will complete the following steps:
    1.  Welcome
    2.  End User License Agreement
    3.  Prerequisites
    4.  Configuration
    5.  Installation
    6.  First Part of Installation is Complete
 

 --------------------------------------------------------------------------------
 Press "Enter" key to continue or "q" to quit:  

基本的には、Enterで進めていくとインストールされます
途中で、必要なアプリケーションなどのチェックが入りますので、
もし不足している場合は、画面に従いインストールが必要になります

#必要であればここで別端末を利用してソフトウェアをインストールします
openvino@ubuntu$ sudo apt-get install cmake
defaultのまま設定をすすめていくと、

 First Part of Installation is Complete
 --------------------------------------------------------------------------------
 The first part of Intel® Distribution of OpenVINO™ toolkit 2020.2 for Linux*
 has been successfully installed in 
 /opt/intel/openvino_2020.2.120.
 

 ADDITIONAL STEPS STILL REQUIRED: 
 

 Open the Installation guide at:
  https://docs.openvinotoolkit.org/2020.2/_docs_install_guides_installing_openvin
 o_linux.html 
 and follow the guide instructions to complete the remaining tasks listed below:
 

  • Set Environment variables 
  • Configure Model Optimizer 
  • Run the Verification Scripts to Verify Installation and Compile Samples
 

 --------------------------------------------------------------------------------
 Press "Enter" key to quit:  

インストール完了です

外部依存ソフトウェアのインストール

openvino@ubuntu$ cd /opt/intel/openvino/install_dependencies/
openvino@ubuntu$ sudo -E ./install_openvino_dependencies.sh 
 

 This script installs the following OpenVINO 3rd-party dependencies:
   1. GTK+, FFmpeg and GStreamer libraries used by OpenCV
   2. libusb library required for Myriad plugin for Inference Engine
   3. build dependencies for OpenVINO samples
   4. build dependencies for GStreamer Plugins
  
環境により異なりますが、上記ライブラリが自動的にインストールされます

環境変数の設定

openvino@ubuntu$ source /opt/intel/openvino/bin/setupvars.sh

#ログイン時に環境変数を有効にする場合
vi <user_directory>/.bashrc
#最終行に下記を追加
source /opt/intel/openvino/bin/setupvars.sh 

Model Optimizer の設定

openvino@ubuntu$ cd /opt/intel/openvino/deployment_tools/model_optimizer/install_prerequisites
 openvino@ubuntu$ sudo ./install_prerequisites.sh

インストールのチェック

openvino@ubuntu$ cd /opt/intel/openvino/deployment_tools/demo
openvino@ubuntu$ ./demo_squeezenet_download_convert_run.sh 

Demo check

openvino@ubuntu$ cd /opt/intel/openvino/deployment_tools/demo
openvino@ubuntu$ ./demo_security_barrier_camera.sh

2020.1リリースよりも若干早くなっているようです

GPU関連モジュールインストール

openvino@ubuntu$ cd /opt/intel/openvino/install_dependencies/
openvino@ubuntu$ sudo -E su
root@ubuntu# ./install_NEO_OCL_driver.sh

ここで過去にOpenVINOをインストールしていた場合、依存関係で下記のようなエラーが発生します

 dpkg: dependency problems prevent removal of intel-igc-core:
  intel-igc-opencl depends on intel-igc-core (= 1.0.2597).
 

 dpkg: error processing package intel-igc-core (--purge):
  dependency problems - not removing
 Errors were encountered while processing:
  intel-igc-core
 ERROR: unable to remove intel-igc-core
        please resolve it manually and try to launch the script again. 

intel-igc-openclがintel-igc-coreに依存関係があるためで、この場合は手動でintel-igc-openclを削除しましょう(下記コマンド参照)

dpkg -r intel-igc-opencl

Demo check

openvino@ubuntu$ cd /opt/intel/openvino/deployment_tools/demo
openvino@ubuntu$ ./demo_security_barrier_camera.sh -d GPU

CPUと比較して3倍弱のスピードが出ています

Neural Compute Stick2 の使用

#usersグループにカレントユーザーを加えます 
$ sudo usermod -a -G users "$(whoami)"

#NCS2用のルールをコピーし、リブートします
$ sudo cp /opt/intel/openvino/inference_engine/external/97-myriad-usbboot.rules /etc/udev/rules.d/
$ sudo udevadm control --reload-rules
$ sudo udevadm trigger
$ sudo ldconfig 

2020.1と同様にGPUと同等程度の速さとなりました

お疲れ様でした
インストールはこれで完了です