The world of machine learning, and more specifically deep learning, is a
rapidly growing field. It’s growing in the sense of how quickly the
market is expanding, especially as deep learning moves to the edge. In my
microcosm at NXP, I see the deep learning customer base increasing
dramatically, as more and more engineers build applications that include some
form of vision- or voice-based machine learning technology. The number of deep
learning frameworks, tools and other capabilities that allow developers to
build and deploy neural network models are also expanding.
One example of such a tool is the Glow neural network (NN) model compiler.
Aligned with the proliferation of deep learning frameworks such as PyTorch, NN
compilers provide optimizations to accelerate inferencing on a range of
hardware platforms. In May 2018, Facebook introduced Glow (the graph lowering
compiler) as an open source community project and it has evolved significantly
over the last two years thanks to the efforts of more than 130 worldwide
contributors.
Recently, we rolled out our official support for this Glow compiler and
we’re very excited about the performance and memory benefits
it’s delivering for our devices. We have tightly integrated Glow into
our MCUXpresso SDK, which packages the Glow compiler and quantization tools
into an easy-to-use installer along with detailed documentation and labs to
get running quickly with your own models.
Glow Flexible Functionality
As an NN compiler, Glow takes in a computation graph and generates optimized
machine code over two phases. In the first phase, it optimizes the operators
and layers of the model using standard compiler techniques such as kernel
fusion, lowering of complex operations to simple kernels and transpose
elimination. In the second, or backend phase of the model compilation, the
Glow compiler uses LLVM modules to enable target-specific optimizations. Glow
supports ahead of time (AOT) compilation, where compilation is performed
offline to generate an object file (called a Glow bundle) which is later
linked with the user’s application code. When this object file is
generated, all unnecessary overhead is eliminated, reducing the number of
computations as well as the memory overhead. This is ideal for deploying on
memory-constrained and low-cost microcontrollers.
Target-Specific Optimizations
While any device in our i.MX RT series will run a Glow compiled model, we
started our testing on the i.MX RT1060 MCU because we also have TensorFlow™
running on this device, and it allowed us to have a direct performance
comparison. We also started with the i.MX RT685 MCU because this is a new
device and the only one in our i.MX RT series with a DSP optimized for
processing neural network operators. The i.MX RT1060 MCU contains a 600 MHz
Arm® Cortex®-M7 and 1MB of SRAM. The i.MX
RT685 MCU contains a 600 MHz Cadence® Tensilica®
HiFi 4 DSP core paired with a 300 MHz Cortex-M33 core and 4.5 MB of on-chip
SRAM.
The standard version of Glow from GitHub is device agnostic; it can compile
for basic architectures of interest. For example, for cross-compiling a bundle
for the Arm Cortex-M7 core, use the command line –target=arm -mcpu=cortex-m7. However, as I mentioned, Glow’s LLVM backend support can
cross-compile bundles for different target architectures. NXP has taken
advantage of this by using Arm CMSIS-NN to leverage the full capability of the
Cortex-M7 as well as the memory subsystem of the i.MX RT1060 device. CMSIS-NN
is an Arm-developed library supporting Arm Cortex-M0, -M3, -M4, -M7 and -M33
cores and it implements standard NN operations like convolution, fully
connected, pooling and activation. Simply use the compilation flag -use-cmsis
when building quantized bundles, and the performance will increase
significantly above the standard compilation. For example, as measured by NXP
on a CIFAR-10 model, performance increases by almost 2x, when using the
CMSIS-NN library to accelerate NN operations.
The HiFi 4 DSP core on the i.MX RT685 device is also capable of accelerating a
wide range of NN operators when used with Cadence’s NN library (NNLib)
as another LLVM backend for Glow. NN Lib is like CMSIS-NN, except it provides
a much more comprehensive set of hand-tuned operators optimized for the HiFi 4
DSP. For the same CIFAR-10 example, this DSP delivers a 25x performance
increase compared to the Glow standard implementation.
PyTorch for Embedded Systems
PyTorch can directly export models into the ONNX format for use by Glow.
Alternatively, since many well-known models were created in other formats
(e.g. TensorFlow), there are also open source model conversion tools to
convert them to the ONNX format. The most used tools for format conversion are
MMDNN, a set of tools supported by Microsoft® to help users
inter-operate among different deep learning frameworks and
tf2onnx
to convert TensorFlow models to ONNX. Furthermore, NXP has upstreamed to the
Glow community a support feature to bring TensorFlow Lite models in directly
to Glow. More recently, Glow can be directly accessed through PyTorch,
allowing users to build and compile their models in the same development
environment, thereby eliminating steps and simplifying the compilation
process.
However, because of its broad use in datacenters by companies such as
Facebook, people have questioned PyTorch’s ability to serve as a
framework for embedded MCUs. With Glow becoming directly accessible from
PyTorch, is there reason to be concerned that ’PyTorch, and hence Glow,
is not targeted at MCUs?’ The short answer is ‘no’,
especially given the AOT implementation of Glow.
To explain this further, it would be a valid impression that PyTorch itself
isn’t targeted towards MCUs. It’s a community project and no one
has stepped up yet to develop and maintain this approach. For obvious reasons,
Facebook does not use PyTorch on MCUs, but the embedded community is welcome
to contribute and add the end-to-end support for MCUs and embedded platforms
in general. I suspect that this is only a matter of time because of the
growing attraction to PyTorch, especially among academic and research users.
According to statistics1, PyTorch’s dominance is strongest
at vision and language conferences (outnumbering TensorFlow by 2:1 and 3:1
respectively), and PyTorch has also become more popular than TensorFlow at
general machine learning conferences like
ICLR
and
ICML. Eventually some of these researchers will migrate into the industrial space
and adapt PyTorch for the edge computing environment.
To specifically address the question of PyTorch as a good choice for MCUs,
since it can generate ONNX models which can be compiled by Glow, processing
platform restrictions are minimal. And with Glow as an extension of PyTorch,
it will be even easier to generate bundles. The user can generate bundles
directly from the Python script, without having to first generate ONNX models.
Soon, NXP will release an application note that provides guidance to create
and deploy a model using PyTorch and Glow.
We’ve recently rolled out our official support for Glow. In fact,
it’s tightly integrated into our MCUXpresso SDK with several project
examples. We’ve also packaged the Glow compile and quantization tools
into an easy-to-use installer with detailed documentation and labs to help you
get up and running quickly with your own models. With such significant
performance and memory benefits, this compiler will be a great boon for
embedded system developers deploying machine learning with the i.MX RT series
of crossover MCUs from NXP.
Want to Learn More?
Read the
press release
Explore our eIQ® Machine Learning software development
environment
Visit
Pytorch
1 Visit
The Gradient
