TensorFlow Lite delegate
========================

Mesa contains a TensorFlow Lite delegate that can make use of NPUs to accelerate ML inference. It is implemented in the form of a *external delegate*, a shared library that the TensorFlow Lite runtime can load at startup. See https://www.tensorflow.org/api_docs/python/tf/lite/experimental/load_delegate.

.. list-table:: Supported acceleration hardware
   :header-rows: 1

   * - Gallium driver
     - NPU supported
     - Hardware tested
   * - Etnaviv
     - ``VeriSilicon VIPNano-QI.7120``
     - ``Amlogic A311D on Libre Computer AML-A311D-CC Alta and Khadas VIM3``

.. list-table:: Tested models
   :header-rows: 1

   * - Model name
     - Data type
     - Link (may be outdated)
     - Status
     - Inference speed on AML-A311D-CC Alta
   * - MobileNet V1
     - UINT8
     - http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz
     - Fully supported
     - ~15 ms
   * - MobileNet V2
     - UINT8
     - https://storage.googleapis.com/mobilenet_v2/checkpoints/quantized_v2_224_100.tgz
     - Fully supported
     - ~15.5 ms
   * - SSDLite MobileDet
     - UINT8
     - https://raw.githubusercontent.com/google-coral/test_data/master/ssdlite_mobiledet_coco_qat_postprocess.tflite
     - Fully supported
     - ~53 ms

Build
-----

Build Mesa as usual, with the -Dteflon=true argument.

Example instructions:

.. code-block:: console

   # Install build dependencies
   ~ # apt-get -y build-dep mesa
   ~ # apt-get -y install git cmake

   # Download sources
   ~ $ git clone https://gitlab.freedesktop.org/mesa/mesa.git

   # Build Mesa
   ~ $ cd mesa
   mesa $ meson setup build -Dgallium-drivers=etnaviv -Dvulkan-drivers= -Dteflon=true
   mesa $ meson compile -C build

Install runtime dependencies
----------------------------

Your board should have booted into a mainline 6.7 or greater kernel and have the etnaviv driver loaded. You will also need to enable the NPU device in the device tree by means of an overlay or by a change such as the below (and rebuild the DTB):

.. code-block:: diff

   diff --git a/arch/arm64/boot/dts/amlogic/meson-g12b-a311d-khadas-vim3.dts b/arch/arm64/boot/dts/amlogic/meson-g12b-a311d-khadas-vim3.dts
   index 4aa2b20bfbf2..4e8266056bca 100644
   --- a/arch/arm64/boot/dts/amlogic/meson-g12b-a311d-khadas-vim3.dts
   +++ b/arch/arm64/boot/dts/amlogic/meson-g12b-a311d-khadas-vim3.dts
   @@ -50,6 +50,10 @@ galcore {
         };
   };
   
   +&npu {
   +       status = "okay";
   +};
   +
   /*
   * The VIM3 on-board  MCU can mux the PCIe/USB3.0 shared differential
   * lines using a FUSB340TMX USB 3.1 SuperSpeed Data Switch between


.. code-block:: console

   # Install Python 3.10 and dependencies (as root)
   ~ # echo deb-src http://deb.debian.org/debian testing main >> /etc/apt/sources.list
   ~ # echo deb http://deb.debian.org/debian unstable main >> /etc/apt/sources.list
   ~ # echo 'APT::Default-Release "testing";' >> /etc/apt/apt.conf
   ~ # apt-get update
   ~ # apt-get -y install python3.10 python3-pytest python3-exceptiongroup

   # Install TensorFlow Lite Python package (as non-root)
   ~ $ python3.10 -m pip install --break-system-packages tflite-runtime==2.13.0

Do some inference with MobileNetV1
----------------------------------

.. code-block:: console

   ~ $ cd mesa/
   mesa $ TEFLON_DEBUG=verbose ETNA_MESA_DEBUG=ml_dbgs python3.10 src/gallium/frontends/teflon/tests/classification.py -i ~/tensorflow/assets/grace_hopper.bmp -m src/gallium/targets/teflon/tests/mobilenet_v1_1.0_224_quant.tflite -l src/gallium/frontends/teflon/tests/labels_mobilenet_quant_v1_224.txt -e build/src/gallium/targets/teflon/libteflon.so

   Loading external delegate from build/src/gallium/targets/teflon/libteflon.so with args: {}
   Teflon delegate: loaded etnaviv driver

   teflon: compiling graph: 89 tensors 28 operations
   idx scale     zp has_data size        
   =======================================
   0 0.023528   0 no       1x1x1x1024
   1 0.166099  42 no       1x1x1x1001
   2 0.000117   0 yes      1001x0x0x0
   3 0.004987  4a yes      1001x1x1x1024
   4 0.166099  42 no       1x1001x0x0
   5 0.166099  42 yes      2x0x0x0
   6 0.000171   0 yes      32x0x0x0
   7 0.023528   0 no       1x112x112x32
   8 0.021827  97 yes      32x3x3x3
   9 0.023528   0 no       1x14x14x512
   ...

   idx type    in out  operation type-specific
   ================================================================================================
   0 CONV    88   7  w: 8 b: 6 stride: 2 pad: SAME
   1 DWCONV   7  33  w: 35 b: 34 stride: 1 pad: SAME
   2 CONV    33  37  w: 38 b: 36 stride: 1 pad: SAME
   3 DWCONV  37  39  w: 41 b: 40 stride: 2 pad: SAME
   4 CONV    39  43  w: 44 b: 42 stride: 1 pad: SAME
   5 DWCONV  43  45  w: 47 b: 46 stride: 1 pad: SAME
   6 CONV    45  49  w: 50 b: 48 stride: 1 pad: SAME
   7 DWCONV  49  51  w: 53 b: 52 stride: 2 pad: SAME
   8 CONV    51  55  w: 56 b: 54 stride: 1 pad: SAME
   9 DWCONV  55  57  w: 59 b: 58 stride: 1 pad: SAME
   10 CONV    57  61  w: 62 b: 60 stride: 1 pad: SAME
   11 DWCONV  61  63  w: 65 b: 64 stride: 2 pad: SAME
   12 CONV    63  67  w: 68 b: 66 stride: 1 pad: SAME
   13 DWCONV  67  69  w: 71 b: 70 stride: 1 pad: SAME
   14 CONV    69  73  w: 74 b: 72 stride: 1 pad: SAME
   15 DWCONV  73  75  w: 77 b: 76 stride: 1 pad: SAME
   16 CONV    75  79  w: 80 b: 78 stride: 1 pad: SAME
   17 DWCONV  79  81  w: 83 b: 82 stride: 1 pad: SAME
   18 CONV    81  85  w: 86 b: 84 stride: 1 pad: SAME
   19 DWCONV  85   9  w: 11 b: 10 stride: 1 pad: SAME
   20 CONV     9  13  w: 14 b: 12 stride: 1 pad: SAME
   21 DWCONV  13  15  w: 17 b: 16 stride: 1 pad: SAME
   22 CONV    15  19  w: 20 b: 18 stride: 1 pad: SAME
   23 DWCONV  19  21  w: 23 b: 22 stride: 2 pad: SAME
   24 CONV    21  25  w: 26 b: 24 stride: 1 pad: SAME
   25 DWCONV  25  27  w: 29 b: 28 stride: 1 pad: SAME
   26 CONV    27  31  w: 32 b: 30 stride: 1 pad: SAME
   27 POOL    31   0  filter: 0x0 stride: 0 pad: VALID

   teflon: compiled graph, took 10307 ms
   teflon: invoked graph, took 21 ms
   teflon: invoked graph, took 17 ms
   teflon: invoked graph, took 17 ms
   teflon: invoked graph, took 17 ms
   teflon: invoked graph, took 16 ms
   0.866667: military uniform
   0.031373: Windsor tie
   0.015686: mortarboard
   0.007843: bow tie
   0.007843: academic