[libcamera-devel] [RFC PATCH] Documentation: IPU3 IPA Design guide

[Kieran Bingham]

The IPU3 IPA implements the basic 3a using the ImgU ISP.

Provide an overview document to describe it's operations, and provide a
block diagram to help visualise how the components are put together to
assist any new developers exploring the code.

Signed-off-by: Kieran Bingham <kieran.bingham at ideasonboard.com>

---
This is really just an RFC: Particularly:

 - Should this content actually live in Documentation?
   or is it more suited to a page under src/ipa/ipu3/...

 - This is not complete to document all functionality
   but gives an overview of the current implementation

 - Is there anything anyone would find helpful for me to
   extend/write about on this?

 - It could likely get merged in with some of the work
   that Jean-Michel is doing, and might duplicate some of
   the parts he is documenting, but this was an aim to
   write a single page overview as a getting started with
   the IPU3 IPA ....


 .../guides/ipu3-ipa-design-guide.rst          | 144 ++++++++++++++++++
 1 file changed, 144 insertions(+)
 create mode 100644 Documentation/guides/ipu3-ipa-design-guide.rst

diff --git a/Documentation/guides/ipu3-ipa-design-guide.rst b/Documentation/guides/ipu3-ipa-design-guide.rst
new file mode 100644
index 000000000000..a1d0f13fbb00
--- /dev/null
+++ b/Documentation/guides/ipu3-ipa-design-guide.rst
@@ -0,0 +1,144 @@
+IPU3 IPA Architecture Design and Overview
+=========================================
+
+The IPU3 IPA is built as a modular and extensible framework with an
+upper layer to manage the interactions with the pipeline handler, and
+the image processing algorithms split to compartmentalise the processing
+required for each accellerator cluster provided by the ImgU ISP.
+
+The core IPU3 class is responsible for initialisation and construction
+of the algorithm components, processing controls set by the requests
+from applications, and managing events from the Pipeline handler.
+
+::
+
+        ┌───────────────────────────────────────────┐
+        │      IPU3 Pipeline Handler                │
+        │   ┌────────┐    ┌────────┐    ┌────────┐  │
+        │   │        │    │        │    │        │  │
+        │   │ Sensor ├───►│  CIO2  ├───►│  ImgU  ├──►
+        │   │        │    │        │    │        │  │
+        │   └────────┘    └────────┘    └─▲────┬─┘  │    P: Parameter Buffer
+        │                                 │P   │    │    S: Statistics Buffer
+        │                                 │    │S   │
+        └─┬───┬───┬──────┬────┬────┬────┬─┴────▼─┬──┘    1: init()
+          │   │   │      │ ▲  │ ▲  │ ▲  │ ▲      │       2: configure()
+          │1  │2  │3     │4│  │4│  │4│  │4│      │5      3: mapBuffers(), start()
+          ▼   ▼   ▼      ▼ │  ▼ │  ▼ │  ▼ │      ▼       4: processEvent()
+        ┌──────────────────┴────┴────┴────┴─────────┐    5: stop(), unmapBuffers()
+        │ IPU3 IPA                                  │
+        │                 ┌───────────────────────┐ │
+        │ ┌───────────┐   │ Algorithms            │ │
+        │ │IPAContext │   │          ┌─────────┐  │ │
+        │ │ ┌───────┐ │   │          │ ...     │  │ │
+        │ │ │       │ │   │        ┌─┴───────┐ │  │ │
+        │ │ │  SC   │ │   │        │ Tonemap ├─┘  │ │
+        │ │ │       │ ◄───►      ┌─┴───────┐ │    │ │
+        │ │ ├───────┤ │   │      │ AWB     ├─┘    │ │
+        │ │ │       │ │   │    ┌─┴───────┐ │      │ │
+        │ │ │  FC   │ │   │    │ AGC     ├─┘      │ │
+        │ │ │       │ │   │    │         │        │ │
+        │ │ └───────┘ │   │    └─────────┘        │ │
+        │ └───────────┘   └───────────────────────┘ │
+        └───────────────────────────────────────────┘
+         SC: IPASessionConfiguration
+         FC: IPAFrameContext(s)
+
+The IPA instance is constructed and initialised at the point a Camera is
+created by the IPU3 pipeline handler. The initialisation call provides
+details about which Camera Sensor is being used, and the controls that
+it has available, along with their defaults and ranges.
+
+Buffers
+~~~~~~~
+
+The IPA will have Parameter and Statistics buffers shared with it from
+the IPU3 Pipeline handler. These buffers will be passed to the IPA
+before the ``start()`` operation occurs.
+
+The IPA will map the buffers into CPU accessible memory, associated with
+a buffer ID, and further events for sending or receiving parameter and
+statistics buffers will reference the ID to avoid expensive memory
+mapping operations, or the passing of file handles during streaming.
+
+After the ``stop()`` operation occurs, these buffers will be unmapped,
+and no further access to the buffers is permitted.
+
+Context
+~~~~~~~
+
+Algorithm calls will always have the ``IPAContext`` available to them.
+This context comprises of two parts:
+
+-  IPA Session Configuration
+-  IPA Frame Context
+
+The session configuration structure ``IPASessionConfiguration``
+represents constant parameters determined from either before streaming
+commenced during ``configure()`` or updated parameters when processing
+controls.
+
+The IPA Frame Context provides the storage for algorithms for a single
+frame operation.
+
+The ``IPAFrameContext`` structure may be extended to an array, list, or
+queue to store historical state for each frame, allowing algorithms to
+obtain and reference results of calculations which are deeply pipelined.
+This may only be done if an algorithm needs to know the context that was
+applied at the frame the statistics were produced for, rather than the
+previous or current frame.
+
+Presently there is a single ``IPAFrameContext`` without historical data,
+and the context is maintained and updated through successive processing
+operations.
+
+Operating
+~~~~~~~~~
+
+There are three main parts of interactions with the algorithms for the
+IPU3 IPA to operate when running:
+
+-  configure()
+-  processEvent(``EventFillParams``)
+-  processEvent(``EventStatReady``)
+
+The configuration phase allows the pipeline-handler to inform the IPA of
+the current stream configurations, which is then passed into each
+algorithm to provide an opportunity to identify and track state of the
+hardware, such as image size or ImgU pipeline configurations.
+
+Pre-frame preparation
+~~~~~~~~~~~~~~~~~~~~~
+
+When configured, the IPA is notified by the pipeline handler of the
+Camera ``start()`` event, after which incoming requests will be queued
+for processing, requiring a parameter buffer (``ipu3_uapi_params``) to
+be populated for the ImgU. This is passed directly to each algorithm
+through the ``prepare()`` call allowing the ISP configuration to be
+updated for the needs of each component that the algorithm is
+responsible for.
+
+The algorithm should set the use flag (``ipu3_uapi_flags``) for any
+structure that it modifies, and it should take care to ensure that any
+structure set by a use flag is fully initialised to suitable values.
+
+The parameter buffer is returned to the pipeline handler through the
+``ActionParamFilled`` event, and from there queued to the ImgU along
+with a raw frame captured with the CIO2.
+
+Post-frame completion
+~~~~~~~~~~~~~~~~~~~~~
+
+When the capture of an image is completed, and successfully processed
+through the ImgU, the generated statistics buffer
+(``ipu3_uapi_stats_3a``) is given to the IPA through the
+``EventStatReady`` event. This provides the IPA with an opportunity to
+examine the results of the ISP and run the calculations required by each
+algorithm on the new data. The algorithms may require context from the
+operations of other algorithms, for example, the AWB might choose to use
+a scene brightness determined by the AGC. It is important that the
+algorithms are ordered to ensure that required results are determined
+before they are needed.
+
+The ordering of the algorithm processing is determined by their
+placement in the ``IPU3::algorithms_`` ordered list.
-- 
2.25.1