[libcamera-devel] [RFC PATCH] Documentation: IPU3 IPA Design guide

[Laurent Pinchart]

Hi Kieran,

On Thu, Sep 16, 2021 at 01:36:43PM +0100, Kieran Bingham wrote:
> On 13/09/2021 15:29, Umang Jain wrote:
> > On 9/13/21 3:00 AM, Kieran Bingham wrote:
> >> The IPU3 IPA implements the basic 3a using the ImgU ISP.
> > 3a or 3A?
> >>
> >> Provide an overview document to describe it's operations, and provide a
> >> block diagram to help visualise how the components are put together to
> >> assist any new developers exploring the code.
> >>
> >> Signed-off-by: Kieran Bingham <kieran.bingham at ideasonboard.com>
> >>
> >> ---
> >> This is really just an RFC: Particularly:
> >>
> >>   - Should this content actually live in Documentation?
> >>     or is it more suited to a page under src/ipa/ipu3/...
> > 
> > 
> > I would probably keep it under Documentation/ but there is already a IPA
> > writing guide. So maybe we need a new dir, to document overviews of
> > existing IPAs?
> 
> Indeed, the existing document really describes the IPA IPC interfaces, I
> wonder if it should be renamed ...
> 
> The aim of this is to provide some overview on how the IPU3 IPA operates
> from a higher level.
> 
> Along with the discussions with Laurent, I'm going to move this to
> src/ipa/ipu3/ipu3-ipa-design-guide.rst for now.

I had assumed it would be moved to ipu3.cpp. Seems like explicit
communication would work better than assumptions.

I'm not sure I want to set a precedent of storing .rst documentation
somewhere else than in the Documentation/ directory. If you want to keep
it as .rst, I think it should stay there, and get compiled to catch
errors. My preference would be ipu3.cpp though.

> That will keep the overview document in the actual component for now.
> 
> It may be that we write further 'generic' IPA design guides, which may
> take inspiration from this, and could live in Documentation, but I think
> I/we need to spend some dedicated cycles on how we clean up our
> documentation anyway, so I'd rather keep this file close to it's
> implementation for now.
> 
> >>
> >>   - This is not complete to document all functionality
> >>     but gives an overview of the current implementation
> >>
> >>   - Is there anything anyone would find helpful for me to
> >>     extend/write about on this?
> >>
> >>   - It could likely get merged in with some of the work
> >>     that Jean-Michel is doing, and might duplicate some of
> >>     the parts he is documenting, but this was an aim to
> >>     write a single page overview as a getting started with
> >>     the IPU3 IPA ....
> >>
> >>
> >>   .../guides/ipu3-ipa-design-guide.rst          | 144 ++++++++++++++++++
> >>   1 file changed, 144 insertions(+)
> >>   create mode 100644 Documentation/guides/ipu3-ipa-design-guide.rst
> >>
> >> diff --git a/Documentation/guides/ipu3-ipa-design-guide.rst
> >> b/Documentation/guides/ipu3-ipa-design-guide.rst
> >> new file mode 100644
> >> index 000000000000..a1d0f13fbb00
> >> --- /dev/null
> >> +++ b/Documentation/guides/ipu3-ipa-design-guide.rst
> >> @@ -0,0 +1,144 @@
> >> +IPU3 IPA Architecture Design and Overview
> >> +=========================================
> >> +
> >> +The IPU3 IPA is built as a modular and extensible framework with an
> >> +upper layer to manage the interactions with the pipeline handler, and
> >> +the image processing algorithms split to compartmentalise the processing
> >> +required for each accellerator cluster provided by the ImgU ISP.
> >> +
> >> +The core IPU3 class is responsible for initialisation and construction
> >
> > s/IPU3/IPAIPU3 maybe?
> >
> >> +of the algorithm components, processing controls set by the requests
> >> +from applications, and managing events from the Pipeline handler.
> >> +
> >> +::
> >> +
> >> +        ┌───────────────────────────────────────────┐
> >> +        │      IPU3 Pipeline Handler                │
> >> +        │   ┌────────┐    ┌────────┐    ┌────────┐  │
> >> +        │   │        │    │        │    │        │  │
> >> +        │   │ Sensor ├───►│  CIO2  ├───►│  ImgU  ├──►
> >> +        │   │        │    │        │    │        │  │
> >> +        │   └────────┘    └────────┘    └─▲────┬─┘  │    P: Parameter Buffer
> >> +        │                                 │P   │    │    S: Statistics Buffer
> >> +        │                                 │    │S   │
> >> +        └─┬───┬───┬──────┬────┬────┬────┬─┴────▼─┬──┘    1: init()
> >> +          │   │   │      │ ▲  │ ▲  │ ▲  │ ▲      │       2: configure()
> >> +          │1  │2  │3     │4│  │4│  │4│  │4│      │5      3: mapBuffers(), start()
> >> +          ▼   ▼   ▼      ▼ │  ▼ │  ▼ │  ▼ │      ▼       4: processEvent()
> >> +        ┌──────────────────┴────┴────┴────┴─────────┐    5: stop(), unmapBuffers()
> >> +        │ IPU3 IPA                                  │
> >> +        │                 ┌───────────────────────┐ │
> >> +        │ ┌───────────┐   │ Algorithms            │ │
> >> +        │ │IPAContext │   │          ┌─────────┐  │ │
> >> +        │ │ ┌───────┐ │   │          │ ...     │  │ │
> >> +        │ │ │       │ │   │        ┌─┴───────┐ │  │ │
> >> +        │ │ │  SC   │ │   │        │ Tonemap ├─┘  │ │
> >> +        │ │ │       │ ◄───►      ┌─┴───────┐ │    │ │
> >> +        │ │ ├───────┤ │   │      │ AWB     ├─┘    │ │
> >> +        │ │ │       │ │   │    ┌─┴───────┐ │      │ │
> >> +        │ │ │  FC   │ │   │    │ AGC     ├─┘      │ │
> >> +        │ │ │       │ │   │    │         │        │ │
> >> +        │ │ └───────┘ │   │    └─────────┘        │ │
> >> +        │ └───────────┘   └───────────────────────┘ │
> >> +        └───────────────────────────────────────────┘
> >> +         SC: IPASessionConfiguration
> >> +         FC: IPAFrameContext(s)
> >> +
> >> +The IPA instance is constructed and initialised at the point a Camera is
> >> +created by the IPU3 pipeline handler. The initialisation call provides
> >> +details about which Camera Sensor is being used, and the controls that
> >> +it has available, along with their defaults and ranges.
> > 
> > Brief line about how sensor controls are used in IPA will be useful?
> > This will explain "why" sensor controls are passed to IPA and what IPA
> > does with it (for e.g gather necessary info to run algorithms).
> 
> I have wondered about putting a CameraSensorHelper box in the IPA too,
> but I thought that would be too cluttered, and I don't think it adds
> much to convey the context of the running IPA (at least for the moment) ...
> 
> But it might certainly be worth explaining about how controls are also
> sent to the pipeline handler. I'll see if I can add a section:
> 
> Controls
> ~~~~~~~~
> 
> The AutoExposure and AutoGain (AGC) algorithm differs slightly from the
> others as it requires operating directly on the sensor, as opposed to
> through the ImgU ISP. To support this, there is a dedicated action
> `ActionSetSensorControls` to allow the IPA to request controls to be set
> on the camera sensor through the pipeline handler.
> 
> >> +
> >> +Buffers
> >> +~~~~~~~
> >> +
> >> +The IPA will have Parameter and Statistics buffers shared with it from
> >> +the IPU3 Pipeline handler. These buffers will be passed to the IPA
> >> +before the ``start()`` operation occurs.
> >> +
> >> +The IPA will map the buffers into CPU accessible memory, associated with
> >> +a buffer ID, and further events for sending or receiving parameter and
> >> +statistics buffers will reference the ID to avoid expensive memory
> >> +mapping operations, or the passing of file handles during streaming.
> >> +
> >> +After the ``stop()`` operation occurs, these buffers will be unmapped,
> >> +and no further access to the buffers is permitted.
> >> +
> >> +Context
> >> +~~~~~~~
> >> +
> >> +Algorithm calls will always have the ``IPAContext`` available to them.
> >> +This context comprises of two parts:
> >> +
> >> +-  IPA Session Configuration
> >> +-  IPA Frame Context
> >> +
> >> +The session configuration structure ``IPASessionConfiguration``
> >> +represents constant parameters determined from either before streaming
> >> +commenced during ``configure()`` or updated parameters when processing
> >> +controls.
> >> +
> >> +The IPA Frame Context provides the storage for algorithms for a single
> >> +frame operation.
> >> +
> >> +The ``IPAFrameContext`` structure may be extended to an array, list, or
> >> +queue to store historical state for each frame, allowing algorithms to
> >> +obtain and reference results of calculations which are deeply pipelined.
> >> +This may only be done if an algorithm needs to know the context that was
> >> +applied at the frame the statistics were produced for, rather than the
> >> +previous or current frame.
> >> +
> >> +Presently there is a single ``IPAFrameContext`` without historical data,
> >> +and the context is maintained and updated through successive processing
> >> +operations.
> >> +
> >> +Operating
> >> +~~~~~~~~~
> >> +
> >> +There are three main parts of interactions with the algorithms for the
> >> +IPU3 IPA to operate when running:
> >> +
> >> +-  configure()
> >> +-  processEvent(``EventFillParams``)
> > 
> > 
> > One liner explanation for EventFillParams is missing. It should be
> > present in Pre-frame preparation section, no? Can you please include a
> > brief line analogous to "EventStatReady" event in the Post-frame
> > completion section?
> 
> Ah yes, indeed they should both be covered in the same way, so that's
> missing thanks.
> 
> I've updated it.
> 
> > Other than that, looks fine to me:
> > 
> > Reviewed-by: Umang Jain <umang.jain at ideasonboard.com>
> > 
> >> +-  processEvent(``EventStatReady``)
> >> +
> >> +The configuration phase allows the pipeline-handler to inform the IPA of
> >> +the current stream configurations, which is then passed into each
> >> +algorithm to provide an opportunity to identify and track state of the
> >> +hardware, such as image size or ImgU pipeline configurations.
> >> +
> >> +Pre-frame preparation
> >> +~~~~~~~~~~~~~~~~~~~~~
> >> +
> >> +When configured, the IPA is notified by the pipeline handler of the
> >> +Camera ``start()`` event, after which incoming requests will be queued
> >> +for processing, requiring a parameter buffer (``ipu3_uapi_params``) to
> >> +be populated for the ImgU. This is passed directly to each algorithm
> >> +through the ``prepare()`` call allowing the ISP configuration to be
> >> +updated for the needs of each component that the algorithm is
> >> +responsible for.
> >> +
> >> +The algorithm should set the use flag (``ipu3_uapi_flags``) for any
> >> +structure that it modifies, and it should take care to ensure that any
> >> +structure set by a use flag is fully initialised to suitable values.
> >> +
> >> +The parameter buffer is returned to the pipeline handler through the
> >> +``ActionParamFilled`` event, and from there queued to the ImgU along
> >> +with a raw frame captured with the CIO2.
> >> +
> >> +Post-frame completion
> >> +~~~~~~~~~~~~~~~~~~~~~
> >> +
> >> +When the capture of an image is completed, and successfully processed
> >> +through the ImgU, the generated statistics buffer
> >> +(``ipu3_uapi_stats_3a``) is given to the IPA through the
> >> +``EventStatReady`` event. This provides the IPA with an opportunity to
> >> +examine the results of the ISP and run the calculations required by each
> >> +algorithm on the new data. The algorithms may require context from the
> >> +operations of other algorithms, for example, the AWB might choose to use
> >> +a scene brightness determined by the AGC. It is important that the
> >> +algorithms are ordered to ensure that required results are determined
> >> +before they are needed.
> >> +
> >> +The ordering of the algorithm processing is determined by their
> >> +placement in the ``IPU3::algorithms_`` ordered list.

-- 
Regards,

Laurent Pinchart