[libcamera-devel] [PATCH v9] Documentation: Add IPA writers guide

[Kieran Bingham]

Hi Paul,

On 08/03/2021 07:50, Paul Elder wrote:
> Add a guide about writing IPAs.
> 
> Signed-off-by: Paul Elder <paul.elder at ideasonboard.com>
> 
> ---
> Changes in v9:
> - update documentation to include customizable init() and direct return
>   of int32
> 
> No change in v8
> 
> Changes in v8:
> - fix bullet points
> - update wording about struct field names
> - fix typos
> 
> Changes in v7:
> - fix TODO syntax
> - update the generated struct fiels
>   - no more postfix underscore
> 
> Changes in v6:
> - namespacing is now required
> - start() is now customizable
> - {pipeline_name} is no longer required
> - fix code block indentations
> 
> Changes in v5:
> - fix doxygen compile errors
> - update example struct names from raspberry pi changes
> - add todo for restricting pre-start() to sync and post-start() to async
> 
> Changes in v4.1:
> - Add section on namespacing, custom data structures, compiling, and
>   usage of the data structures and interface
> - Add examples to the main ipa interface and event ipa interface
>   sections
> 
> New in v4
> ---
>  Documentation/guides/ipa.rst | 508 +++++++++++++++++++++++++++++++++++
>  Documentation/index.rst      |   1 +
>  Documentation/meson.build    |   1 +
>  3 files changed, 510 insertions(+)
>  create mode 100644 Documentation/guides/ipa.rst
> 
> diff --git a/Documentation/guides/ipa.rst b/Documentation/guides/ipa.rst
> new file mode 100644
> index 00000000..5590d2d5
> --- /dev/null
> +++ b/Documentation/guides/ipa.rst
> @@ -0,0 +1,508 @@
> +.. SPDX-License-Identifier: CC-BY-SA-4.0
> +
> +IPA Writers Guide
> +=================
> +
> +IPA are Image Processing Algorithm modules. They provide functionality that

'IPA are' sounds odd.

'IPA modules are' ?

> +the pipeline handler can use for image processing.
> +
> +This guide so far only covers definition the IPA interface, and how to plumb

definition the ??


'the definition of the' perhaps?


> +the connection between the pipeline handler and the IPA.
> +
> +The IPA interface and protocol
> +------------------------------
> +
> +The IPA interface defines the interface between the pipeline handler and the
> +IPA. Specifically, it defines the functions that the IPA exposes that the
> +pipeline handler can call, and the Signals that the pipeline handler can
> +connect to to receive data from the IPA asynchronously. In addition, it

to to ?

to, in order to


> +contains any custom data structures that the pipeline handler and IPA may
> +pass to each other.
> +
> +The IPA protocol refers to the agreement between the pipeline handler and the
> +IPA regarding the expected response(s) from the IPA for given calls to the IPA.
> +This protocol doesn't need to be declared anywhere in code, but it would be
> +useful to document it, as there may be many different IPAs for one pipeline

/many different IPAs/ multiple IPA implementations / ?

> +handler.
> +
> +The IPA interface must be defined in a mojom file. The interface includes:
> +
> +- the functions that the pipeline handler can call from the IPA
> +
> +- Signals in the pipeline handler that the IPA can emit
> +
> +- any data structures that are to be passed between the pipeline handler and the IPA
> +
> +All IPA of a given pipeline handler use the same IPA interface. The IPA

All IPA modules of ...

> +interface definition is thus likely to be written by the pipeline handler
> +author, based on how they imagine the pipeline handler will interact with
> +the IPA.
> +
> +The entire IPA interface, including the functions, Signals, and any custom
> +structs shall be defined in in a file named {pipeline_name}.mojom under

in in


> +include/libcamera/ipa/ using the mojo interface definition language (IDL). This
> +will be covered in detail in the following sections.
> +
> +Namespacing
> +-----------
> +
> +Namespacing is required, to avoid potential collisions with libcamera types.
> +Use mojo's module directive for this.
> +
> +It must be the first meaningful line in the mojo data definition file, for
> +example (defining a raspberry pi IPA):
> +
> +.. code-block:: none
> +
> +        module ipa.rpi;
> +
> +This will become the ipa::rpi namespace in C++ code.
> +
> +Data containers
> +---------------
> +
> +Since the data passed between the pipeline handler and the IPA must support
> +serialization, any custom data containers must be defined with the mojo IDL.
> +
> +The following list of libcamera objects are supported in the interface
> +definition, and may be used as function parameter types or struct field types:
> +
> +- CameraSensorInfo
> +
> +- ControlInfoMap
> +
> +- ControlList
> +
> +- FileDescriptor
> +
> +- IPABuffer
> +
> +- IPASettings
> +
> +- IPAStream
> +
> +- Point
> +
> +- Size
> +
> +- SizeRange
> +
> +- Rectangle
> +
> +To use them, core.mojom must be included in the mojo data definition file:
> +
> +.. code-block:: none
> +
> +        import "include/libcamera/ipa/core.mojom";
> +
> +Other custom structs may be defined and used as well. There is no requirement
> +that they must be defined before usage. enums and structs are supported.
> +
> +The following is an example of a definition of an enum, for the purpose of
> +being used as flags:
> +
> +.. code-block:: none
> +
> +        enum ConfigParameters {
> +                ConfigLsTable = 0x01,
> +                ConfigStaggeredWrite = 0x02,
> +                ConfigSensor = 0x04,
> +                ConfigDropFrames = 0x08,
> +        };
> +
> +The following is an example of a definition of a struct:
> +
> +.. code-block:: none
> +
> +        struct ConfigInput {
> +                uint32 op;
> +                uint32 transform;
> +                FileDescriptor lsTableHandle;
> +                int32 lsTableHandleStatic = -1;
> +                map<uint32, IPAStream> streamConfig;
> +                array<IPABuffer> buffers;
> +        };
> +
> +This example has some special things about it. First of all, it uses the
> +FileDescriptor data type. This type must be used to ensure that the file
> +descriptor that it contains is translated property across the IPC boundary
> +(when the IPA is in an isolated process).
> +
> +This does mean that if the file descriptor should be sent without being
> +translated (for example, for the IPA to tell the pipeline handler which
> +fd *that the pipeline handler holds* to act on), then it must be in a
> +regular int32 type.
> +
> +This example also illustrates that struct fields may have default values, as
> +is assigned to lsTableHandleStatic. This is the value that the field will
> +take when the struct is constructed with the default constructor.
> +
> +Arrays and maps are supported as well. They are translated to C++ vectors and
> +maps, respectively. The members of the arrays and maps are embedded, and cannot
> +be const.
> +
> +Note that nullable fields, static-length arrays, handles, and unions, which
> +are supported by mojo, are not supported by our code generator.
> +
> +TODO: what about versioning, and numbered fields?

Is this a comment or the rst? or expected to be viewable in the end
document until it's handled?


> +
> +The Main IPA interface
> +----------------------
> +
> +The IPA interface is split in two parts, the Main IPA interface, which
> +describes the functions that the pipeline handler can call from the IPA,
> +and the Event IPA interface, which describes the Signals in the pipeline
> +handler that the IPA can emit. Both must be defined. This section focuses
> +on the Main IPA interface.
> +
> +The main interface must be named as IPA{pipeline_name}Interface.
> +
> +At minimum, the following three functions must be present (and implemented):

At a minimum

> +
> +- init();
> +
> +- start();
> +
> +- stop();
> +
> +All three of these functions are synchronous.
> +
> +TODO: Restrict pre-start to synchronous, and post-start to asynchronous
> +
> +The parameters for start() and init() may be customized.
> +
> +A configure() method is recommended. Any ContolInfoMap instances that will be
> +used by the IPA must be sent to the IPA from the pipeline handler, at configure
> +time, for example.
> +
> +All input parameters will become const references, except for arithmetic types,
> +which will be passed by value. Output parameters will become pointers, unless
> +the first output parameter is an int32, or there is only one primitive output
> +parameter, in which case it will become a a regular return value.

a a

Seems we have a word duplicator here somewhere ;-)



> +
> +const is not allowed inside of arrays and maps. mojo arrays will become C++
> +std::vector<>.
> +
> +By default, all methods defined in the main interface are synchronous. This
> +means that in the case of IPC (ie. isolated IPA), the function call will not

/ie./i.e./


> +return until the return value or output parameters are ready. To specify an
> +asynchronous function, the [async] attribute can be used. Asynchronous
> +methods must not have any return value or output parameters, since in the
> +case of IPC the call needs to return immediately.
> +
> +It is also possible that the IPA will not be run in isolation. In this case,
> +the IPA thread will not exist until start() is called. This means that in the
> +case of no isolation, asynchronous calls cannot be made before start(). Since
> +the IPA interface must be the same regardless of isolation, the same
> +restriction applies to the case of isolation, and any function that will be
> +called before start() must be synchronous.
> +
> +In addition, any call made after start() and before stop() must be
> +asynchronous. The motivation for this is to avoid damaging real time

real-time here I think. Otherwise it could mis-read as 'real .... time
performance'


> +performance of the pipeline handler. If the pipeline handler wants some data
> +from the IPA, the IPA should return the data asynchronously via an event
> +(see "The Event IPA interface").
> +
> +The following is an example of a main interface definition:
> +
> +.. code-block:: none
> +
> +        interface IPARPiInterface {
> +                init(IPASettings settings, string sensorName)
> +                        => (int32 ret, bool metadataSupport);
> +                start() => (int32 ret);
> +                stop();
> +
> +                configure(CameraSensorInfo sensorInfo,
> +                          map<uint32, IPAStream> streamConfig,
> +                          map<uint32, ControlInfoMap> entityControls,
> +                          ConfigInput ipaConfig)
> +                => (int32 ret, ConfigOutput results);
> +
> +                mapBuffers(array<IPABuffer> buffers);
> +                unmapBuffers(array<uint32> ids);
> +
> +                [async] signalStatReady(uint32 bufferId);
> +                [async] signalQueueRequest(ControlList controls);
> +                [async] signalIspPrepare(ISPConfig data);
> +        };
> +
> +
> +The first three functions are the required functions. Functions do not need to
> +have return values, like stop(), mapBuffers(), and unmapBuffers(). In the case
> +of asynchronous functions, as explained before, they *must not* have return
> +values.
> +
> +The Event IPA interface
> +-----------------------
> +
> +The event IPA interface describes the Signals in the pipeline handler that the
> +IPA can emit. It must be defined. If there are no event functions, then it may
> +be empty. These emissions are meant to notify the pipeline handler of some
> +event, such as reqeust data is ready, and *must not* be used to drive the

request

> +camera pipeline from the IPA.
> +
> +The event interface must be named as IPA{pipeline_name}EventInterface.
> +
> +Methods defined in the event interface are implictly asynchronous.

implicitly


> +They thus cannot return any value. Specifying the [async] tag is not

They thus? or Thus they?

> +necessary.
> +
> +Methods defined in the event interface will become Signals in the IPA
> +interface. The IPA can emit signals, while the pipeline handler can connect
> +slots to them.
> +
> +The following is an example of an event interface definition:
> +
> +.. code-block:: none
> +
> +        interface IPARPiEventInterface {
> +                statsMetadataComplete(uint32 bufferId, ControlList controls);
> +                runIsp(uint32 bufferId);
> +                embeddedComplete(uint32 bufferId);
> +                setIsp(ControlList controls);
> +                setStaggered(ControlList controls);
> +        };
> +
> +Compiling the IPA interface
> +---------------------------
> +
> +After the IPA interface is defined in include/libcamera/ipa/{pipeline_name}.mojom,
> +an entry for it must be added in meson so that it can be compiled. The filename
> +must be added to the ipa_mojom_files object in include/libcamera/ipa/meson.build.
> +
> +For example, adding the raspberrypi.mojom file to meson:
> +
> +.. code-block:: none
> +
> +        ipa_mojom_files = [
> +            'raspberrypi.mojom',
> +        ]
> +
> +This will cause the mojo data definition file to be compiled. Specifically, it
> +generates five files:
> +
> +- a header describing the custom data structures, and the complete IPA interface
> +
> +- a serializer implementing de/serialization for the custom data structures
> +
> +- a proxy header describing a specialized IPA proxy
> +
> +- a proxy source implementing the IPA proxy
> +
> +- a proxy worker source implementing the other end of the IPA proxy
> +
> +The pipeline handler and the IPA only require the header and the proxy header.
> +The serializer is only used internally by the proxy.
> +
> +Using the custom data structures
> +--------------------------------
> +
> +To use the custom data structures that are defined in the mojo data definition
> +file, the following header must be included:
> +
> +.. code-block:: C++
> +
> +   #include <libcamera/ipa/{pipeline_name}_ipa_interface.h>
> +
> +The POD types of the structs simply become their C++ counterparts, eg. uint32
> +in mojo will become uint32_t in C++. mojo map becomes C++ std::map, and mojo
> +array becomes C++ std::vector. All members of maps and vectors are embedded,
> +and are not pointers. The members cannot be const.
> +
> +The names of all the fields of structs can be used in C++ exactly the same way

'in C++ exactly the same' sounds odd?

'can be used in C++ in exactly the same way...' ?

Not sure how necessary but in my head I think I need the 'in' or
something there.


> +they are defined in the data definition file. For example, the following
> +struct as defined in the mojo file:
> +
> +.. code-block:: none
> +
> +   struct SensorConfig {
> +        uint32 gainDelay = 1;
> +        uint32 exposureDelay;
> +        uint32 sensorMetadata;
> +   };
> +
> +Will become this in C++:
> +
> +.. code-block:: C++
> +
> +   struct SensorConfig {
> +        uint32_t gainDelay;
> +        uint32_t exposureDelay;
> +        uint32_t sensorMetadata;
> +   };

does uint32 really get converted to uint32_t ? or is that a copy/paste
issue?



> +
> +The generated structs will also have two constructors, a constructor that
> +fills all fields with the default values, and a second constructor that takes
> +a value for every field. The default value constructor will fill in the fields
> +with the specified default value, if it exists. In the above example, `gainDelay_`

/value,/value/

> +will be initialized to 1. If no default value is specified, then it will be
> +filled in as zero (or -1 for a FileDescriptor type).
> +
> +All fields and constructors/deconstructors in these generated structs are public.

Shouldn't deconstructor be destructor?

> +
> +Using the IPA interface (pipeline handler)
> +------------------------------------------
> +
> +The following headers are necessary to use an IPA in the pipeline handler
> +(with raspberrypi as an example):
> +
> +.. code-block:: C++
> +
> +   #include <libcamera/ipa/raspberrypi_ipa_interface.h>
> +   #include <libcamera/ipa/raspberrypi_ipa_proxy.h>
> +
> +The first header includes definitions of the custom data structures, and
> +the definition of the complete IPA interface (including both the Main and
> +the Event IPA interfaces). The name of the header file comes from the name
> +of the mojom file, which in this case was raspberrypi.mojom.
> +
> +The second header inclues the definition of the specialized IPA proxy. It

includes


> +exposes the complete IPA interface. We will see how to use it in this section.
> +
> +In the pipeline handler, we first need to construct a specialized IPA proxy.
> +From the point of view of the pipeline hander, this is the object that is the
> +IPA.
> +
> +To do so, we invoke the IPAManager:
> +
> +.. code-block:: C++
> +
> +        std::unique_ptr<ipa::rpi::IPAProxyRPi> ipa_ =
> +                IPAManager::createIPA<ipa::rpi::IPAProxyRPi>(pipe_, 1, 1);
> +
> +The ipa::rpi namespace comes from the namespace that we defined in the mojo
> +data definition file, in the "Namespacing" section. The name of the proxy,
> +IPAProxyRPi, comes from the name given to the main IPA interface,
> +IPARPiInterface, in the "The Main IPA interface" section.
> +
> +The return value of IPAManager::createIPA shall be error-checked, to confirm
> +that the returned pointer is not a nullptr.
> +
> +After this, before initializing the IPA, slots should be connected to all of
> +the IPA's Signals, as defined in the Event IPA interface:
> +
> +.. code-block:: C++
> +
> +	ipa_->statsMetadataComplete.connect(this, &RPiCameraData::statsMetadataComplete);
> +	ipa_->runIsp.connect(this, &RPiCameraData::runIsp);
> +	ipa_->embeddedComplete.connect(this, &RPiCameraData::embeddedComplete);
> +	ipa_->setIsp.connect(this, &RPiCameraData::setIsp);
> +	ipa_->setStaggered.connect(this, &RPiCameraData::setStaggered);
> +
> +The slot functions have a function signature based on the function definition
> +in the Event IPA interface. All plain old data (POD) types are as-is (with
> +their C++ versions, eg. uint32 -> uint32_t), and all structs are const references.

Ahh ok - so it was real above.


> +
> +For example, for the following entry in the Event IPA interface:
> +
> +.. code-block:: none
> +
> +   statsMetadataComplete(uint32 bufferId, ControlList controls);
> +
> +A function with the following function signature shall be connected to the
> +signal:
> +
> +.. code-block:: C++
> +
> +   statsMetadataComplete(uint32_t bufferId, const ControlList &controls);
> +
> +After connecting the slots to the signals, the IPA should be initialized
> +(using the main interface definition example from earlier):
> +
> +.. code-block:: C++
> +
> +   IPASettings settings{};
> +   bool metadataSupport;
> +   int ret = ipa_->init(settings, "sensor name", &metadataSupport);
> +
> +At this point, any IPA functions that were defined in the Main IPA interface
> +can be called as if they were regular member functions, for example (based on
> +the main interface definition example from earlier):
> +
> +.. code-block:: C++
> +
> +   ipa_->start();
> +   int ret = ipa_->configure(sensorInfo_, streamConfig, entityControls, ipaConfig, &result);
> +   ipa_->signalStatReady(RPi::BufferMask::STATS | static_cast<unsigned int>(index));
> +
> +Remember that any functions designated as asynchronous *must not* be called
> +before start().

Can we trap this in anyway in the code to signal an error?

I.e. can the proxies have a flag 'running' and only sync functions can
run when running == false, and async functions when running == true?

(An idea to discuss separately, not an update to this patch).


> +
> +Notice that for both init() and configure(), the first output parameter is a
> +direct return, since it is an int32, while the other output parameter is a
> +pointer-based output parameter.
> +
> +TODO: anything special about start() and stop() ?

They're both synchronous? Stop must have stopped everything running, and
no events should be signalled to the pipeline handler after this point?



> +
> +Using the IPA interface (IPA)
> +-----------------------------
> +
> +The following header is necessary to implement an IPA (with raspberrypi as
> +an example):
> +
> +.. code-block:: C++
> +
> +   #include <libcamera/ipa/raspberrypi_ipa_interface.h>
> +
> +This header includes definitions of the custom data structures, and
> +the definition of the complete IPA interface (including both the Main and
> +the Event IPA interfaces). The name of the header file comes from the name
> +of the mojom file, which in this case was raspberrypi.mojom.
> +
> +The IPA must implement the IPA interface class that is defined in the header.
> +In the case of our example, that is ipa::rpi::IPARPiInterface. The ipa::rpi
> +namespace comes from the namespace that we defined in the mojo data definition
> +file, in the "Namespacing" section. The name of the interface is the same as
> +the name given to the Main IPA interface.
> +
> +The function signature rules are the same as for the slots in the pipeline
> +handler side; PODs are passed by value, and structs are passed by const
> +reference. For the Main IPA interface, output values are also allowed (only
> +for synchronous calls), so there may be output parameters as well. If the
> +output parameter is a single POD it will be returned by value, otherwise
> +(multiple PODs or struct(s)) it will be returned by output parameter pointers.
> +
> +For example, for the following function specification in the Main IPA interface
> +definition:
> +
> +.. code-block:: none
> +
> +   configure(CameraSensorInfo sensorInfo,
> +             uint32 exampleNumber,
> +             map<uint32, IPAStream> streamConfig,
> +             map<uint32, ControlInfoMap> entityControls,
> +             ConfigInput ipaConfig)
> +   => (int32 ret, ConfigOutput results);
> +
> +We will need to implement a function with the following function signature:
> +
> +.. code-block:: C++
> +
> +	int configure(const CameraSensorInfo &sensorInfo,
> +                      uint32_t exampleNumber,
> +		      const std::map<unsigned int, IPAStream> &streamConfig,

some odd indentation above on exampleNumber?

> +		      const std::map<unsigned int, ControlInfoMap> &entityControls,
> +		      const ipa::rpi::ConfigInput &data,
> +		      ipa::rpi::ConfigOutput *response);
> +
> +The return value is int, because the first output parameter is int32.  The rest
> +of the output parameters (in this case, only response) become output parameter
> +pointers. The non-POD input parameters become const references, and the POD
> +input parameter is passed by value.
> +
> +At any time (though usually only in response to an IPA call), the IPA may send
> +data to the pipeline handler by emitting signals. These signals are defined
> +in the C++ IPA interface class (which is in the generated and included header).
> +
> +For example, for the following function defined in the Event IPA interface:
> +
> +.. code-block:: none
> +
> +   statsMetadataComplete(uint32 bufferId, ControlList controls);
> +
> +We can emit a signal like so:
> +
> +.. code-block:: C++
> +
> +   statsMetadataComplete.emit(bufferId & RPi::BufferMask::ID, libcameraMetadata_);
> diff --git a/Documentation/index.rst b/Documentation/index.rst
> index 285ca7c3..b40a4586 100644
> --- a/Documentation/index.rst
> +++ b/Documentation/index.rst
> @@ -16,6 +16,7 @@
>     Developer Guide <guides/introduction>
>     Application Writer's Guide <guides/application-developer>
>     Pipeline Handler Writer's Guide <guides/pipeline-handler>
> +   IPA Writer's guide <guides/ipa>
>     Tracing guide <guides/tracing>
>     Environment variables <environment_variables>
>     Sensor driver requirements <sensor_driver_requirements>
> diff --git a/Documentation/meson.build b/Documentation/meson.build
> index 9950465d..8cf68a07 100644
> --- a/Documentation/meson.build
> +++ b/Documentation/meson.build
> @@ -54,6 +54,7 @@ if sphinx.found()
>          'environment_variables.rst',
>          'guides/application-developer.rst',
>          'guides/introduction.rst',
> +        'guides/ipa.rst',
>          'guides/pipeline-handler.rst',
>          'guides/tracing.rst',
>          'index.rst',


With all that, throw a:

Reviewed-by: Kieran Bingham <kieran.bingham at ideasonboard.com>

on there.



-- 
Regards
--
Kieran