[PATCH v4 5/9] Documentation: Remove libcamera architecture from introduction.rst
Laurent Pinchart
laurent.pinchart at ideasonboard.com
Tue Aug 20 18:37:42 CEST 2024
Hi Dan,
Thank you for the patch.
On Tue, Aug 20, 2024 at 02:07:36PM +0100, Daniel Scally wrote:
> The libcamera Architecture section of the introduction is largely a
> duplicate of the section broken out from docs.rst. Remove it from the
> introduction.rst file and consolidate anything that wasn't duplicated
> into libcamera_architecture.rst and feature_requirements.rst. Take the
> opportunity to also expand the list of Platform Support which is now a
> bit out of date.
>
> Signed-off-by: Daniel Scally <dan.scally at ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart at ideasonboard.com>
> ---
> Changes since v3:
>
> - Merged the contents rather than replaced one with the other, following
> Laurent's suggestion. Also merged one section into the page detailing
> feature requirements
>
> Changes since v2:
>
> - Reworked to change libcamera architecture section instead of camera
> stack. DDropped R-b
> - Updated the list of supported platforms on the new libcamera
> architecture page
>
> Changes since v1:
>
> - None
>
> Documentation/feature_requirements.rst | 13 +-
> Documentation/guides/introduction.rst | 156 +--------------
> Documentation/libcamera_architecture.rst | 244 +++++++++++++----------
> 3 files changed, 148 insertions(+), 265 deletions(-)
>
> diff --git a/Documentation/feature_requirements.rst b/Documentation/feature_requirements.rst
> index cae7e9ab..e6b74a62 100644
> --- a/Documentation/feature_requirements.rst
> +++ b/Documentation/feature_requirements.rst
> @@ -90,10 +90,15 @@ on top of the minimum required set for the profile they expose.
> 3A and Image Enhancement Algorithms
> -----------------------------------
>
> -The camera devices shall implement auto exposure, auto gain and auto white
> -balance. Camera devices that include a focus lens shall implement auto
> -focus. Additional image enhancement algorithms, such as noise reduction or
> -video stabilization, may be implemented.
> +The library shall provide a basic implementation of Image Processing Algorithms
> +to serve as a reference for the internal API. This shall including auto exposure
> +and gain and auto white balance. Camera devices that include a focus lens shall
> +implement auto focus. Additional image enhancement algorithms, such as noise
> +reduction or video stabilization, may be implemented. Device vendors are
> +expected to provide a fully-fledged implementation compatible with their
> +Pipeline Handler. One goal of the libcamera project is to create an environment
> +in which the community will be able to compete with the closed-source vendor
> +biaries and develop a high quality open source implementation.
>
> All algorithms may be implemented in hardware or firmware outside of the
> library, or in software in the library. They shall all be controllable by
> diff --git a/Documentation/guides/introduction.rst b/Documentation/guides/introduction.rst
> index d1e67654..12d1b7d4 100644
> --- a/Documentation/guides/introduction.rst
> +++ b/Documentation/guides/introduction.rst
> @@ -26,10 +26,8 @@ desirable results from the camera.
> .. _Media Controller: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html
>
>
> -In this developers guide, we will explore the internal `Architecture`_ of
> -the libcamera library with its components. The current `Platform Support`_ is
> -detailed, as well as an overview of the `Licensing`_ requirements of the
> -project.
> +In this developers guide the `Licensing`_ requirements of the project are
> +detailed.
>
> This introduction is followed by a walkthrough tutorial to newcomers wishing to
> support a new platform with the `Pipeline Handler Writers Guide`_ and for those
> @@ -41,156 +39,6 @@ provides a tutorial of the key APIs exposed by libcamera.
>
> .. TODO: Correctly link to the other articles of the guide
>
> -Architecture
> -------------
> -
> -While offering a unified API towards upper layers, and presenting itself as a
> -single library, libcamera isn't monolithic. It exposes multiple components
> -through its public API and is built around a set of separate helpers internally.
> -Hardware abstractions are handled through the use of device-specific components
> -where required and dynamically loadable plugins are used to separate image
> -processing algorithms from the core libcamera codebase.
> -
> -::
> -
> - --------------------------< libcamera Public API >---------------------------
> - ^ ^
> - | |
> - v v
> - +-------------+ +---------------------------------------------------+
> - | Camera | | Camera Device |
> - | Manager | | +-----------------------------------------------+ |
> - +-------------+ | | Device-Agnostic | |
> - ^ | | | |
> - | | | +--------------------------+ |
> - | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
> - | | | | { +-----------------+ } |
> - | | | | } | //// Image //// | { |
> - | | | | <-> | / Processing // | } |
> - | | | | } | / Algorithms // | { |
> - | | | | { +-----------------+ } |
> - | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
> - | | | | ========================== |
> - | | | | +-----------------+ |
> - | | | | | // Pipeline /// | |
> - | | | | <-> | /// Handler /// | |
> - | | | | | /////////////// | |
> - | | +--------------------+ +-----------------+ |
> - | | Device-Specific |
> - | +---------------------------------------------------+
> - | ^ ^
> - | | |
> - v v v
> - +--------------------------------------------------------------------+
> - | Helpers and Support Classes |
> - | +-------------+ +-------------+ +-------------+ +-------------+ |
> - | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
> - | | Support | | Allocator | | IPC | | Manager | |
> - | +-------------+ +-------------+ +-------------+ +-------------+ |
> - | +-------------+ +-------------+ |
> - | | Pipeline | | ... | |
> - | | Runner | | | |
> - | +-------------+ +-------------+ |
> - +--------------------------------------------------------------------+
> -
> - /// Device-Specific Components
> - ~~~ Sandboxing
> -
> -
> -Camera Manager
> - The Camera Manager enumerates cameras and instantiates Pipeline Handlers to
> - manage each Camera that libcamera supports. The Camera Manager supports
> - hotplug detection and notification events when supported by the underlying
> - kernel devices.
> -
> - There is only ever one instance of the Camera Manager running per application.
> - Each application's instance of the Camera Manager ensures that only a single
> - application can take control of a camera device at once.
> -
> - Read the `Camera Manager API`_ documentation for more details.
> -
> -.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html
> -
> -Camera Device
> - The Camera class represents a single item of camera hardware that is capable
> - of producing one or more image streams, and provides the API to interact with
> - the underlying device.
> -
> - If a system has multiple instances of the same hardware attached, each has its
> - own instance of the camera class.
> -
> - The API exposes full control of the device to upper layers of libcamera through
> - the public API, making it the highest level object libcamera exposes, and the
> - object that all other API operations interact with from configuration to
> - capture.
> -
> - Read the `Camera API`_ documentation for more details.
> -
> -.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html
> -
> -Pipeline Handler
> - The Pipeline Handler manages the complex pipelines exposed by the kernel
> - drivers through the Media Controller and V4L2 APIs. It abstracts pipeline
> - handling to hide device-specific details from the rest of the library, and
> - implements both pipeline configuration based on stream configuration, and
> - pipeline runtime execution and scheduling when needed by the device.
> -
> - The Pipeline Handler lives in the same process as the rest of the library, and
> - has access to all helpers and kernel camera-related devices.
> -
> - Hardware abstraction is handled by device specific Pipeline Handlers which are
> - derived from the Pipeline Handler base class allowing commonality to be shared
> - among the implementations.
> -
> - Derived pipeline handlers create Camera device instances based on the devices
> - they detect and support on the running system, and are responsible for
> - managing the interactions with a camera device.
> -
> - More details can be found in the `PipelineHandler API`_ documentation, and the
> - `Pipeline Handler Writers Guide`_.
> -
> -.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html
> -
> -Image Processing Algorithms
> - An image processing algorithm (IPA) component is a loadable plugin that
> - implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other
> - algorithms.
> -
> - The algorithms run on the CPU and interact with the camera devices through the
> - Pipeline Handler to control hardware image processing based on the parameters
> - supplied by upper layers, maintaining state and closing the control loop
> - of the ISP.
> -
> - The component is sandboxed and can only interact with libcamera through the
> - API provided by the Pipeline Handler and an IPA has no direct access to kernel
> - camera devices.
> -
> - Open source IPA modules built with libcamera can be run in the same process
> - space as libcamera, however external IPA modules are run in a separate process
> - from the main libcamera process. IPA modules have a restricted view of the
> - system, including no access to networking APIs and limited access to file
> - systems.
> -
> - IPA modules are only required for platforms and devices with an ISP controlled
> - by the host CPU. Camera sensors which have an integrated ISP are not
> - controlled through the IPA module.
> -
> -Platform Support
> -----------------
> -
> -The library currently supports the following hardware platforms specifically
> -with dedicated pipeline handlers:
> -
> - - Intel IPU3 (ipu3)
> - - Rockchip RK3399 (rkisp1)
> - - RaspberryPi 3 and 4 (rpi/vc4)
> -
> -Furthermore, generic platform support is provided for the following:
> -
> - - USB video device class cameras (uvcvideo)
> - - iMX7, Allwinner Sun6i (simple)
> - - Virtual media controller driver for test use cases (vimc)
> -
> Licensing
> ---------
>
> diff --git a/Documentation/libcamera_architecture.rst b/Documentation/libcamera_architecture.rst
> index 1258db23..abbb0d17 100644
> --- a/Documentation/libcamera_architecture.rst
> +++ b/Documentation/libcamera_architecture.rst
> @@ -5,124 +5,136 @@
> libcamera Architecture
> ======================
>
> +While offering a unified API towards upper layers, and presenting itself as a
> +single library, libcamera isn't monolithic. It exposes multiple components
> +through its public API and is built around a set of separate helpers internally.
> +Hardware abstractions are handled through the use of device-specific components
> +where required and dynamically loadable plugins are used to separate image
> +processing algorithms from the core libcamera codebase.
> +
> ::
>
> - ---------------------------< libcamera Public API >---------------------------
> - ^ ^
> - | |
> - v v
> - +-------------+ +-------------------------------------------------+
> - | Camera | | Camera Device |
> - | Devices | | +---------------------------------------------+ |
> - | Manager | | | Device-Agnostic | |
> - +-------------+ | | | |
> - ^ | | +------------------------+ |
> - | | | | ~~~~~~~~~~~~~~~~~~~~~ |
> - | | | | { +---------------+ } |
> - | | | | } | ////Image//// | { |
> - | | | | <-> | /Processing// | } |
> - | | | | } | /Algorithms// | { |
> - | | | | { +---------------+ } |
> - | | | | ~~~~~~~~~~~~~~~~~~~~~ |
> - | | | | ======================== |
> - | | | | +---------------+ |
> - | | | | | //Pipeline/// | |
> - | | | | <-> | ///Handler/// | |
> - | | | | | ///////////// | |
> - | | +--------------------+ +---------------+ |
> - | | Device-Specific |
> - | +-------------------------------------------------+
> - | ^ ^
> - | | |
> - v v v
> - +--------------------------------------------------------------------+
> - | Helpers and Support Classes |
> - | +-------------+ +-------------+ +-------------+ +-------------+ |
> - | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
> - | | Support | | Allocator | | IPC | | Manager | |
> - | +-------------+ +-------------+ +-------------+ +-------------+ |
> - | +-------------+ +-------------+ |
> - | | Pipeline | | ... | |
> - | | Runner | | | |
> - | +-------------+ +-------------+ |
> - +--------------------------------------------------------------------+
> -
> - /// Device-Specific Components
> - ~~~ Sandboxing
> -
> -While offering a unified API towards upper layers, and presenting
> -itself as a single library, libcamera isn't monolithic. It exposes
> -multiple components through its public API, is built around a set of
> -separate helpers internally, uses device-specific components and can
> -load dynamic plugins.
> -
> -Camera Devices Manager
> - The Camera Devices Manager provides a view of available cameras
> - in the system. It performs cold enumeration and runtime camera
> - management, and supports a hotplug notification mechanism in its
> - public API.
> -
> - To avoid the cost associated with cold enumeration of all devices
> - at application start, and to arbitrate concurrent access to camera
> - devices, the Camera Devices Manager could later be split to a
> - separate service, possibly with integration in platform-specific
> - device management.
> + --------------------------< libcamera Public API >---------------------------
> + ^ ^
> + | |
> + v v
> + +-------------+ +---------------------------------------------------+
> + | Camera | | Camera Device |
> + | Manager | | +-----------------------------------------------+ |
> + +-------------+ | | Device-Agnostic | |
> + ^ | | | |
> + | | | +--------------------------+ |
> + | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
> + | | | | { +-----------------+ } |
> + | | | | } | //// Image //// | { |
> + | | | | <-> | / Processing // | } |
> + | | | | } | / Algorithms // | { |
> + | | | | { +-----------------+ } |
> + | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
> + | | | | ========================== |
> + | | | | +-----------------+ |
> + | | | | | // Pipeline /// | |
> + | | | | <-> | /// Handler /// | |
> + | | | | | /////////////// | |
> + | | +--------------------+ +-----------------+ |
> + | | Device-Specific |
> + | +---------------------------------------------------+
> + | ^ ^
> + | | |
> + v v v
> + +--------------------------------------------------------------------+
> + | Helpers and Support Classes |
> + | +-------------+ +-------------+ +-------------+ +-------------+ |
> + | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
> + | | Support | | Allocator | | IPC | | Manager | |
> + | +-------------+ +-------------+ +-------------+ +-------------+ |
> + | +-------------+ +-------------+ |
> + | | Pipeline | | ... | |
> + | | Runner | | | |
> + | +-------------+ +-------------+ |
> + +--------------------------------------------------------------------+
> +
> + /// Device-Specific Components
> + ~~~ Sandboxing
> +
> +
> +Camera Manager
> + The Camera Manager enumerates cameras and instantiates Pipeline Handlers to
> + manage each Camera that libcamera supports. The Camera Manager supports
> + hotplug detection and notification events when supported by the underlying
> + kernel devices.
> +
> + There is only ever one instance of the Camera Manager running per application.
> + Each application's instance of the Camera Manager ensures that only a single
> + application can take control of a camera device at once.
> +
> + Read the `Camera Manager API`_ documentation for more details.
> +
> +.. _Camera Manager API: https://libcamera.org/api-html/classlibcamera_1_1CameraManager.html
>
> Camera Device
> - The Camera Device represents a camera device to upper layers. It
> - exposes full control of the device through the public API, and is
> - thus the highest level object exposed by libcamera.
> + The Camera class represents a single item of camera hardware that is capable
> + of producing one or more image streams, and provides the API to interact with
> + the underlying device.
> +
> + If a system has multiple instances of the same hardware attached, each has its
> + own instance of the camera class.
> +
> + The API exposes full control of the device to upper layers of libcamera through
> + the public API, making it the highest level object libcamera exposes, and the
> + object that all other API operations interact with from configuration to
> + capture.
> +
> + Read the `Camera API`_ documentation for more details.
>
> - Camera Device instances are created by the Camera Devices
> - Manager. An optional function to create new instances could be exposed
> - through the public API to speed up initialization when the upper
> - layer knows how to directly address camera devices present in the
> - system.
> +.. _Camera API: https://libcamera.org/api-html/classlibcamera_1_1Camera.html
>
> Pipeline Handler
> - The Pipeline Handler manages complex pipelines exposed by the kernel drivers
> - through the Media Controller and V4L2 APIs. It abstracts pipeline handling to
> - hide device-specific details to the rest of the library, and implements both
> - pipeline configuration based on stream configuration, and pipeline runtime
> - execution and scheduling when needed by the device.
> -
> - This component is device-specific and is part of the libcamera code base. As
> - such it is covered by the same free software license as the rest of libcamera
> - and needs to be contributed upstream by device vendors. The Pipeline Handler
> - lives in the same process as the rest of the library, and has access to all
> - helpers and kernel camera-related devices.
> + The Pipeline Handler manages the complex pipelines exposed by the kernel
> + drivers through the Media Controller and V4L2 APIs. It abstracts pipeline
> + handling to hide device-specific details from the rest of the library, and
> + implements both pipeline configuration based on stream configuration, and
> + pipeline runtime execution and scheduling when needed by the device.
> +
> + The Pipeline Handler lives in the same process as the rest of the library, and
> + has access to all helpers and kernel camera-related devices.
> +
> + Hardware abstraction is handled by device specific Pipeline Handlers which are
> + derived from the Pipeline Handler base class allowing commonality to be shared
> + among the implementations.
> +
> + Derived pipeline handlers create Camera device instances based on the devices
> + they detect and support on the running system, and are responsible for
> + managing the interactions with a camera device.
> +
> + More details can be found in the `PipelineHandler API`_ documentation, and the
> + :doc:`Pipeline Handler Writers Guide <guides/pipeline-handler>`.
> +
> +.. _PipelineHandler API: https://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html
>
> Image Processing Algorithms
> Together with the hardware image processing and hardware statistics
> - collection, the Image Processing Algorithms implement 3A (Auto-Exposure,
> + collection, the Image Processing Algorithms (IPA) implement 3A (Auto-Exposure,
> Auto-White Balance and Auto-Focus) and other algorithms. They run on the CPU
> - and interact with the kernel camera devices to control hardware image
> - processing based on the parameters supplied by upper layers, closing the
> - control loop of the ISP.
> -
> - This component is device-specific and is loaded as an external plugin. It can
> - be part of the libcamera code base, in which case it is covered by the same
> - license, or provided externally as an open-source or closed-source component.
> -
> - The component is sandboxed and can only interact with libcamera through
> - internal APIs specifically marked as such. In particular it will have no
> - direct access to kernel camera devices, and all its accesses to image and
> - metadata will be mediated by dmabuf instances explicitly passed to the
> - component. The component must be prepared to run in a process separate from
> - the main libcamera process, and to have a very restricted view of the system,
> - including no access to networking APIs and limited access to file systems.
> -
> - The sandboxing mechanism isn't defined by libcamera. One example
> - implementation will be provided as part of the project, and platforms vendors
> - will be able to provide their own sandboxing mechanism as a plugin.
> -
> - libcamera should provide a basic implementation of Image Processing
> - Algorithms, to serve as a reference for the internal API. Device vendors are
> - expected to provide a full-fledged implementation compatible with their
> - Pipeline Handler. One goal of the libcamera project is to create an
> - environment in which the community will be able to compete with the
> - closed-source vendor binaries and develop a high quality open source
> - implementation.
> + and control hardware image processing based on the parameters supplied by
> + upper layers, closing the control loop of the ISP.
> +
> + IPAs are loaded as external plugins named IPA Modules. IPA Modules can be part
> + of the libcamera code base or provided externally by camera vendors as
> + open-source or closed-source components.
> +
> + Open source IPA Modules built with libcamera are run in the same process space
> + as libcamera. External IPA Modules are run in a separate sandboxed process. In
> + either case, they can only interact with libcamera through the API provided by
> + the Pipeline Handler. They have a restricted view of the system, with no direct
> + access to kernel camera devices, no access to networking APIs, and limited
> + access to file systems. All their accesses to image and metadata are mediated
> + by dmabuf instances explicitly passed by the Pipeline Handler to the IPA
> + Module.
> +
> + IPA Modules are only required for platforms and devices with an ISP controlled
> + by the host CPU. Camera sensors which have an integrated ISP are not
> + controlled through the IPA Module.
>
> Helpers and Support Classes
> While Pipeline Handlers are device-specific, implementations are expected to
> @@ -136,3 +148,21 @@ Helpers and Support Classes
> self-contained support classes, even if such code is present only once in the
> code base, in order to keep the source code clean and easy to read. This
> should be the case for instance for plugin management.
> +
> +Platform Support
> +----------------
> +
> +The library currently supports the following hardware platforms specifically
> +with dedicated pipeline handlers:
> +
> + - Arm Mali-C55
> + - Intel IPU3 (ipu3)
> + - NXP i.MX8MP (imx8-isi and rkisp1)
> + - RaspberryPi 3, 4 and zero (rpi/vc4)
> + - Rockchip RK3399 (rkisp1)
> +
> +Furthermore, generic platform support is provided for the following:
> +
> + - USB video device class cameras (uvcvideo)
> + - iMX7, IPU6, Allwinner Sun6i (simple)
> + - Virtual media controller driver for test use cases (vimc)
--
Regards,
Laurent Pinchart
More information about the libcamera-devel
mailing list