[libcamera-devel] [PATCH v4 1/3] Documentation: Guides: Developers Guide to Libcamera
Kieran Bingham
kieran.bingham at ideasonboard.com
Thu Aug 20 15:47:49 CEST 2020
From: Chris Chinchilla <chris at gregariousmammal.com>
Create an introduction and overview for new developers to libcamera.
Provide an overview of the Camera Stack, and Architecture of libcamera
and introduce the main concepts of libcamera.
Signed-off-by: Chris Chinchilla <chris at gregariousmammal.com>
[Kieran: Rework/Reflow, add diagrams, licensing]
Signed-off-by: Kieran Bingham <kieran.bingham at ideasonboard.com>
---
Documentation/guides/introduction.rst | 319 ++++++++++++++++++++++++++
Documentation/index.rst | 2 +
Documentation/meson.build | 1 +
3 files changed, 322 insertions(+)
create mode 100644 Documentation/guides/introduction.rst
diff --git a/Documentation/guides/introduction.rst b/Documentation/guides/introduction.rst
new file mode 100644
index 000000000000..f34d2cf7cbdc
--- /dev/null
+++ b/Documentation/guides/introduction.rst
@@ -0,0 +1,319 @@
+.. SPDX-License-Identifier: CC-BY-SA-4.0
+
+Developers guide to libcamera
+=============================
+
+The Linux kernel handles multimedia devices through the 'Linux media' subsystem
+and provides a set of APIs (application programming interfaces) known
+collectively as V4L2 (`Video for Linux 2`_) and the `Media Controller`_ API
+which provide an interface to interact and control media devices.
+
+Included in this subsystem are drivers for camera sensors, CSI2 (Camera
+Serial Interface) recievers, and ISPs (Image Signal Processors)
+
+The usage of these drivers to provide a functioning camera stack is a
+responsibility that lies in userspace which is commonly implemented separately
+by vendors without a common architecture or API for application developers.
+
+libcamera provides a complete camera stack for Linux based systems to abstract
+functionality desired by camera application developers and process the
+configuration of hardware and image control algorithms requried to obtain
+desireable results from the camera.
+
+.. _Video for Linux 2: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/v4l2.html
+.. _Media Controller: https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/mediactl/media-controller.html
+
+
+In this developers guide, we will explore the `Camera Stack`_ and how it is
+can be visualised at a high level, and explore the internal `Architecture`_ of
+the libcamera library with its components. The current `Platform Support`_ is
+detailed, as well as an overview of the `Licensing`_ requirements of the
+project.
+
+This introduction is followed by a walkthrough tutorial to newcomers wishing to
+support a new platform with the `Pipeline Handler Writers Guide`_ and for those
+looking to make use of the libcamera native API an `Application Writers Guide`_
+provides a tutorial of the key APIs exposed by libcamera.
+
+.. _Pipeline Handler Writers Guide: pipeline-handler.html
+.. _Application Writers Guide: application-developer.html
+
+.. TODO: Correctly link to the other articles of the guide
+
+Camera Stack
+------------
+
+The libcamera library is implemented in userspace, and makes use of underlying
+kernel drivers that directly interact with hardware.
+
+Applications can make use of libcamera through the native `libcamera API`_'s or
+through an adaptation layer integrating libcamera into a larger framework.
+
+.. _libcamera API: https://www.libcamera.org/api-html/index.html
+
+::
+
+ Application Layer
+ / +--------------+ +--------------+ +--------------+ +--------------+
+ | | Native | | Framework | | Native | | Android |
+ | | V4L2 | | Application | | libcamera | | Camera |
+ | | Application | | (gstreamer) | | Application | | Framework |
+ \ +--------------+ +--------------+ +--------------+ +--------------+
+
+ ^ ^ ^ ^
+ | | | |
+ | | | |
+ v v | v
+ Adaptation Layer |
+ / +--------------+ +--------------+ | +--------------+
+ | | V4L2 | | gstreamer | | | Android |
+ | | Compatability| | element | | | Camera |
+ | | (preload) | |(libcamerasrc)| | | HAL |
+ \ +--------------+ +--------------+ | +--------------+
+ |
+ ^ ^ | ^
+ | | | |
+ | | | |
+ v v v v
+ libcamera Framework
+ / +--------------------------------------------------------------------+
+ | | |
+ | | libcamera |
+ | | |
+ \ +--------------------------------------------------------------------+
+
+ ^ ^ ^
+ Userspace | | |
+ --------------------- | ---------------- | ---------------- | ---------------
+ Kernel | | |
+ v v v
+
+ +-----------+ +-----------+ +-----------+
+ | Media | <--> | Video | <--> | V4L2 |
+ | Device | | Device | | Subdev |
+ +-----------+ +-----------+ +-----------+
+
+The camera stack comprises of four software layers. From bottom to top:
+
+* The kernel drivers control the camera hardware and expose a low-level
+ interface to userspace through the Linux kernel V4L2 family of APIs
+ (Media Controller API, V4L2 Video Device API and V4L2 Subdev API).
+
+* The libcamera framework is the core part of the stack. It handles all control
+ of the camera devices in its core component, libcamera, and exposes a native
+ C++ API to upper layers.
+
+* The libcamera adaptation layer is an umbrella term designating the components
+ that interface to libcamera in other frameworks. Notable examples are the V4L2
+ compatibility layer, the gstreamer libcamera element, and the Android camera
+ HAL implementation based on libcamera which are provided as a part of the
+ libcamera project.
+
+* The applications and upper level frameworks are based on the libcamera
+ framework or libcamera adaptation, and are outside of the scope of the
+ libcamera project, however example native applications (cam, qcam) are
+ provided for testing.
+
+
+V4L2 Compatibility Layer
+ V4L2 compatibility is achieved through a shared library that traps all
+ accesses to camera devices and routes them to libcamera to emulate high-level
+ V4L2 camera devices. It is injected in a process address space through
+ `LD_PRELOAD` and is completely transparent for applications.
+
+ The compatibility layer exposes camera device features on a best-effort basis,
+ and aims for the level of features traditionally available from a UVC camera
+ designed for video conferencing.
+
+Android Camera HAL
+ Camera support for Android is achieved through a generic Android camera HAL
+ implementation on top of libcamera. The HAL implements features required by
+ Android and out of scope from libcamera, such as JPEG encoding support.
+
+ This component is used to provide support for ChromeOS platforms
+
+GStreamer element (gstlibcamerasrc)
+ A `GStreamer element`_ is provided to allow capture from libcamera supported
+ devices through GStreamer pipelines, and connect to other elements for further
+ processing.
+
+ Development of this element is ongoing and is limited to a single stream.
+
+Native libcamera API
+ Applications can make use of the libcamera API directly using the C++
+ API. An example application and walkthrough using the libcamera API can be
+ followed in the `Application Writers Guide`_
+
+.. _GStreamer element: https://gstreamer.freedesktop.org/documentation/application-development/basics/elements.html
+
+Architecture
+------------
+
+While offering a unified API towards upper layers, and presenting itself as a
+single library, libcamera isn’t monolithic. It exposes multiple components
+through its public API and is built around a set of separate helpers internally.
+Hardware abstractractions are handled through the use of device-specific
+components where required and dynamically loadable plugins are used to separate
+image processing algorithms from the core libcamera codebase.
+
+::
+
+ --------------------------< libcamera Public API >---------------------------
+ ^ ^
+ | |
+ v v
+ +-------------+ +---------------------------------------------------+
+ | Camera | | Camera Device |
+ | Manager | | +-----------------------------------------------+ |
+ +-------------+ | | Device-Agnostic | |
+ ^ | | | |
+ | | | +--------------------------+ |
+ | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
+ | | | | { +-----------------+ } |
+ | | | | } | //// Image //// | { |
+ | | | | <-> | / Processing // | } |
+ | | | | } | / Algorithms // | { |
+ | | | | { +-----------------+ } |
+ | | | | ~~~~~~~~~~~~~~~~~~~~~~~ |
+ | | | | ========================== |
+ | | | | +-----------------+ |
+ | | | | | // Pipeline /// | |
+ | | | | <-> | /// Handler /// | |
+ | | | | | /////////////// | |
+ | | +--------------------+ +-----------------+ |
+ | | Device-Specific |
+ | +---------------------------------------------------+
+ | ^ ^
+ | | |
+ v v v
+ +--------------------------------------------------------------------+
+ | Helpers and Support Classes |
+ | +-------------+ +-------------+ +-------------+ +-------------+ |
+ | | MC & V4L2 | | Buffers | | Sandboxing | | Plugins | |
+ | | Support | | Allocator | | IPC | | Manager | |
+ | +-------------+ +-------------+ +-------------+ +-------------+ |
+ | +-------------+ +-------------+ |
+ | | Pipeline | | ... | |
+ | | Runner | | | |
+ | +-------------+ +-------------+ |
+ +--------------------------------------------------------------------+
+
+ /// Device-Specific Components
+ ~~~ Sandboxing
+
+
+Camera Manager
+ The Camera Manager enumerates cameras and instantiates Pipeline Handlers to
+ manage each Camera that libcamera supports. The Camera Manager supports
+ hotplug detection and notification events when supported by the underlying
+ kernel devices.
+
+ There is only ever one instance of the Camera Manager running per application.
+ Each application's instance of the Camera Manager ensures that only a single
+ application can take control of a camera device at once.
+
+ Read the `Camera Manager API`_ documentation for more details.
+
+.. _Camera Manager API: http://libcamera.org/api-html/classlibcamera_1_1CameraManager.html
+
+Camera Device
+ The Camera class represents a single item of camera hardware that is capable
+ of producing one or more image streams, and provides the API to interact with
+ the underlying device.
+
+ If a system has multiple instances of the same hardware attached, each has it's
+ own instance of the camera class.
+
+ The API exposes full control of the device to upper layers of libcamera through
+ the public API, making it the highest level object libcamera exposes, and the
+ object that all other API operations interact with from configuration to
+ capture.
+
+ Read the `Camera API`_ documentation for more details.
+
+.. _Camera API: http://libcamera.org/api-html/classlibcamera_1_1Camera.html
+
+Pipeline Handler
+ The Pipeline Handler manages the complex pipelines exposed by the kernel
+ drivers through the Media Controller and V4L2 APIs. It abstracts pipeline
+ handling to hide device-specific details from the rest of the library, and
+ implements both pipeline configuration based on stream configuration, and
+ pipeline runtime execution and scheduling when needed by the device.
+
+ The Pipeline Handler lives in the same process as the rest of the library, and
+ has access to all helpers and kernel camera-related devices.
+
+ Hardware abstraction is handled by device specific Pipeline Handlers which are
+ derived from the Pipeline Handler base class allowing commonality to be shared
+ among the implementations.
+
+ Derived pipeline handlers create Camera device instances based on the devices
+ they detect and support on the running system, and are responsible for
+ managing the interactions with a camera device.
+
+ More details can be found in the `PipelineHandler API`_ documentation, and the
+ `Pipeline Handler Writers Guide`_.
+
+.. _PipelineHandler API: http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html
+
+Image Processing Algorithms
+ An image processing algorithm (IPA) component is a loadable plugin that
+ implements 3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) and other
+ algorithms.
+
+ The algorithms run on the CPU and interact with the camera devices through the
+ Pipeline Handler to control hardware image processing based on the parameters
+ supplied by upper layers, maintaining state and closing the control loop
+ of the ISP.
+
+ The component is sandboxed and can only interact with libcamera through the
+ API provided by the Pipeline Handler and an IPA has no direct access to kernel
+ camera devices.
+
+ Open source IPA modules built with libcamera can be run in the same process
+ space as libcamera, however external IPA modules are run in a separate process
+ from the main libcamera process. IPA modules have a restricted view of the
+ system, including no access to networking APIs and limited access to file
+ systems.
+
+ IPA modules are only required for platforms and devices with an ISP controlled
+ by the host CPU. Camera sensors which have an integrated ISP are not
+ controlled through the IPA module.
+
+Platform Support
+----------------
+
+The library currently supports the following hardware platforms specifically
+with dedicated pipeline handlers:
+
+ - Intel IPU3 (ipu3)
+ - Rockchip RK3399 (rkisp1)
+ - RaspberryPi 3 and 4 (raspberrypi)
+
+Furthermore, generic platform support is provided for the following:
+
+ - USB video device class cameras (uvcvideo)
+ - iMX7, Allwinner Sun6i (simple)
+ - Virtual media controller driver for test use cases (vimc)
+
+Licensing
+---------
+
+The libcamera core, is covered by the `LGPL-2.1-or-later`_ license. Pipeline
+Handlers are a part of the libcamera code base and need to be contributed
+upstream by device vendors. IPA modules included in libcamera are covered by a
+free software license, however third-parties may develop IPA modules outside of
+libcamera and distribute them under a closed-source license, provided they do
+not include source code from the libcamera project.
+
+The libcamera project itself contains multiple libraries, applications and
+utilities. Licenses are expressed through SPDX tags in text-based files that
+support comments, and through the .reuse/dep5 file otherwise. A copy of all
+licenses are stored in the LICENSES directory, and a full summary of the
+licensing used throughout the project can be found in the COPYING.rst document.
+
+Applications which link dynamically against libcamera and use only the public
+API are an independent work of the authors and have no license restrictions
+imposed upon them from libcamera.
+
+.. _LGPL-2.1-or-later: https://spdx.org/licenses/LGPL-2.1-or-later.html
\ No newline at end of file
diff --git a/Documentation/index.rst b/Documentation/index.rst
index 4e746bb17c4a..cfcfd388699b 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -12,3 +12,5 @@
Home <self>
Docs <docs>
Contribute <contributing>
+
+ Developer Guide <guides/introduction>
diff --git a/Documentation/meson.build b/Documentation/meson.build
index 6d9a397cf1a3..dd7ae700af11 100644
--- a/Documentation/meson.build
+++ b/Documentation/meson.build
@@ -52,6 +52,7 @@ if sphinx.found()
'contributing.rst',
'docs.rst',
'index.rst',
+ 'guides/introduction.rst',
]
release = 'release=v' + libcamera_git_version
--
2.25.1
More information about the libcamera-devel
mailing list