[libcamera-devel] [PATCH] Create libcamera overview and gloassry

Fri Jun 26 13:17:32 CEST 2020

From: Chris Chinchilla <chris at gregariousmammal.com>

Creates a libcamera overview guide and glossary to give new developers an idea of why they should use libcamera, its architecture, and features it offers.

Signed-off-by: Chris Chinchilla <chris at gregariousmammal.com>
---
 Documentation/introduction.rst | 195 +++++++++++++++++++++++++++++++++
 1 file changed, 195 insertions(+)
 create mode 100644 Documentation/introduction.rst

diff --git a/Documentation/introduction.rst b/Documentation/introduction.rst
new file mode 100644
index 0000000..0539e60
--- /dev/null
+++ b/Documentation/introduction.rst
@@ -0,0 +1,195 @@
+An introduction to libcamera
+============================
+
+The Video for Linux 2 (`V4L2 <https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/v4l2.html>`_) API provides kernel drivers for devices that provide and manipulate
+images and video. However, Linux was missing a convenient way for application developers to take 
+advantage of these kernel drivers in userspace. Vendors of devices that provide image input sources 
+referred to as “image signal processors” (ISPs) sometimes contribute open-source V4L2 drivers. 
+However, ISPs vary so much, it’s a hard task for developers to write portable ISP-based
+applications.
+
+The libcamera library aims to fill this gap by providing a complete userspace camera 
+stack for Linux-based systems that supports a wide variety of ISPs, including systems with multiple ISPs attached.
+
+The library currently supports:
+
+-  Hardware
+
+   -  Intel IPU3
+   -  Rockchip RK3399
+   -  RaspberryPi 3 and 4
+   -  USB video device class (UVC) cameras
+   -  Virtual media controller (vimc) driver
+
+-  Software
+
+   -  Other Linux-kernel-based operating systems (such as Android and ChromeOS)
+   -  Image enhancement algorithms
+   -  A V4L2 compatibility layer
+
+The library provides a public API for managing ISPs, frame capture, video streams, frame and 
+request metadata, events and callbacks, image processing, and more.
+
+Where libcamera sits in the camera stack
+----------------------------------------
+
+The libcamera library sits in userspace, just on top of the kernel drivers that directly interact 
+with hardware and the V4L2 family of APIs (Media Controller, V4L2 Video Device, and V4L2 sub-device APIs).
+
+When using libcamera in a camera stack, it is the core component, taking control of all camera 
+devices, and exposing a native C++ API to upper layers of the framework. Other language bindings are coming soon.
+
+Compatibility Layers
+~~~~~~~~~~~~~~~~~~~~
+
+In a layer above the core framework are compatibility libraries to help existing applications and their developers.
+
+V4L2 Compatibility Layer
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+To emulate high-level V4L2 camera devices and capture all access to camera devices, libcamera uses 
+a shared library. The shared library is transparent to libcamera-based applications and injected 
+into a process address space using dynamic linking with ``LD_PRELOAD``.
+
+The compatibility layer exposes camera device features on a best-effort basis and aims for the 
+level of features traditionally available from a UVC camera designed for video conferencing.
+
+Android Camera HAL
+^^^^^^^^^^^^^^^^^^
+
+The libcamera library offers Android camera support through a generic `hardware abstraction layer (HAL) <https://source.android.com/devices/camera/camera3_requests_hal>`_ implementation. 
+The HAL focuses on supporting features required by Android that are missing from libcamera, such as JPEG encoding.
+
+The Android camera HAL implementation initially targets the ``LIMITED``
+`hardware level <https://source.android.com/devices/camera/versioning#camera_api2>`_,
+with support for the ``FULL`` level then gradually implemented.
+
+gstreamer element
+^^^^^^^^^^^^^^^^^
+
+The library provides `a gstreamer element <https://gstreamer.freedesktop.org/documentation/application-development/basics/elements.html?gi-language=c>`_ for routing libcamera data to gstreamer for processing.
+
+libcamera Architecture
+----------------------
+
+The libcamera library exposes one unified API, but behind that is built from a 
+set of specific components for each supported device, components for different 
+functionalities, and helpers for those components.
+
+Camera Devices Manager
+~~~~~~~~~~~~~~~~~~~~~~
+
+Provides access and manages all cameras on a system. It enumerates cameras at 
+build and runtime and supports a hotplug notification mechanism.
+
+There is only ever one instance of the Camera Devices Manager running per application.
+
+`Read the Camera Manager API documentation for more details <http://libcamera.org/api-html/classlibcamera_1_1CameraManager.html>`_
+
+Camera Device
+~~~~~~~~~~~~~
+
+A camera device is a device capable of producing one or more image streams from 
+a single image source and can include multiple identical camera devices.
+
+The libcamera library represents a camera device to the upper layers of the library. 
+It exposes full control of the device through the public API, making it the highest 
+level object libcamera exposes, and the object that all other API operations interact 
+with from configuration to capture.
+
+`Read the Camera API documentation for more details <http://libcamera.org/api-html/classlibcamera_1_1Camera.html>`_
+
+Frame controls
+^^^^^^^^^^^^^^
+
+Depending on the capabilities of a camera device and the hardware it is connected 
+to, libcamera supports controlling capture parameters for each stream on a per-frame basis.
+
+These controls include auto exposure, gain, brightness, contrast, lux, white balance, color temperature, and saturation.
+
+`Read the Control API documentation for more details <http://libcamera.org/api-html/classlibcamera_1_1Control.html>`_
+
+Pipeline Handler
+~~~~~~~~~~~~~~~~
+
+A pipeline handler is a private API that manages the pipelines exposed by the kernel 
+drivers through the Media Controller and V4L2 APIs.
+
+Each supported camera device has its own pipeline handler component, which hides 
+and abstracts device-specific details, providing what the rest of the library 
+needs to interact with a device.
+
+A pipeline handler manages most aspects of interacting with a camera device including:
+
+-  acquiring use of the camera device
+-  frame controls
+-  pipelines and stream configuration
+-  the data the camera produces, and the buffers it needs to hold the data
+-  creating requests applications need to access camera data
+
+A pipeline handler is part of the libcamera codebase, and hardware vendors generally need to create them.
+
+`Read the PipelineHandler API documentation for more details <http://libcamera.org/api-html/classlibcamera_1_1PipelineHandler.html>`_
+
+Image Processing Algorithms
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An image processing algorithm (IPA) component is a private API that implements 
+3A (Auto-Exposure, Auto-White Balance, and Auto-Focus) algorithms.
+
+Each supported camera device has its own IPA component that runs on the CPU and 
+interacts with the kernel camera devices to control hardware image processing 
+based on the parameters supplied by upper layers, and helps maintain state, 
+closing the control loop of the ISP.
+
+An IPA component can be part of the libcamera code base, in which case the same 
+license covers it, or provided externally as an open-source or closed-source component.
+
+.. TODO: Better in the IPA guide?
+
+The component is sandboxed and can only interact with libcamera through specifically 
+marked APIs. A component has no direct access to kernel camera devices, and dmabuf 
+instances explicitly passed to the component control its access to image data and 
+metadata. The component should run in a process separate from the main libcamera 
+process, and has a restricted view of the system, including no access to networking APIs
+and limited access to file systems.
+
+The sandboxing mechanism isn’t defined by libcamera. Platform vendors need to provide sandboxing 
+mechanisms as a plugin.
+
+To support this security and sandboxing model, libcamera provides an IPC mechanism 
+for an IPA and Pipeline Handler to communicate, but developers need to create the 
+API themselves. Platform vendors can also use any other IPC mechanism that supports 
+passing file descriptors.
+
+3A and Image Enhancement Algorithms
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Camera devices should implement auto exposure, auto gain, and auto white balance, 
+and those that include a focus lens should also implement autofocus.
+
+Vendors can implement these algorithms in hardware or firmware outside of libcamera, 
+or in an IPA component.
+
+
+.. TODO: Add link to guide when completed
+
+Helpers and Support Classes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. TODO: Feel like this is implied in other places of the doc
+
+To help developers create device-specific pipeline handlers and image processing 
+algorithms, libcamera provides helpers and support classes that sit on top of the 
+Media Controller and V4L2 APIs.
+
+Glossary
+--------
+
+-  **ISP**: Image Signal Processor.
+-  **SoC**: System on a Chip.
+-  **Media controller API**: `The Linux Kernel API <https://www.kernel.org/doc/html/v4.10/media/uapi/mediactl/media-controller-intro.html>`_ that handles audio and video input and output.
+-  **V4L2**: `Video for Linux API version 2 <https://www.linuxtv.org/downloads/v4l-dvb-apis-new/userspace-api/v4l/v4l2.html>`_. Device drivers and API for video capture on Linux.
+-  **UVC camera**: `USB Video Class <https://en.wikipedia.org/wiki/USB_video_device_class>`_ that describes devices capable of streaming video.
+-  **3A**: Common algorithms used by camera devices for auto-exposure, auto-white balance and auto-focus).
+-  **IPC**: `Inter-process communications based protocol <https://en.wikipedia.org/wiki/Inter-process_communication>`_.
