[libcamera-devel] [PATCH v4 01/12] documentation: Introduce Camera Sensor Model
Jacopo Mondi
jacopo.mondi at ideasonboard.com
Thu Sep 21 15:48:22 CEST 2023
Hi Laurent
On Thu, Sep 21, 2023 at 03:04:40PM +0300, Laurent Pinchart wrote:
> Hi Jacopo,
>
> Thank you for the patch.
>
> On Sat, Sep 16, 2023 at 02:19:19PM +0200, Jacopo Mondi via libcamera-devel wrote:
> > Introduce a documentation page about the 'camera sensor model'
> > implemented by libcamera.
> >
> > The camera sensor model serves to provide to applications a reference
> > description of the processing steps that take place in a camera sensor
> > in order to precisely control the sensor configuration through the
> > forthcoming SensorConfiguration class.
> >
> > Signed-off-by: Jacopo Mondi <jacopo.mondi at ideasonboard.com>
> > Reviewed-by: Naushir Patuck <naush at raspberrypi.com>
> > Reviewed-by: Kieran Bingham <kieran.bingham at ideasonboard.com>
> > ---
> > Documentation/binning.svg | 5053 +++++++++++++++++++++++++
> > Documentation/camera-sensor-model.rst | 170 +
> > Documentation/index.rst | 1 +
> > Documentation/meson.build | 1 +
> > Documentation/sensor_model.svg | 4866 ++++++++++++++++++++++++
> > Documentation/skipping.svg | 1720 +++++++++
> > 6 files changed, 11811 insertions(+)
> > create mode 100644 Documentation/binning.svg
> > create mode 100644 Documentation/camera-sensor-model.rst
> > create mode 100644 Documentation/sensor_model.svg
> > create mode 100644 Documentation/skipping.svg
> >
> > diff --git a/Documentation/binning.svg b/Documentation/binning.svg
> > new file mode 100644
> > index 000000000000..c6a3b6394acd
> > --- /dev/null
> > +++ b/Documentation/binning.svg
> > @@ -0,0 +1,5053 @@
> > +<?xml version="1.0" encoding="UTF-8" standalone="no"?>
>
> <!-- SPDX-License-Identifier: CC-BY-SA-4.0 -->
>
> Same for the other SVG files below.
>
> This will likely need manual merge when updating the diagrams :-S
> Another option is to specify the license in .reuse/dep5.
Good point, I'll add to .reuse/dep5
Files: Documentation/binning.svg
Documentation/camera-sensor-model.rst
Documentation/sensor_model.svg
Copyright: Copyright 2023 Ideas On Board Oy
License: CC-BY-SA-4.0
>
> > +<!-- Created with Inkscape (http://www.inkscape.org/) -->
>
> I won't review the rest of the SVG here :-) However, neither Firefox nor
> Chrome render the arrow heads :-S I wonder if inkscape does something
> wrong. This isn't a blocker, we can fix it on top.
>
Yes, I notiched with Chromium and did blame the broswer as the arrows
render correctly in Inkscape... But it firefox fails too I wonder if
something is not wrong in Inkscape
> [snip]
>
> > diff --git a/Documentation/camera-sensor-model.rst b/Documentation/camera-sensor-model.rst
> > new file mode 100644
> > index 000000000000..7ff0bdc50564
> > --- /dev/null
> > +++ b/Documentation/camera-sensor-model.rst
> > @@ -0,0 +1,170 @@
> > +.. SPDX-License-Identifier: CC-BY-SA-4.0
> > +
> > +.. _camera-sensor-model:
>
> Could you add a comment here to indicate this should be moved to the
> doxygen-generated documentation ?
>
> .. todo: Move to Doxygen-generated documentation
>
Ack
> > +
> > +The libcamera camera sensor model
> > +=================================
> > +
> > +libcamera defines an abstract camera sensor model in order to provide
> > +a description of each of the processing steps that result in image data being
> > +sent on the media bus and that form the image stream delivered to applications.
> > +
> > +Applications should use the abstracted camera sensor model defined here to
>
> s/abstracted/abstract/
>
> > +precisely control the operations of the camera sensor.
> > +
> > +The libcamera camera sensor model targets image sensors producing frames in
> > +RAW format, delivered through a MIPI CSI-2 compliant bus implementation.
> > +
> > +The abstract sensor model maps libcamera components to the characteristics and
> > +operations of an image sensor, and serves as a reference to model the libcamera
> > +CameraSensor class and SensorConfiguration classes and operations.
> > +
> > +In order to control the configuration of the camera sensor through the
> > +SensorConfiguration class, applications should understand this model and map it
> > +to the combination of image sensor and kernel driver in use.
> > +
> > +The camera sensor model defined here is based on the *MIPI CCS specification*,
> > +particularly on *Section 8.2 - Image readout* of *Chapter 8 - Video Timings*.
> > +
> > +Glossary
> > +---------
> > +
> > +- *Pixel array*: The full grid of pixels, active and inactive ones
> > +
> > +- *Pixel array active area*: The portion(s) of the pixel array that
> > + contains valid and readable pixels; corresponds to the libcamera
> > + properties::PixelArrayActiveAreas
> > +
> > +- *Analog crop rectangle*: The portion of the *pixel array active area* which
> > + is read out and passed to further processing stages
> > +
> > +- *Subsampling*: Pixel processing techniques that reduce the image size by
> > + binning or by skipping adjacent pixels
> > +
> > +- *Digital crop*: Crop of the sub-sampled image data before scaling
> > +
> > +- *Frame output*: The frame (image) as output on the media bus by the
> > + camera sensor
>
> Sphinx has a glossary directive:
>
> .. glossary::
>
> Pixel array
> The full grid of pixels, active and inactive ones
>
> Pixel array active area
> The portion(s) of the pixel array that contains valid and readable
> pixels; corresponds to the libcamera properties::PixelArrayActiveAreas
>
> Analog crop rectangle
> The portion of the *pixel array active area* which is read out and passed
> to further processing stages
>
> Subsampling
> Pixel processing techniques that reduce the image size by binning or by
> skipping adjacent pixels
>
> Digital crop
> Crop of the sub-sampled image data before scaling
>
> Frame output
> The frame (image) as output on the media bus by the camera sensor
>
> > +
> > +Camera sensor model
> > +-------------------
> > +
> > +The abstract sensor model is described in the following diagram
>
> s/$/./
>
> > +
> > +.. figure:: sensor_model.svg
> > +
> > +
> > +1. The sensor reads pixels from the *pixel array*. The pixels being read out are
> > + selected by the *analog crop rectangle*.
> > +
> > +2. The pixels can be subsampled to reduce the image size without affecting the
> > + field of view. Two subsampling techniques can be used:
> > +
> > + - Binning: combines adjacent pixels of the same colour by averaging or
> > + summing their values, in the analog domain and/or the digital domain.
> > +
> > + .. figure:: binning.svg
> > +
> > +
> > + - Skipping: skips the read out of a number of adjacent pixels.
> > +
> > + .. figure:: skipping.svg
> > +
> > +
> > +3. The output of the optional sub-sampling stage is then cropped after the
> > +conversion of the analogue pixel values in the digital domain.
>
> s/^/ /
>
> > +
> > +4. The resulting output frame is sent on the media bus by the sensor.
> > +
> > +Camera Sensor configuration parameters
> > +--------------------------------------
> > +
> > +The libcamera camera sensor model defines parameters that allow users to
> > +control:
> > +
> > +1. The image format bit depth
> > +
> > +2. The size and position of the *Analog crop rectangle*
> > +
> > +3. The subsampling factors used to downscale the pixel array readout data to a
> > + smaller frame size without reducing the image *field of view*. Two
> > + configuration parameters are made available to control the downscaling factor
>
> s/^/:/
>
> > +
> > + - binning
> > + - a vertical and horizontal binning factor can be specified, the image
>
> s/- a/A/
>
> > + will be downscaled in its vertical and horizontal sizes by the specified
> > + factor
>
> s/$/./
>
> > +
> > + .. code-block::
> > + :caption: Definition: The horizontal and vertical binning factors
> > +
> > + horizontal_binning = xBin;
> > + vertical_binning = yBin;
> > +
> > +
> > + - skipping
> > + - reduces the image resolution by skipping the read-out of a
> > + number of adjacent pixels
> > + - the skipping factor is specified by the 'increment' number (number of
> > + pixels to 'skip') in the vertical and horizontal directions and for
> > + even and odd rows and columns
>
> Skipping reduces the image resolution by skipping the read-out of a number
> of adjacent pixels. The skipping factor is specified by the 'increment'
> number (number of pixels to 'skip') in the vertical and horizontal
> directions and for even and odd rows and columns.
>
> > +
> > + .. code-block::
> > + :caption: Definition: The horizontal and vertical skipping factors
> > +
> > + horizontal_skipping = (xOddInc + xEvenInc) / 2
> > + vertical_skipping = (yOddInc + yEvenInc) / 2
> > +
> > +
> > + - different sensors perform the binning and skipping stages in different
> > + orders
> > + - for the sake of computing the final output image size the order of
> > + execution is not relevant.
> > +
> > + - the overall down-scaling factor is obtained by combining the binning and
> > + skipping factors
>
> As the text above introduces a list of two scaling factors, let's turn
> the rest into text, not bullet points:
>
> Different sensors perform the binning and skipping stages in different
> orders. For the sake of computing the final output image size the order of
> execution is not relevant. The overall down-scaling factor is obtained by
> combining the binning and skipping factors.
>
Ack
> > +
> > + .. code-block::
> > + :caption: Definition: The total scaling factor (binning + sub-sampling)
> > +
> > + total_horizontal_downscale = horizontal_binning + horizontal_skipping
> > + total_vertical_downscale = vertical_binning + vertical_skipping
> > +
> > +
> > +4. The output data frame size
> > + - the output size is used to specify any additional cropping on the
> > + sub-sampled frame
>
> Ditto, drop the bullet, start with a capital letter, end with a period.
> Same below.
>
> > +
> > +5. The total line length and frame height (*visibile* pixels + *blankings*) as
> > + sent on the MIPI CSI-2 bus
> > +
> > +6. The pixel transmission rate on the MIPI CSI-2 bus
> > +
> > +The above parameters are combined to obtain the following high-level
> > +configurations:
> > +
> > +- **frame output size**
> > +
> > + Obtained by applying a crop to the physical pixel array size in the analog
> > + domain, followed by optional binning and sub-sampling (in any order),
> > + followed by an optional crop step in the output digital domain.
> > +
> > +- **frame rate**
> > +
> > + The combination of the *total frame size*, the image format *bit depth* and
> > + the *pixel rate* of the data sent on the MIPI CSI-2 bus allows to compute the
> > + image stream frame rate. The equation is the well known:
> > +
> > + .. code-block::
> > +
> > + frame_duration = total_frame_size / pixel_rate
> > + frame_rate = 1 / frame_duration
> > +
> > +
> > + where the *pixel_rate* parameter is the result of the sensor's configuration
> > + of the MIPI CSI-2 bus *(the following formula applies to MIPI CSI-2 when
> > + used on MIPI D-PHY physical protocol layer only)*
> > +
> > + .. code-block::
> > +
> > + pixel_rate = CSI-2_link_freq * 2 * nr_of_lanes / bits_per_sample
> > diff --git a/Documentation/index.rst b/Documentation/index.rst
> > index 43d8b017d3b4..63fac72d11ed 100644
> > --- a/Documentation/index.rst
> > +++ b/Documentation/index.rst
> > @@ -23,3 +23,4 @@
> > Sensor driver requirements <sensor_driver_requirements>
> > Lens driver requirements <lens_driver_requirements>
> > Python Bindings <python-bindings>
> > + Camera Sensor Model <camera-sensor-model>
> > diff --git a/Documentation/meson.build b/Documentation/meson.build
> > index b2a5bf15e6ea..7c1502592baa 100644
> > --- a/Documentation/meson.build
> > +++ b/Documentation/meson.build
> > @@ -63,6 +63,7 @@ endif
> >
> > if sphinx.found()
> > docs_sources = [
> > + 'camera-sensor-model.rst',
> > 'coding-style.rst',
> > 'conf.py',
> > 'contributing.rst',
> > diff --git a/Documentation/sensor_model.svg b/Documentation/sensor_model.svg
> > new file mode 100644
> > index 000000000000..1f76d41cf6a5
> > --- /dev/null
> > +++ b/Documentation/sensor_model.svg
> > @@ -0,0 +1,4866 @@
>
> It looks quite neat. The only improvement I can see could be nice is to
> scale the image in 4 to have the same pixel size as in 3, to avoid
> implying there's any scaling. It can be done on top, or you can the
> change before pushing.
>
Ah good point, I think it's a leftover from when I had the digital
scaling block here
> [snip]
>
> > diff --git a/Documentation/skipping.svg b/Documentation/skipping.svg
> > new file mode 100644
> > index 000000000000..7bef37cfcc64
> > --- /dev/null
> > +++ b/Documentation/skipping.svg
> > @@ -0,0 +1,1720 @@
>
> [snip]
>
> Reviewed-by: Laurent Pinchart <laurent.pinchart at ideasonboard.com>
Thanks
>
> --
> Regards,
>
> Laurent Pinchart
More information about the libcamera-devel
mailing list