[PATCH v1 1/1] Documentation: Add first version of the tuning-guide

[Stefan Klug]

Hi Kieran,

Thank you for the review. I marked most comments as done. They will be
included in the next version. There was only one comment with a response
from my side.

On Tue, Aug 06, 2024 at 05:55:06PM +0100, Kieran Bingham wrote:
> Quoting Stefan Klug (2024-08-06 15:44:41)
> > This patch adds a initial version of the tuning guide for libcamera. The
> > process is established based on the imx8mp and will be improved when we
> > do more tuning for different ISPs with our own tooling.
> > 
> > Signed-off-by: Stefan Klug <stefan.klug at ideasonboard.com>
> > ---
> >  Documentation/conf.py                  |   4 +-
> >  Documentation/guides/tuning/tuning.rst | 276 +++++++++++++++++++++++++
> >  Documentation/index.rst                |   1 +
> >  Documentation/meson.build              |   1 +
> >  4 files changed, 280 insertions(+), 2 deletions(-)
> >  create mode 100644 Documentation/guides/tuning/tuning.rst
> > 
> > diff --git a/Documentation/conf.py b/Documentation/conf.py
> > index 7eeea7f3865b..5387942b9af5 100644
> > --- a/Documentation/conf.py
> > +++ b/Documentation/conf.py
> > @@ -21,8 +21,8 @@
> >  # -- Project information -----------------------------------------------------
> >  
> >  project = 'libcamera'
> > -copyright = '2018-2019, The libcamera documentation authors'
> > -author = u'Kieran Bingham, Jacopo Mondi, Laurent Pinchart, Niklas Söderlund'
> > +copyright = '2018-2024, The libcamera documentation authors'
> > +author = u'Kieran Bingham, Jacopo Mondi, Laurent Pinchart, Niklas Söderlund, Stefan Klug'
> >  
> >  # Version information is provided by the build environment, through the
> >  # sphinx command line.
> > diff --git a/Documentation/guides/tuning/tuning.rst b/Documentation/guides/tuning/tuning.rst
> > new file mode 100644
> > index 000000000000..1c30f3f20b4e
> > --- /dev/null
> > +++ b/Documentation/guides/tuning/tuning.rst
> > @@ -0,0 +1,276 @@
> > +.. SPDX-License-Identifier: CC-BY-SA-4.0
> > +
> > +Sensor Tuning Guide
> > +===================
> > +
> > +To create visually good images out of the raw data provided by the camera
> 
> s/out of the/from the/

done

> 
> > +sensor, a lot of image processing takes place. This is usually done inside an
> > +ISP (Image Signal Processor). To be able to do the necessary processing, the
> > +corresponding algorithms need to be parameterized according to the used hardware
> 
> s/to the used hardware/to the hardware in use/

done

> 
> > +(typically sensor, light and lens). Creating these parameters is called tuning.
> 
> s/Creating/Calculating/ ?

done

> 
> s/is called/is a process called/

done

> 
> > +The tuning process results in a tuning file which is then used by libcamera to
> > +parameterize the algorithms at runtime.
> 
> s/parameterize the/provide calibrated parameters to the/

done

> 
> > +
> > +The processing block of an ISP vary from vendor to vendor and can be
> 
> s/block/blocks

done

> 
> > +arbitrarily complex. Never the less a diagram of the most common and important
> 
> s/Never the less/Nevertheless/ ?

Oh yes it get's written as one word...

> 
> - based on https://dictionary.cambridge.org/dictionary/english/nevertheless

I get "500 Internal server error" :-)
> 
> Is it correct to say these are the most common and important ? That
> sounds too subjective and dependent upon use cases...
> 
> Perhaps:
> 
> "a diagram of common blocks frequently found in an ISP design is shown
> below:"

Definitely sounds better. Taken.

> 
> But I don't mind your version.
> 
> 
> > +blocks is shown below:
> > +
> > + ::
> > +
> > +  +--------+ 
> > +  |  Light |
> > +  +--------+
> > +      |
> > +      v
> > +  +--------+     +-----+    +-----+    +-----+    +-----+    +-------+
> > +  | Sensor |  -> | BLC | -> | AWB | -> | LSC | -> | CCM | -> | Gamma |
> > +  +--------+     +-----+    +-----+    +-----+    +-----+    +-------+
> > +
> > +**Light** The light used to light the scene has a crucial influence on the
> > +resulting image. The human eye and brain are specialized in automatically
> > +adapting to different lights. In a camera this has to be done by an algorithm.
> > +Light is a complex topic and to correctly describe light sources you need to
> > +describe the spectrum of the light. To simplify things, lights are categorized
> > +according to their `color temperature (K)
> > +<https://en.wikipedia.org/wiki/Color_temperature>`_. This is important to
> > +keep in mind as it means that calibrations may differ between light sources even
> > +though they have the same nominal color temperature. For best results the tuning
> > +images need to be taken for a complete range of light sources.
> 
> I would put that last statement on it's own line as it's a
> distinct/important point ("For best results...")
> 

done

> > +
> > +**Sensor** The sensor captures the incoming light and produces the raw images.
> 
> "and converts a measured analogue signal to a digital representation which conveys
> the raw images."

done

> 
> > +Data is commonly filtered by a color filter array into red, green and blue
> 
> Data? or light ... I think the light is filtered by a colour filter...

right :)

> 
> > +channels. As these filters are not perfect some postprocessing needs to be done
> > +to recreate the correct color.
> > +
> > +**BLC** Black level correction. Even without incoming light a sensor produces
> > +pixel values above zero. This is due to a artificially added pedestal value
> 
> s/due to a/due to an/

done

> 
> > +(Which is added to get a evenly distributed noise level around zero instead of
> > +only the upper half) and other effects in the electronics light dark currents.
> 
> Rather than bracket that, I'd suggest:
> 
> "
> 
> This is due to two system artifacts that impact the measured light
> levels on the sensor. Firstly, a deliberate and artificially added
> pedestal value is added to get an evenly distributed noise level around
> zero to avoid negative values and clipping the data.
> 
> Secondly, additional underlying electrical noise can be caused by
> various external factors including thermal noise and electrical
> interferences.

taken

> 
> > +To get good images with real black, that black level needs to be subtracted. As
> > +that level is typically known for a sensor it is hardcoded in libcamera and does
> > +not need any calibration at the moment.
> 
> "As the pedestal value is typically known" (we don't know the electrical noise
> component)

I'm not sure here. To my understanding some sensors (e.g. AR0521)
regularly measure the optical black and adjust the internal regulation.
This would then the electrical noise component also.

> 
> Is it hardcoded though? Can't it be overridden by the tuning file still?

Yes, that is still possible. I added a sentence for that.

> 
> > +
> > +**AWB** Auto white balance. For a proper image the color channels need to be
> 
> I'm not sure if 'proper image' is defined well. But I haven't got a
> better option in my head yet...
> 
> > +adjusted, to get correct white balance. This means that monochrome objects in
> > +the scene appear monochrome in the output image (white is white and gray is
> > +gray). In libcamera this is done based on a gray world model. No tuning is
> > +necessary for this step.
> 
> "Presently in the libipa implementation of libcamera, this is managed by
> a grey world model." ?
> 

done

> 
> > +
> > +**LSC** Lens shading correction. The lens in use has a big influence on the
> > +resulting image. The typical effects on the image are lens-shading (also called
> > +vignetting). This means that due to the physical properties of a lens the
> > +transmission of light falls of to the corners of the lens. To make things even
> 
> 'falls off towards the corners' ?

done

> 
> > +harder, this falloff can be different for different colors/wave lengths. LSC is
> 
> s/wave lengths/wavelengths/

done

> 
> 
> > +therefore tuned for a set of light sources and for each color channel
> > +individually.
> > +
> > +**CCM** Color correction matrix. After the previous processing blocks the grays
> > +are preserved, but colors are still not correct. This is mostly due to the
> > +color temperature of the light source and the imperfections in the color filter
> > +array of the sensor. To correct for this a so called color correction matrix is
> 
> s/a so called color correction matrix/a 'color correction matrix'/ ?

done

> 
> > +calculated. This is a 3x3 matrix, that is used to optimize the captured colors
> > +regarding the perceived color error. To do this a chart of known and precisely
> > +measured colors (macbeth chart) is captured. Then the matrix is calculated
> 
> s/(macbeth chart)/commonly called a macbeth chart/

done

> 
> > +using linear optimization to best map each measured colour of the chart to it's
> > +known value. The color error is measured in `deltaE
> > +<https://en.wikipedia.org/wiki/Color_difference>`_.
> > +
> > +**Gamma** Gamma correction. Today's display usually apply a gamma of 2.2 for
> > +display. For images to be perceived correctly by the human eye, they need to be
> > +encoded with the corresponding inverse gamma. See also
> > +<https://en.wikipedia.org/wiki/Gamma_correction>. This block doesn't need
> > +tuning, but is crucial for correct visual display.
> > +
> > +Materials needed for the tuning process
> > +---------------------------------------
> > +
> > +Precisely calibrated optical equipment is very expensive and out of the scope of
> > +this document. Still it is possible to get reasonably good calibration results
> > +at little costs. The most important devices needed are: 
> > +
> > +   - A light box with the ability to produce defined light of different color
> > +     temperatures. Typical temperatures used for calibration are 2400K
> > +     (incandescent), 2700K (fluorescent), 5000K (daylight fluorescent), 6500K
> > +     (daylight). As a first test, keylights for webcam streaming can be used.
> > +     These are available with support for color temperatures ranging from 2500K
> > +     to 9000K. For better results professional light boxes are needed.
> > +   - A ColorChecker chart. These are sold from calibrite and it makes sense to
> > +     get the original one.
> > +   - A integration sphere. This is used to create completely homogenious light
> > +     for the lens shading calibration. We had good results with the use of
> > +     a homogeneous area light (the above mentioned keylight).
> 
> I would perhaps call this "a large light panel with sufficient diffusion
> to get an even distribution of light."

done

> 
> 
> > +   - An environment without external light sources. Ideally calibration is done
> > +     without any external lights in a fully dark room. A black curtain is one
> > +     solution, working by night is the cheap alternative.
> > +
> 
> Indeed, I think I'm happier with a dark curtain so far so I don't have
> to stay up so late.

done

> 
> 
> 
> And I've hit 50% through the document, but run out of time so I'll snip
> and pause here!

Me too :-)

Thank you.

Cheers,
Stefan

> 
> <snip>
> --
> Kieran