[libcamera-devel] [RFC PATCH] [RFC] [DNI] utils: tuning: Draft a new tuning infrastructure

[Laurent Pinchart]

Hi Paul,

On Mon, Sep 26, 2022 at 02:09:31PM +0900, Paul Elder wrote:
> This patch introduces the interface of a new tuning infrastructure, with
> samples of how I imagine different platforms would use it (aptly named
> "libtuning" for now) to create their own tuning scripts.
> 
> The samples include (in reverse order of how they appear in the patch,
> but in semantic order of how I want to show them):
> - rkisp1, which for now only has ALSC, and comes with most of the
>   descriptions in the form of comments
> - raspberrypi's alsc_only, which is similar to rkisp1's, to show how
>   different platforms could implement similar-but-slightly-different
>   tuning scripts without much duplication
> - raspberrypi's main one, which only has ALSC and a dummy AWB one for
>   now
> 
> The two raspberrypi tuning scripts have a shared ALSC component, which
> is in a separate file to show where platforms could place custom
> components if libtuning doesn't support exactly what the platform's
> tuning script wants.
> 
> Obviously none of this runs (hence the DNI) and none of it is
> implemented yet, but I wanted to show how it would be used first and to
> gather comments on it.
> 
> There are some questions nested in the descriptions comments as well.
> 
> Signed-off-by: Paul Elder <paul.elder at ideasonboard.com>
> ---
>  utils/tuning/libtuning/__init__.py         |  0
>  utils/tuning/libtuning/modules/__init__.py |  0
>  utils/tuning/raspberrypi.py                | 37 ++++++++++++
>  utils/tuning/raspberrypi/__init__.py       |  0
>  utils/tuning/raspberrypi/alsc.py           | 16 ++++++
>  utils/tuning/raspberrypi_alsc_only.py      | 16 ++++++
>  utils/tuning/rkisp1.py                     | 67 ++++++++++++++++++++++
>  7 files changed, 136 insertions(+)
>  create mode 100644 utils/tuning/libtuning/__init__.py
>  create mode 100644 utils/tuning/libtuning/modules/__init__.py
>  create mode 100644 utils/tuning/raspberrypi.py
>  create mode 100644 utils/tuning/raspberrypi/__init__.py
>  create mode 100644 utils/tuning/raspberrypi/alsc.py
>  create mode 100644 utils/tuning/raspberrypi_alsc_only.py
>  create mode 100644 utils/tuning/rkisp1.py
> 
> diff --git a/utils/tuning/libtuning/__init__.py b/utils/tuning/libtuning/__init__.py
> new file mode 100644
> index 00000000..e69de29b
> diff --git a/utils/tuning/libtuning/modules/__init__.py b/utils/tuning/libtuning/modules/__init__.py
> new file mode 100644
> index 00000000..e69de29b
> diff --git a/utils/tuning/raspberrypi.py b/utils/tuning/raspberrypi.py
> new file mode 100644
> index 00000000..322b9c36
> --- /dev/null
> +++ b/utils/tuning/raspberrypi.py
> @@ -0,0 +1,37 @@
> +import sys
> +
> +import libtuning as lt
> +from libtuning.modules import AWB
> +from libtuning.parsers import RaspberryPiParser
> +from libtuning.generators import RaspberryPiOutput
> +
> +from raspberrypi.alsc import ALSC as RaspberryPiALSC
> +
> +tuner = lt.Camera()

Nitpicking on naming, it feels weird to name a variable "tuner" when it
is of type "Camera".

> +
> +# These modules can also be custom modules. libtuning will come with utilities
> +# for handling stuff like images, so there shouldn't be too much boilerplate
> +# involved in creating custom modules, though I don't yet have a concrete
> +# vision on how custom implementations of modules would look.
> +tuner.add(RaspberryPiALSC)
> +
> +# Other tuning modules can be added like so.
> +# The order that the tuning modules will be executed is determined by the order
> +# that they're added.
> +# This is kind of an implementation detail, but the "context" is saved
> +# internally in lt.Camera, so modules that are added (and therefore executed)
> +# later can use the output of the previous modules. I'm thinking that a module
> +# that depends on values from another module has two modes of execution, for
> +# when those values are available and another for when they're not. Not quite
> +# sure concretely how to handle this yet.
> +tuner.add(AWB( # module parameters
> +               ))
> +
> +tuner.setInputType(RaspberryPiParser)
> +tuner.setOutputType(RaspberryPiOutput)
> +
> +# The order of the output doesn't necessarily have to be the same as the order
> +# of input, which is specified by the order of adding the modules above.
> +tuner.setOutputOrder(AWB, ALSC)
> +
> +tuner.run(sys.argv)
> diff --git a/utils/tuning/raspberrypi/__init__.py b/utils/tuning/raspberrypi/__init__.py
> new file mode 100644
> index 00000000..e69de29b
> diff --git a/utils/tuning/raspberrypi/alsc.py b/utils/tuning/raspberrypi/alsc.py
> new file mode 100644
> index 00000000..ff8a02fd
> --- /dev/null
> +++ b/utils/tuning/raspberrypi/alsc.py
> @@ -0,0 +1,16 @@
> +ALSC(do_color = lt.paramsIfUnset(True), \
> +               debug = [lt.debug.Plot], \
> +               luminance_strength = lt.params.IfUnSet(0.5) \
> +               sector_shape = (16, 12), \
> +               sector_x_gradient = lt.gradients.Linear, \
> +               sector_y_gradient = lt.gradients.Linear, \
> +               sector_x_remainder = lt.remainder.Append, \
> +               sector_y_remainder = lt.remainder.Append, \
> +               sector_average_function = lt.average_functions.Mean, \
> +               smoothing_function = lt.smoothing.MedianBlur, \
> +               smoothing_parameters = [3], \
> +               output_type = lt.type.Float, \
> +               output_color_channels = [lt.color.R, lt.color.G, lt.color.B], \
> +               output_luminance_channels = [lt.color.G] \
> +               output_range = [0, 3.999] \
> +               )
> diff --git a/utils/tuning/raspberrypi_alsc_only.py b/utils/tuning/raspberrypi_alsc_only.py
> new file mode 100644
> index 00000000..11a1fc23
> --- /dev/null
> +++ b/utils/tuning/raspberrypi_alsc_only.py
> @@ -0,0 +1,16 @@
> +import sys
> +
> +import libtuning as lt
> +from libtuning.modules import ALSC
> +from libtuning.parsers import RaspberryPiParser
> +from libtuning.generators import RaspberryPiOutput
> +
> +from raspberrypi.alsc import ALSC as RaspberryPiALSC
> +
> +tuner = lt.Camera()
> +tuner.add(RaspberryPiALSC)
> +tuner.setInputType(RaspberryPiParser)
> +tuner.setOutputType(RaspberryPiOutput)
> +tuner.setOutputOrder(ALSC)
> +
> +tuner.run(sys.argv)
> diff --git a/utils/tuning/rkisp1.py b/utils/tuning/rkisp1.py
> new file mode 100644
> index 00000000..8e26beba
> --- /dev/null
> +++ b/utils/tuning/rkisp1.py
> @@ -0,0 +1,67 @@
> +import sys
> +
> +import libtuning as lt
> +from libtuning.modules import ALSC
> +from libtuning.parsers import YamlParser
> +from libtuning.generators import YamlOutput
> +
> +tuner = lt.Camera()
> +tuner.add(ALSC(do_color = lt.paramsIfUnset(True), \
> +
> +               # This can support other debug options (I can't think of any rn
> +               # but for future-proofing)
> +               debug = [lt.debug.Plot], \
> +
> +               # The name of IfUnSet can obviously be changed, but really I
> +               # just want something that says "it must be specified in the
> +               # configuration, and get the value from there" or "if the value
> +               # is not in the configuration, use this"
> +               luminance_strength = lt.params.IfUnSet(0.5) \
> +
> +               sector_shape = (16, 16), \
> +
> +               # Other functions might include Circular, Hyperbolic, Gaussian,
> +               # Linear, Exponential, Logarithmic, etc
> +               # Of course, both need not be the same function
> +               # Some functions would need a sector_x_parameter (eg. sigma for Gaussian)
> +               # Alternatively: sector_x_sizes = [] ? I don't think it would work tho
> +               sector_x_gradient = lt.gradients.Parabolic, \
> +               sector_y_gradient = lt.gradients.Parabolic, \
> +
> +               # This is the function that will be used to average the pixels in each sector
> +               # This can also be a custom function.
> +               sector_average_function = lt.average_functions.Mean, \
> +
> +               # This is the function that will be used to smooth the color ratio values
> +               # This can also be a custom function.
> +               smoothing_function = lt.smoothing.MedianBlur, \
> +               smoothing_parameters = [3], \
> +
> +               # Are there any platforms that use integer values for their lsc table?
> +               output_type = lt.type.Float, \
> +
> +               # Required if and only if do_color can be or is True
> +               output_color_channels = [lt.color.R, lt.color.GR, lt.color.GB, lt.color.B], \
> +
> +               # Required if and only if do_color can be or is False
> +               output_luminance_channels = [lt.color.GR, lt.color.GB], \
> +
> +               # Automatically get the precision from this
> +               output_range = [0, 3.999] \
> +
> +               # Do we need a flag to enable/disable calculating sigmas? afaik
> +               # the rkisp1 IPA doesn't use it? But we could output it anyway
> +               # and algorithms can decide whether or not if they want to use
> +               # it. Oh well, no flag for now, we can always add it later if
> +               # there's a big demand, plus it's only two to three values and
> +               # not an entire table.
> +               ))
> +tuner.setInputType(YamlParser)
> +tuner.setOutputType(YamlOutput)
> +tuner.setOutputOrder(ALSC)

I may be missing something obvious, but isn't tuning supposed to use
images ? :-)

> +
> +# Maybe this should be wrapped in an if __main__ = '__main__' ? That way the

__name__ == '__main__'

> +# developer can control which tuner they want to be executed based on another
> +# layer of arguments? But I was thinking that this would handle *all* arguments
> +# based on the modules' and the input/output configurations.
> +tuner.run(sys.argv)

It looks like you have quite a few ideas on how to implement all this,
but based on this skeleton only, I have a hard time understanding them
:-) It would make sense to me to move forward a bit more and then
continue the discussion.

-- 
Regards,

Laurent Pinchart