XISP : Creating new IPA for AMD/Xilinx HLS-based ISP

[Laurent Pinchart]

Hi Mario,

On Mon, Feb 10, 2025 at 02:09:11PM -0500, Mario Bergeron wrote:
> I have a preliminary "pipeline handler" working for my use case, with
> integration to a petalinux 2024.1 project with yocto recipes, pulling libcamera
> v0.4.0, and applying my modifications as patches.
> I have documented my progress here : https://avnet.me/
> tria-rpicam-fun-2024.1-part4

Really nice work !

> This is still using the HLS-based AWB algorithm, which is very very fast and
> stable, but not producing as good image quality as the raspberry pi 5.
> 
> I have captured a side-by-side video showing the image quality and adaptive
> rate of the AWB for both platforms : https://youtu.be/roYnDhWECWw
> 
> Before diving into creating a "IPA" for this use case, I need to determine
> which modifications I need to make to the PL-based capture pipeline:
> 
>   ● add capability to capture RAW images (? do I need this ?)

This isn't strictly required for normal operation, but you will need it
for tuning purpose.

>   ● add capture of meta-data (containing exposure/gain registers for
>     imx219,imx708,imx477,imx500, in addition to classification results and/or
>     detection boxes for imx500)

Capturing CSI-2 embedded data is very useful, as it improves stability
of AGC, but it's not mandatory.

>   ● expose image statistics to linux driver (currently not exposed to host CPU)

That would be required to implement AGC, or host-side AWB.

>   ● probably augment the statistics that are being generated
> 
> 
> Which statistics should be generated for the host CPU ?

Typically, for AWB, you will want colour means with unsaturated pixel
counts (ideally calculated on a grid with separate accumulators for each
grid element). AGC can use a histogram, or mean luminance data
(grid-based can be useful here too).

> Which soft ISP should I refer to for my initial implementation ?

Do you mean which IPA module ? I would recommend looking at both the
Raspberry Pi IPA module (it has the most advanced algorithms at the
moment) and the rkisp1 IPA module (it's catching up, and is being
designed to share code with other IPA modules).

One interesting challenge will be to find out how to cope with the
programmability of the FPGA, whose ISP pipeline can be configured and
tweaked in many ways, while keeping the software implementation in
libcamera neat and clean.

> Any recommendations would be greatly appreciated.
> 
> Also, any collaboration is welcome !

-- 
Regards,

Laurent Pinchart