[libcamera-devel] libcamera on Librem 5, phase detection auto focus

[Laurent Pinchart]

Hi Dorota,

On Mon, Nov 21, 2022 at 02:15:28PM +0100, Dorota Czaplejewicz wrote:
> On Mon, 21 Nov 2022 14:55:49 +0200 Laurent Pinchart wrote:
> 
> > Hi Pavel,
> > 
> > (CC'ing Dorota)
> > 
> > On Mon, Nov 21, 2022 at 12:32:30PM +0100, Pavel Machek via libcamera-devel wrote:
> > > Hi!
> > > 
> > > I'm playing with camera on Librem 5. So far I added af/ae/awb support
> > > to millipixels done completely in software, but I guess libcamera
> > > would be more suitable place for it. If you have some hints/example
> > > where to look at, it would be nice.  
> > 
> > The target for the Librem 5 is to use the GPU to do the heavy-lifting,
> > and in particular to perform CFA interpolation (a.k.a. debayering) and
> > to apply the colour gains. Dorota is working on this. The, for the AE
> > and AWB statistics, we should make use of the CSI bridge stats engine if
> > possible. This will require some work on the driver side (this feature
> > isn't implemented at the moment), and integrate that in libcamera.
> > 
> > We could also do all this purely in software, but I'm not sure it's
> > worth it here. libcamera doesn't have the necessary infrastructure at
> > the moment, as all the platforms we've worked on have an ISP or use YUV
> > sensors. It could be done, but I think the effort would be better
> > invested, on the Librem 5, in making use of the hardware features.
> 
> Right now I'm aiming at doing the statistics on the GPU. Barring any
> extra snags, this will start getting integrated into my experimental
> branch before the end of the year.

I'm curious to see how that will look like. I've seen a few clever hacks
to calculate histograms on a GPU, but overall, GPUs (at least with
OpenGL shaders) seem ill-suited for that kind of processing. OpenCL may
help, I haven't really checked.

Would you consider trying statistics computation with the CSI bridge ?
It should be more or less free, at least when it comes to CPU or GPU
time.

> > > So far I'm working with contrast detection, but samsung chip should
> > > support phase detection auto focus, too. So far I hacked
> > > drivers/media/i2c/s5k3l6xx.ko not to filter out those bits and took
> > > photos (and looked at them in darktable), but I don't see any obvious
> > > "defects" indicating presense of focus data. Do you have any pointers
> > > to more information by chance?
> > > 
> > > I took deliberately defocused photo to make "defects" easier to see. I
> > > guess I should take another one with less gain to keep other noise
> > > down. Any ideas how to locate those bits?  
> > 
> > The PDAF pixels will be located at fixed positions in the pixel array,
> > documented by the sensor vendor. Some sensors will process the PDAF
> > pixels and provide the processed information in the CSI-2 embedded data.
> > I don't have access to the sensor datasheet, so I can't help much I'm
> > afraid :-(
> 
> The focus pixels will appear sometimes - I haven't quite figured out
> when - when the resolution is full and when denoising is off. That's
> not described in the datasheet very well. I did prepare a bunch of
> scripts to mess with the sensors registers in this repo (beware, here
> be dragons) https://source.puri.sm/dorota.czaplejewicz/nani
> 
> Otherwise, I tried to describe the registers in the s5k3l6xx driver,
> inline. That driver isn't mainline btw, so only the folks at Purism
> have any experience with it.
> 
> The problem on the L5 is that the full resolution is too much for the
> CPU to handle, so the debayering/scaling must at least be done on the
> GPU, or the frame rate will be horrible and the entire exercise of
> phase-shift focusing useless.

-- 
Regards,

Laurent Pinchart