[libcamera-devel] libcamera on Librem 5, phase detection auto focus

[Pavel Machek]

Hi!

> > PDAF. It was easier than expected (and reverse engineering similar
> > chip should be relatively easy) and I got it to work. Fun project :-).
> 
> That's great to hear :-) Do you have a high-level, executive summary of
> what was required ? I would assume splitting the "right" and "left" PDAF
> pixels into two images, and computing cross-correlation with different
> offsets ?

Yes, that's pretty much it. Pixels come in groups of two, one of them
is left, one is right, you split them into two lines, and compute
cross-correlations.

What was surprising to me is that cross-correlation only show more
than one pixel in extremely unfocused images, and best correlation can
be at 0 shift with image still out of focus. So it is neccessary to
get at sub-pixel precision.

Plus for some reason AF pixels on this chip are brighter then the
rest, and turn to 255 in either complete darkness and complete
white. "Interesting", but easy to work around.

(And I'm wondering why the AF pixels are blue. There's already 2x more
green pixels than blue, and now some blue are used by AF).

> > It needs 13Mp images, which is a bit of challenge from performance
> > standpoint.
> 
> Do you mean the performance impact on the processing of the image data
> (debayering, ...) in full resolution, or is the performance of PDAF
> pixels processing an issue too ?

PDAF processing is quite trivial, fortunately. I only process single
line and did not really optimize the algorithm.

But when millipixels asks libcamera for 13MPix, we only get 8fps for
some reason... We do debayer+downscale in one step, so it is same
ammount of CPU for 13Mpix image and 1Mpix, but it is more memory bandwidth.

> Another issue is that you'll need software defective pixel correction to
> hide the PDAF pixels in the final image, so that's another performance
> hit. I expect the GPU to help there eventually.

No. :-). Actually, that's more complex.

So, for preview, we really need downscaled image, so it should be
possible to select unaffected pixels for display. And we really should
switch from bayer8 to bayer10 for image capture (we currently don't),
so we can turn off AF pixels at the same time (hw can do that).

So.. yes, GPU would be nice for debayer. GPU would be really nice for
JPEG/MP4 encoding, as we want video. But we are not nearly using CPUs
to their full potential, and I can optimize for CPUs.

Best regards,
								Pavel
-- 
People of Russia, stop Putin before his war on Ukraine escalates.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20221122/42322a48/attachment.sig>