Librem 5: hang with cam -C1

[Kieran Bingham]

Quoting Pavel Machek (2025-03-26 11:53:53)
> Hi!
> 
> > > With -C100, it actually works, for both front and back cameras.
> > 
> > great news \o/
> 
> :). There's still bug somewhere, -C2 results in two buffer captured
> and exit, -C1 results in hang, which is quite confusing.
> 
> > > > I'm afraid a more detailed log is needed.
> > > > Run the above command with
> > > >
> > > > LIBCAMERA_LOG_LEVELS=0
> > > >
> > > > As librem5 has no ISP but a RAW sensor as far as I remember (I still
> > > > can't figure out what they were thinking when they selected this weird
> > > > combination) make sure you have the SimplePipelineHandler and
> > > > SoftwareISP compiled in and enabled.
> > >
> > > Yep, that's correct. I'm afraid combination of RAW sensor and no ISP
> > 
> > In this specific case, the imx8mq SoC on the librem5 has no physical ISP
> > 
> > > or no drivers for ISP is going to be common for a while in phone area
> > > :-(.
> > 
> > That's why we're very glad to have the software ISP in libcamera and
> > see great improvements there from Milan and others ! ;)
> 
> Yes, so... I'd like to take a stab at implementing autofocus in
> software ISP, as I understand that one is still missing.
> 
> Any hints how to enable SoftwareISP?
> 
> meson configure build says
> 
> ipas               [ipu3, mali-c55,     [ipu3, mali-c55,     Select which IPA    
>                       rkisp1, rpi/pisp,    rkisp1, rpi/pisp,   modules to build    
>                       rpi/vc4, simple,     rpi/vc4, simple,                        
>                       vimc]                vimc]
> pipelines          [auto]               [all, auto,          Select which        
>                                            imx8-isi, ipu3,     pipeline handlers to
>                                            mali-c55, rkisp1,   build. If this is   
>                                            rpi/pisp, rpi/vc4,  set to "auto", all  
>                                            simple, uvcvideo,   the pipelines       
>                                            vimc, virtual]      applicable to the   
> 
> But I have no idea how if that's enough and what needs to be done at
> runtime.

You've mentioned looking at Autofocus for SoftISP:

Please note the following conversation that happened recently on IRC:

<NekoCWD[m]> Hello! I am trying to make the autofocus algorithm for
soft-ipa. I am having trouble getting the frame to measure contrast, how
can I do this best?

<NekoCWD[m]> BTW, I made [some
changes](https://gitlab.com/NekoCWD/libcamera) that add brightness/AE-on
control to the AGC and an AF stub that exports the focus-lens position
control. I also made [an app](https://gitlab.com/NekoCWD/pipetap) that
drives these controls via pipewire

<kbingham> NekoCWD[m], the SoftISP should have access to the frame, and
is where you would be able to measure the contrast value - and report
that to the IPA...

<kbingham> IPA modules dont' have access to the images themselves...

<NekoCWD[m]> Maybe somewhere in debayer?

<kbingham> src/libcamera/software_isp/ is likely the paths you'll need
to look into

<kbingham> yes, ideally the processing would have to happen at the same
time to be able to keep the CPU accesses to memory low.

<mzamazal[m]> we gather some stats in debayering code for efficiency.  I
don't know how contrast measurement works but maybe the data could be
gathered there too.

<mzamazal[m]> (definitely not pretty but for the sake of not wasting
time on memory accessess...)

<NekoCWD[m]> Thank you, will try to implement it in
␑src/libcamera/software_isp/swstats_cpu.cpp

<kbingham> Ultimately for contrast we're looking for a 'single' number
that represents the complete contrast value ... so I expect there is
some magic high-pass filter that could be applied to generate a single
number ... but I ... honestly have no idea how the contrast value is
calculated without doing some research

<NekoCWD[m]> I'm going to try  laplacian edge detection ago

* hansg (~hans at 2001:1c00:c32:7800:5bfa:a036:83f0:f9ec) has joined

<kbingham> hansg, meet NekoCWD[m], NekoCWD[m] meet hansg  ;-)

<hansg> NekoCWD[m], hi, kbingham just told me you are looking into
autofocus for the softwareISP. I actually already had a group of
students from my local university working on software autofocus. This is
not posted upstream yet because it builds on top of the atomisp
pipelinehandler which is not upstream yet

<hansg> The atomisp pipeline handler uses the softwareISP stats since we
don't know how to interpret the hwstats, so the code the students wrote
adds code to the swstats class to get a sharpness value + IPA changes to
find the focus control value with the highest sharpness

<hansg> IIRC they are using a 3x3 laplace transformation since that is
the fastest algorithm and the sharpness values it gives are good enough

<hansg> The students code is here:
https://github.com/CookieLover3000/libcamera/commits/software-af-done/
and I also have a backup here:
https://github.com/jwrdegoede/libcamera/tree/software-af-done

<hansg> This can of course use some work, but it should be a decent
starting point

<kbingham>
https://github.com/jwrdegoede/libcamera/commit/c58a0d66394dc7dfd25c37449e69ac19ecc4233a
definitely sounds like a starting point!

<NekoCWD[m]> I guess i can import complete af algo form here :D

<NekoCWD[m]>
https://github.com/jwrdegoede/libcamera/commits/software-af-done

<NekoCWD[m]> Stragely that i didn't find it before

<hansg> Well its been hidden quite well, I wanted the students to maybe
post it as a RFC on libcamera-devel but then we ran out of time (it was
a university assignment for them) so my plan was to post it myself once
I got the atomisp pipeline handler upstream and then life happened ...

<hansg> I'm quite happy to see there is interest in this now and that
you might pick this up.

<hansg> Note there is also a whole bunch of prep patches from me
renaming some things, adding AF related hooks etc which you should also
be able to pretty much apply 1:1 .

<pinchartl> hansg: NekoCWD[m]: hardware typically uses the tenengrad
function (sum of the squares of horizontal and vertical sobel filters,
with a 3x3 kernel)

<pinchartl> you need to normalize the value


> 
> Thanks,
>                                                                 Pavel
> -- 
> I don't work for Nazis and criminals, and neither should you.
> Boycott Putin, Trump, and Musk!