[libcamera-devel] [RFC] media: Auto exposure/gain support for atomisp / libcamera integration ?

[Umang Jain]

Hello,

On 11/7/21 11:20 PM, Hans de Goede wrote:
> Hi All,
>
> Now that we have the atomisp2 driver running on some devices like
> the Asus T101HA; and with the exposure + gain controls for the ov2680
> sensor found on that laptop fixed:
>
> https://lore.kernel.org/linux-media/20211107171549.267583-1-hdegoede@redhat.com/
>
> I believe we need to start thinking about various userspace API
> concerns. Thanks to Mauro's great work on various V4L2 API things
> are starting to work (somewhat) with regular V4L2 apps, but we really
> also need some processing / 3A stuff to make the cameras really
> usable.
>
> The most important thing needed is automatic exposure + gain control,
> ATM this relies on a custom ATOMISP ioctl, but I believe that we can
> just export the controls as regular v4l2 controls on the sensor subdev,
> at least for the ov2680 the exposure is already exported this way
> but it is read-only. Does anyone see any problems with just making
> these normal v4l2 controls on the sensor subdev ?
>
> We can then simulate the custom ATOMISP ioctl through the subdevs,
> or we can just remove it alltogether.
>
> Once we have the controls available this way I think we should write
> a libcamera plugin, which like the older versions of the Rasberry Pi
> plugin (if I've understood things correctly) won't use the MC framework
> for now. I believe we first need to understand the atomisp code better
> before we add MC support (*). But I still believe that an (experimental)
> plugin would be good, both to get something usable so that we can get
> more testers / more people interested in contributing.


I am trying to understand what 'plugin' here means? Is it a wrapper 
pertaining to use libcamera (fined tuned for atomisp) that apps can use?

> Another reason is to have another use-case where apps need to support
> libcamera to work properly (like on IPU3 devices) which will hopefully
> motivate people working on apps to put more effort in supporting libcamera


A valid and solid use case yes!

> (preferably through the new pipewire libcamera plugin so that things
> will also work in a flatpack sandbox).


In the longer term plan, I see this happening. I see there SPA support 
for libcamera in pipewire (not sure how much usable it is as of today). 
Once pipewire has a translating layer of ' request-controls' that can be 
mapped to libcamera controls, it would then make good base for 
applications for capturing video feeds by sending in requests with 
appropriate controls.

On the flatpak side, I think there will be more? plumbing work since  
you need to use flatpak-portals, rather than just bundling the library 
in the manifest (the sandbox cannot determine system's h/w capabilites). 
The current org.freedesktop.portal.Camera [1] seems to be geared to use 
pipewire so that's a starting point. CV applications are yet another 
use-case libcamera will be interested in, I think. Getting the 
flatpak-portal support there might get tricky as of 'quirky' requests? 
for the camera and pipewire API seems to be limiting to support that 
use-case? Not sure how would this work out (and also a distant future),

[1] 
https://flatpak.github.io/xdg-desktop-portal/portal-docs.html#gdbus-org.freedesktop.portal.Camera

>
> ###
>
> On other thing which I'm wondering about is the need to call S_INPUT to
> select front / back in this list from Mauro:
>
> 	$ for i in $(ls /dev/video*|sort -k2 -to -n); do echo -n $i:; v4l2-ctl -D -d $i|grep Name; done
> 	/dev/video0:	Name             : ATOMISP ISP CAPTURE output
> 	/dev/video1:	Name             : ATOMISP ISP VIEWFINDER output
> 	/dev/video2:	Name             : ATOMISP ISP PREVIEW output
> 	/dev/video3:	Name             : ATOMISP ISP VIDEO output
> 	/dev/video4:	Name             : ATOMISP ISP ACC
> 	/dev/video5:	Name             : ATOMISP ISP MEMORY input
> 	/dev/video6:	Name             : ATOMISP ISP CAPTURE output
> 	/dev/video7:	Name             : ATOMISP ISP VIEWFINDER output
> 	/dev/video8:	Name             : ATOMISP ISP PREVIEW output
> 	/dev/video9:	Name             : ATOMISP ISP VIDEO output
> 	/dev/video10:	Name             : ATOMISP ISP ACC
>
> I notice that everything is listed twice, I wonder if we can use /dev/video2
> with input 0 together with /dev/video8 for input 1, if that is possible then
> we might set a different default input on each.
>
> And maybe also make them /dev/video0 (already done I see) and /dev/video1,
> possibly combined with a module-option to hide the others for now. This
> should make things better for regular apps. OTOH if we go the libcamera
> route then this is not really necessary I guess?
>
> Regards,
>
> Hans
>
> *) I do believe that in the end MC support makes sense at least
> to tie together the
>