[libcamera-devel] [RFC] media: Auto exposure/gain support for atomisp / libcamera integration ?

[Laurent Pinchart]

Hi Umang,

On Thu, Nov 18, 2021 at 03:43:53PM +0530, Umang Jain wrote:
> On 11/7/21 11:20 PM, Hans de Goede wrote:
> > Hi All,
> >
> > Now that we have the atomisp2 driver running on some devices like
> > the Asus T101HA; and with the exposure + gain controls for the ov2680
> > sensor found on that laptop fixed:
> >
> > https://lore.kernel.org/linux-media/20211107171549.267583-1-hdegoede@redhat.com/
> >
> > I believe we need to start thinking about various userspace API
> > concerns. Thanks to Mauro's great work on various V4L2 API things
> > are starting to work (somewhat) with regular V4L2 apps, but we really
> > also need some processing / 3A stuff to make the cameras really
> > usable.
> >
> > The most important thing needed is automatic exposure + gain control,
> > ATM this relies on a custom ATOMISP ioctl, but I believe that we can
> > just export the controls as regular v4l2 controls on the sensor subdev,
> > at least for the ov2680 the exposure is already exported this way
> > but it is read-only. Does anyone see any problems with just making
> > these normal v4l2 controls on the sensor subdev ?
> >
> > We can then simulate the custom ATOMISP ioctl through the subdevs,
> > or we can just remove it alltogether.
> >
> > Once we have the controls available this way I think we should write
> > a libcamera plugin, which like the older versions of the Rasberry Pi
> > plugin (if I've understood things correctly) won't use the MC framework
> > for now. I believe we first need to understand the atomisp code better
> > before we add MC support (*). But I still believe that an (experimental)
> > plugin would be good, both to get something usable so that we can get
> > more testers / more people interested in contributing.
> 
> I am trying to understand what 'plugin' here means? Is it a wrapper 
> pertaining to use libcamera (fined tuned for atomisp) that apps can use?

I think it means pipeline handler (and IPA module).

> > Another reason is to have another use-case where apps need to support
> > libcamera to work properly (like on IPU3 devices) which will hopefully
> > motivate people working on apps to put more effort in supporting libcamera
> 
> A valid and solid use case yes!
> 
> > (preferably through the new pipewire libcamera plugin so that things
> > will also work in a flatpack sandbox).
> 
> In the longer term plan, I see this happening. I see there SPA support 
> for libcamera in pipewire (not sure how much usable it is as of today). 
> Once pipewire has a translating layer of ' request-controls' that can be 
> mapped to libcamera controls, it would then make good base for 
> applications for capturing video feeds by sending in requests with 
> appropriate controls.
> 
> On the flatpak side, I think there will be more? plumbing work since  
> you need to use flatpak-portals, rather than just bundling the library 
> in the manifest (the sandbox cannot determine system's h/w capabilites). 
> The current org.freedesktop.portal.Camera [1] seems to be geared to use 
> pipewire so that's a starting point. CV applications are yet another 
> use-case libcamera will be interested in, I think. Getting the 
> flatpak-portal support there might get tricky as of 'quirky' requests? 
> for the camera and pipewire API seems to be limiting to support that 
> use-case? Not sure how would this work out (and also a distant future),
> 
> [1] https://flatpak.github.io/xdg-desktop-portal/portal-docs.html#gdbus-org.freedesktop.portal.Camera
> 
> > ###
> >
> > On other thing which I'm wondering about is the need to call S_INPUT to
> > select front / back in this list from Mauro:
> >
> > 	$ for i in $(ls /dev/video*|sort -k2 -to -n); do echo -n $i:; v4l2-ctl -D -d $i|grep Name; done
> > 	/dev/video0:	Name             : ATOMISP ISP CAPTURE output
> > 	/dev/video1:	Name             : ATOMISP ISP VIEWFINDER output
> > 	/dev/video2:	Name             : ATOMISP ISP PREVIEW output
> > 	/dev/video3:	Name             : ATOMISP ISP VIDEO output
> > 	/dev/video4:	Name             : ATOMISP ISP ACC
> > 	/dev/video5:	Name             : ATOMISP ISP MEMORY input
> > 	/dev/video6:	Name             : ATOMISP ISP CAPTURE output
> > 	/dev/video7:	Name             : ATOMISP ISP VIEWFINDER output
> > 	/dev/video8:	Name             : ATOMISP ISP PREVIEW output
> > 	/dev/video9:	Name             : ATOMISP ISP VIDEO output
> > 	/dev/video10:	Name             : ATOMISP ISP ACC
> >
> > I notice that everything is listed twice, I wonder if we can use /dev/video2
> > with input 0 together with /dev/video8 for input 1, if that is possible then
> > we might set a different default input on each.
> >
> > And maybe also make them /dev/video0 (already done I see) and /dev/video1,
> > possibly combined with a module-option to hide the others for now. This
> > should make things better for regular apps. OTOH if we go the libcamera
> > route then this is not really necessary I guess?
> >
> > Regards,
> >
> > Hans
> >
> > *) I do believe that in the end MC support makes sense at least
> > to tie together the

-- 
Regards,

Laurent Pinchart