<div dir="ltr"><div dir="ltr">Hi Jacopo,</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 21 Sept 2021 at 14:47, Jacopo Mondi <<a href="mailto:jacopo@jmondi.org">jacopo@jmondi.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi David,<br>
<br>
On Thu, Sep 16, 2021 at 02:20:13PM +0100, David Plowman wrote:<br>
> Hi everyone<br>
><br>
> Here's a first attempt at functionality that allows applications to<br>
> provide "hints" as to what kind of camera mode they want.<br>
><br>
> 1. Bit Depths and Sizes<br>
><br>
> So far I'm allowing hints about bit depth and the image sizes that can<br>
> be read out of a sensor, and I've gathered these together into<br>
> something I've call a SensorMode.<br>
><br>
> I've added a SensorMode field to the CameraConfiguration so that<br>
> applications can optionally put in there what they think they want,<br>
> and also a CameraSensor::getSensorModes function that returns a list<br>
> of the supported modes (raw formats only...).<br>
<br>
This is more about the theory, but do you think it's a good idea<br>
to allow applications to be written based on some sensor-specific<br>
parameter ? I mean, I understand it's totally plausible to write a<br>
specialized application for say, RPi + imx477. This assumes the<br>
developers knows the sensor modes, the sensor configuration used to<br>
produce it and low level details of the sensor. Most of those<br>
information assumed to be known by the developer also because there's<br>
currently no way to expose from V4L2 how a mode has been<br>
realized in the sensor, if through binning or skipping or cropping. It<br>
can be deduced probably looking at the analogue cropping and full<br>
pixel array size, but it's a guessing exercise.<br>
<br>
Now, if we assume that a very specialized application knows the sensor<br>
it deals with, what is the use case for writing a generic application<br>
that instead inspects the (limited) information of the sensor produced<br>
modes and selects its favourite one generically enough ? Wouldn't it<br>
be better to just assume the application precisely knows what RAW<br>
formats it wants and adds a StreamConfiguration for it ?<br></blockquote><div><br></div><div>This type of usage may be more relevant for Raspberry Pi than other vendors,</div><div>though it does not need to be of course. Our current (non-libcamera based)</div><div>raspicam applications allows the user to select a specific sensor mode to use</div><div>with the "-md" command line parameter with an integer specifying the mode index</div><div>to use. The modes for each of our sensors are fully documented so the user</div><div>knows the exact width/height/crop/bin/bit-depth used. This is one of those features</div><div>that our users have found extremely useful in a wide range of situations, and we</div><div>can't really live without it. Examples of this are using a low bit depth mode for</div><div>fast fps operation, or using binned mode for better noise performance, or forcing</div><div>a specific FoV to name a few.</div><div><br></div><div>The idea behind this change is to empower the users to do similar things with our</div><div>libcamera based apps. Currently, there is no full substitute for our existing mechanism.</div><div>The goal is not really to write an application that is specialised to a particular sensor,</div><div>rather we allow the user a mechanism to choose a particular advertised mode through</div><div>this new mechanism. This can be done either via some command line parameters, or</div><div>even programmatically, that's up to the user/app. So applications still remain generic,</div><div>but now allow the users to have a certain level of sensor specific control. Of course, an</div><div>application may choose to not bother with any of this, and then we revert to the pipeline</div><div>handlers having full choice in the sensor mode to use, like we do right now.</div><div><br></div><div>I'll let David respond to the below comments in more detail, but I just wanted to</div><div>provide some further context for what this change is about. Hope that helps!</div><div><br></div><div>Regards,</div><div>Naush</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
><br>
> There are various ways an application could use this:<br>
><br>
> * It might not care and would ignore the new field altogether. The<br>
> pipeline handler will stick to its current behaviour.<br>
><br>
> * It might have some notion of what it wants, perhaps a larger bit<br>
> depth, and/or a range of sizes. It can fill some or all of those<br>
> into the SensorMode and the pipeline handler should respect it.<br>
><br>
> * Or it could query the CameraSensor for its list of SensorModes and<br>
> then sift through them looking for the one that it likes best.<br>
<br>
This is the part I fail to fully grasp. Could you summarize what are<br>
the parameters that would drive the mode selection policy in the<br>
application ?<br>
<br>
><br>
> 2. Field of View and Framerates<br>
><br>
> The SensorMode should probably include FoV and framerate information<br>
<br>
FoV it's problematic, I understand. We have a property that describes<br>
the pixel array size, and I understand comparing the RAW output size<br>
is not enough as the same resolution can be theoretically be obtained<br>
by cropping or subsampling. I would not be opposite to report it<br>
somehow as it might represent an important selection criteria.<br>
<br>
Duration, on the other hand can't be read from the limits of<br>
controls::FrameDurationLimits ? I do expect those control to be<br>
populated with the sensor's durations (see<br>
<a href="https://git.linuxtv.org/libcamera.git/tree/src/ipa/ipu3/ipu3.cpp#n256" rel="noreferrer" target="_blank">https://git.linuxtv.org/libcamera.git/tree/src/ipa/ipu3/ipu3.cpp#n256</a>)<br>
<br>
Sure, applications should try all the supported RAW modes, configure<br>
the camera with them, and read back the control value.<br>
<br>
> so that applications can make intelligent choices automatically.<br>
> However, this is a bit trickier for various reasons so I've left it<br>
> out. There could be a later phase of work that adds these.<br>
><br>
> Even without this, however, the implementation gets us out of our<br>
> rather critical hole where we simply can't get 10-bit modes. It also<br>
<br>
Help me out here: why can't you select a RAW format with 10bpp ?<br>
<br>
> provides a better alternative to the current nasty practice of<br>
> requesting a raw stream specifically to bodge the camera mode<br>
> selection, even when the raw stream is not actually wanted!<br>
<br>
I see it the other way around actually :) Tying application to sensor<br>
modes goes in the opposite direction of 'abstracting camera<br>
implementation details from application' (tm)<br>
<br>
Also goes in the opposite direction of the long-term dream of having<br>
sensor driver not being tied to a few fixed modes because that's what<br>
producers gave us to start with, but I get this is a bit far-fetched.<br>
<br>
Of course the line between abstraction and control is as usual draw on<br>
the sand and I might be too concerned about exposing sensor details in<br>
our API, even if in an opt-in way.<br>
<br>
Thanks<br>
j<br>
<br>
><br>
> 3. There are 2 commits here<br>
><br>
> The first adds the SensorMode class, puts it into the<br>
> CameraConfiguration, and allows the supported modes to be listed from<br>
> the CameraSensor. (All the non-Pi stuff.)<br>
><br>
> The second commit updates our mode selection code to select according<br>
> to the hinted SensorMode (figuring out defaults if it was empty). But<br>
> it essentially works just the same, if in a slightly more generic way.<br>
><br>
> The code here is fully functional and seems to work fine. Would other<br>
> pipeline handlers be able to adapt to the idea of a "hinted<br>
> SensorMode" as easily?<br>
><br>
><br>
> As always, I'm looking forward to people's thoughts!<br>
><br>
> Thanks<br>
> David<br>
><br>
> David Plowman (2):<br>
> libcamera: Add SensorMode class<br>
> libcamera: pipeline_handler: raspberrypi: Handle the new SensorMode<br>
> hint<br>
><br>
> include/libcamera/camera.h | 3 +<br>
> include/libcamera/internal/camera_sensor.h | 4 +<br>
> include/libcamera/meson.build | 1 +<br>
> include/libcamera/sensor_mode.h | 50 +++++++++<br>
> src/libcamera/camera_sensor.cpp | 15 +++<br>
> src/libcamera/meson.build | 1 +<br>
> .../pipeline/raspberrypi/raspberrypi.cpp | 105 +++++++++++++-----<br>
> src/libcamera/sensor_mode.cpp | 60 ++++++++++<br>
> 8 files changed, 212 insertions(+), 27 deletions(-)<br>
> create mode 100644 include/libcamera/sensor_mode.h<br>
> create mode 100644 src/libcamera/sensor_mode.cpp<br>
><br>
> --<br>
> 2.20.1<br>
><br>
</blockquote></div></div>