[libcamera-devel] [PATCH 03/11] Adds rkisp1 metadata required for the Android HAL to use it.

[Jacopo Mondi]

Hi Nicholas,

On Tue, Oct 25, 2022 at 11:40:11PM -0500, Nicholas Roth wrote:
> +Kieran Bingham <kieran.bingham at ideasonboard.com> since Laurent is out.
>
> > Could this be computed the same way we do it in the IPU3 IPA module ?
>
> It looks like each of the computed IPU3 FrameDurationLimit values
> correspond to the minimum and maximum v4l2 blanking intervals. I suppose
> that makes sense, though I suspect the minimum frame duration will be
> constrained by i2c bandwidth and ISP limitations at the highest resolutions
> rather than the sensor blanking intervals based on the datasheet.
>

Mostly for sake of discussion, as you correctly noticed we report as
the camera frame duration the sensor's frame duration. I think this is
correct, as there might indeed be other bottlenecks in the capture
pipeline, but I guess it's hard to quantify them in terms of latency
and they depend on the current camera configuration, ie the number
and resolution of the streams that are processed in a given capture session.

I wonder if all platforms behaves the same when overloaded, some
might introduce latencies, some might drop buffers ? I'm also afraid
not all the numbers we would need to quantify something are actually
available...

The sensor frame duration is shared by all streams generated from the same
and can be reported as a camera global parameter.

I2c bandwidth doesn't seem a concern to me, during streaming i2c
traffic should be negligible and limited to the control of a few
sensor's parameters which are controlled at run-time, such as the
exposure time and gain.

There might of course be multiple processing steps to take into
account when actually computing the full duration of the frame since
the time it has been produced by the sensor to the time it is
delivered to applications.  Android in example has a dedicated
property for 'stalling' streams that goes through, in example, JPEG
encoding, something that is not exactly in scope for libcamera (and
which we don't fully handle in the HAL yet).

> I'm not really sure what the best thing to do here is. I'm concerned that
> if we follow the IPU3 computation, we could easily end up asking for the
> ISP to push more pixels than it has throughput for. A very conservative

I don't have numbers here and I presume they're very platform
dependent, but I don't think we actually hit that limit on IPU3 which
is where the HAL has been mostly tested on. Which platform are you
working with ?

> approach might be to cap this at a few FPS, which is the advertised rate at
> the highest resolution, but I would not enjoy using the camera that way.
> Curious to read others' thoughts on this.
>
> As a side-note, I'm really not sure why the IPU3 IPA treats
> FrameDurationLimits as a 3-tuple, when control_ids.yaml explicitly states:
>       description: |
>         The minimum and maximum (in that order) frame duration,
>         expressed in microseconds.
> ...
>       size: [2]

What you see initialized with 3 values is the ControlInfo, which
reports the max, min and default values for a control. The control
itself accepts two values as indicated by the documentation.

>
> It looks like updateControls is only called once, during init(). We would
> need a hook that changes this static configuration for each video mode.

I do see updateControls() be called by configure() as well, and I see
the pipeline handler correctly updating the ControlInfoMap of camera
controls. Am I missing something ?
>
>
> On Tue, Oct 25, 2022 at 10:16 AM Nicholas Roth <nicholas at rothemail.net>
> wrote:
>
> > > Careful, we're claiming full per-frame control capabilities with this
> > set to 0 :)
> >
> > Good catch— I’ll revert this line. I couldn’t easily find this in the ISP
> > manual and wanted to see what we’d need for FULL instead of LIMITED support
> > while I was debugging a camera2 crash.
> >
> > > On Oct 25, 2022, at 8:20 AM, Jacopo Mondi <jacopo at jmondi.org> wrote:
> > >
> > > Careful, we're claiming full per-frame control capabilities with this
> > > set to 0 :)
> >