[libcamera-devel] Expected Request queuing behavior by apps
Jacopo Mondi
jacopo at jmondi.org
Fri Dec 16 10:05:51 CET 2022
Hi Umang
thanks for starting the thread.
On Fri, Dec 16, 2022 at 11:30:39AM +0530, Umang Jain via libcamera-devel wrote:
> Hi everyone,
>
> Goal:
> I am kicking-off a thread to discuss the Request queuing behaviour that can
> be expected by libcamera from application's side. In the recent
> discussions, we have found that we don't yet advertise or document this
> aspect explicitly. Hence, it would be helpful to do so, especially it helps
> with internal plumbing of requests routing through pipeline-handlers(s), IPA
> and various requests holding placeholders for state tracking (queues,
> arrays, FCQueue etc.) or even post-processing.
>
> Assumptions up till this point:
> Atleast what _I_ have assumed up until this point is that applications queue
> a certain number of requests (let's say 4) and then continuing queuing the
> request from the `requestCompleted` callback handler (slot). I see this
> behaviour implemented in our `libcamera/src/apps`. This assumption
> inherently suggests that these applications, are not trying to over-queue to
> the hardware hence, respecting a certain depth of the hardware pipeline.
> They makes sure new requests get queued when previous ones gets completed.
> Other argument can be that, reuse of Request objects are cheaper than
> constructing a new one for each queueRequest() and queuing all at once.
>
> Defining the explicit behaviour by applications:
> Note, in the last segment - "Assumption". We do not explicitly say that the
> applications should queue the requests in a certain manner.
>
I agree this is the canonical way, but it's certainly not the only one
> Hence, it is time to defining our expectations on what applications are
> allowed or not allowed to do, with respect of queuing requests. Below are a
> few documentation headers to move the discussion forwards.
>
> - Exposing the hardware capabilities
> Should libcamera expose the underlying hardware capabilities/constraints to
> applications and expect them to rate-limit the queuing of requests based?
>
Let's start by clearly enumerating these, to see how we handle them
today.
I would start by listing the requirements of the current libcamera
backend (V4L2) and see how these have surfaced to the current API and
what we might want to change.
1) Startup:
V4L2 video devices have a requirement of having a min number of buffers
queued before streaming is started. The requirement comes from the
need of having enough memory buffer available to sustain memory
transfer and data capture operations.
The number of min buffers queued is a platform-specific parameter
The only documentation I found is in the vb2 framework documentation
https://www.kernel.org/doc/html/latest/driver-api/media/v4l2-videobuf2.html?highlight=min_buffers_needed
libcamera currently queues buffers to the capture video devices
at queueRequest time only, hence applications need to queue enough
requests before streaming is actually started. The Camera state
machine demands that Camera::start() is called before any
Camera::queueRequest() can be called, but if applications do not
queue enough requests, streaming will never be started.
We currently do not have a clear way to express that, if not by
validating StreamConfiguration::bufferCount which is set by
pipelines to a known valid values for the platform, but nothing in
our API tells applications "you have to queue enough requests to
have the camera actually started" as far as I can see ?
2) Runtime
Following the principle of "1 request - 1 buffer", which I guess we
aim to change in future, you can't queue more requests to a camera
than the number of buffers requested on a video device.
The number of requests buffers is again usually defined by
StreamConfiguration::buffersCount and it usually also dictates the
number of intermediate buffers the pipeline handler allocates
(think of the CIO2 buffers, or the ISP parameter buffers as
examples).
Please note this is an issue for application that allocates buffers
from elsewhere (ie a dmabuf-heap pool or on the display) and import
them in libcamera. If applications allocates buffers in the Camera
they will be limited to work with those buffers, and by definition,
they can't use more than what they have allocated.
It's interesting to notice how some pipeline had to take of that by
themselves
https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/pipeline/ipu3/ipu3.cpp#n853
89dae5844964 ("libcamera: pipeline: ipu3: Store requests in the case a buffer shortage")
Your work on throttling the number of requests in the base
PipelineHandler class according to the depth of the frame contexts
queue could be re-considered in this new context: up to bufferCount
requests can be queued -to the device- at the same time and the
PipelineHandler base class caches them on behalf of the pipeline
handlers. This will make StreamConfiguration::bufferCount a central
part of our API, and in that case the implications of chosing a
non-accurate bufferCount should be clearly documented.
We can also say we expect applications to deal with that by
themselves, but that's not realistic at the moment, more over in
the perspective that libcamera will increasingly be used by other
frameworks and adaptation layers, and all of them should
re-implement the same safety mechanism.
3) Stop
That's where we offer gurantees: requests will all be completed,
and will be completed in order. We don't expect applications to do
much here.
Do you agree with my understaning on these parts ? Are there other
aspects I have not considered ?
All in all, I feel like we have one big topic here: we had the V4L2
buffer handling requirments surfaced to our API, and it has surfaced
not much in the API signature, but rather in how the library behaves
with too many implicit behaviour and an assumption of a certain
understanding of V4L2 from application developers.
> - No limit
> Advertise explicitly that there are no constraints in queuing Request to
> libcamera *whatsoever*, from the applications side?
>
> - Number of requests to be queued by the application, to get X number of
> frames
>
> X = 1 Frame
> X <= frames less than processing blocks on a platform
> X > frames less than processing blocks on a platform
I'm not sure I got this part :)
>
> Depending on the request queuing behaviour libcamera expects, I see that
> documenting the aforementioned might be useful for applications developers
> as well (maybe this goes in per-platform information?). lc-compliance can
> incorporate checks for these as well.
>
> These are just on top off my head, new discussions points are welcomed as
> well.
Let's start by clarifying how things work today and what implicit
requirements we have today and idealy make a plan if our API should
stabilize on the '1 request - 1 buffer' model.
Thanks
j
>
> Thanks,
> Umang
More information about the libcamera-devel
mailing list