[libcamera-devel] Docs about buffers

[Dorota Czaplejewicz]

Hi,

thanks to Jacopo, I read the pipeline handler guide. Most of it was great, confirming some of my reasoning about the code, but I got stuck there in the same place that gave me the most trouble in code: frame buffers.

I got the impression that docs about frame buffers are written for someone with a different background than I have. Perhaps it assumes some knowledge about V4L2 internals, whereas I'm just a lowly consumer of those APIs, and I haven't had to deal with media-device complexity with my driver (no ISP of any kind). As a result, I was unable to understand the vocabulary, and I didn't learn much from the docs that I couldn't guess already. Without false humility, I think that folks like me can be useful in writing pipeline handlers, so I wanted to give my perspective on what gave me trouble about frame buffers.

When I look at some code, I ask myself the crucial questions of "what is the intention?", "how does this work?", "why this way?", and I will focus on that.

I had some smaller difficulties with the FrameBuffer class itself. https://libcamera.org/api-html/classlibcamera_1_1FrameBuffer.html

While the description does quite a good job at explaining the intention and its contents, it's leaving parts of the other questions unanswered. For example, where is the actual data stored? Some other place mentions that the buffer is a DMABUF buffer, which presumably means that the data is referenced using Plane::fd. Since buffers are usually just plain in-memory arrays, this could also use a "why?" explanation (to allow zero-copy between x and y?). I know that there's a sentenct about that:

> The planes are specified when creating the FrameBuffer and are expressed as a set of dmabuf file descriptors, offset and length.

but I think the language is overly complicated, when it could say sth like "The frame buffer data is stored in the dmabuf file descriptors for each Plane object" to easily answer "where is the data?". (I'm saying that cause I missed it myself while tired after reading the guide.)

The description to access the data is placed in the docs for Plane, which makes it slightly harder to discover. Perhaps it would be beneficial to place all the docs regarding Plane and FrameBuffer in one place, since the two classes are basically two aspects of one thing. Having one description would make writing easier: no more questions "should I mention memory mapping here or reference the docs for Plane instead?".

A related thing that I think should get more visibility is to stress that the buffers don't necessarily live on the CPU. (Where do they live? Is FrameBuffer suitable to represent data flowing from the sensor to the MIPI controller? Can it live in any memory area accessible via DMA?)


Another problem is the FrameBuffer::cancel() method. It says:

> Marks the buffer as cancelled.
> If a buffer is not used by a request, it shall be marked as cancelled to indicate that the metadata is invalid.

I'm missing the intention/purpose of this. Why do I want to indicate the metadata is invalid? What can bring metadata to invalidity? I can imagine how it works (some flag), but I also don't know why it changes based on request. What if the buffer *is* used on request? Will the call get silently ignored?

It's also not clear if the FrameBuffer is focused on the memory allocation (mutable place to put data, reusable), or on the data itself (no need to be mutable, used once). I suspect the former, but it could be spelled out.

* * *

Now, on to the parts that I can't get through: allocateBuffers, importBuffers, exportBuffers, and releaseBuffers.

I know what "allocate" means, but when I try to read the description, I'm immediately confused:

> This function wraps buffer allocation with the V4L2 MMAP memory type. 

I'm not sure what "V4L2 memory types" are, somehow I managed to avoid coming across them. The next sentence is better:

> It requests count buffers from the driver, allocating the corresponding memory, and exports them as a set of FrameBuffer objects in buffers.

I'm not really sure about where the allocations take place, but I'm reading that as "this allocates `count` buffers on the device, and makes them accessible to the CPU (to DMA?) via FrameBuffer objects. The FrameBuffer objects are appended to the vector `buffers`".

Next is "export", which is like "allocate", but:

> Unlike allocateBuffers(), this function leaves the driver's internal buffer management uninitialized.

Together, the purpose/intention is missing: when should one be chosen and when the other? Why do driver internals matter? Maybe it's worth to link some external resource here.

About "import", I would have guessed that it works in reverse to export, that is: takes buffers on the CPU, and makes them accessible to the device. Then it would take a collection of buffers, but instead it takes... a number? I'm unable to make any sense out of this alone, not the intention, not how it works, not when to use it.

"release" is also somewhat confusing, in that it's not a dual to "allocate", in that it seems to release ~all buffers associated with the device. What happens if those buffers are used anyway?


I haven't found a more general guide to buffers: when to allocate them, when to reuse them, how to pass them between devices, who owns FrameBuffer objects, etc. I think having something like this would make first encounters go smoothly.

Overall, I don't want to come out as a complainer :) So in exchange for an explanation over email, I offer including what I learn in the docs.

Regards,
Dorota
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20211117/7a958dfa/attachment.sig>