[libcamera-devel] [PATCH-Resend] Add getting started guide for application developers

[Laurent Pinchart]

Hi Chris,

Thank you for your work.

On Mon, Jun 15, 2020 at 11:50:02AM +0100, Kieran Bingham wrote:
> From: Chris Chinchilla <chris at gregariousmammal.com>
> 
> ---
> 
> [Kieran:]
> Resending this inline to ease review on the libcamera mailing-list.

Thanks for that.

>  .../guides/libcamera-application-author.rst   | 472 ++++++++++++++++++
>  1 file changed, 472 insertions(+)
>  create mode 100644 Documentation/guides/libcamera-application-author.rst
> 
> diff --git a/Documentation/guides/libcamera-application-author.rst b/Documentation/guides/libcamera-application-author.rst
> new file mode 100644
> index 000000000000..c5f723820004
> --- /dev/null
> +++ b/Documentation/guides/libcamera-application-author.rst
> @@ -0,0 +1,472 @@
> +Supporting libcamera in your application

"Supporting" or "Using" ?

> +========================================
> +
> +This tutorial shows you how to create an application that uses libcamera

Maybe s/an application/a C++ application/ ?

> +to connect to a camera on a system, capture frames from it for 3
> +seconds, and write metadata about the frames to standard out.
> +
> +.. TODO: How much of the example code runs before camera start etc?
> +
> +Create a pointer to the camera
> +------------------------------
> +
> +Before the ``int main()`` function, create a global shared pointer
> +variable for the camera:
> +
> +.. code:: cpp
> +
> +   std::shared_ptr<Camera> camera;
> +
> +   int main()
> +   {
> +       // Code to follow
> +   }
> +

Trying to imagine how I would react as a newcomer to libcamera, I think
reading this as the very first section would let me wondering why. How
about the following ?

--------
Application skeleton
--------------------

Most of the code in this tutorial will run in the ``int main()``
function. A separate global function will be used to handle events. The
two functions will need to share data, which we will store in global
variables for simplicity. In a real application the various objects
created in the code below would be organized in classes, and the event
handler would be a class member function to provide context data without
requiring global variables.

.. code:: cpp

   // Global variables will go here

   int main()
   {
       // Code to follow
   }
--------

The shared pointer for the camera would then be added later, when the
guide introduces the Camera class.

> +Camera Manager
> +--------------
> +
> +Every libcamera-based application needs an instance of a
> +`CameraManager <http://libcamera.org/api-html/classlibcamera_1_1CameraManager.html>`_

We should try to automate linking to Doxygen-generated documentation.
I'm not sure yet how this could be done, So we can keep the links as-is
for now, it's a technical issue we need to solve on our side.

> +that runs for the life of the application. When you start the Camera
> +Manager, it finds all the cameras available to the current system.
> +Behind the scenes, the libcamera Pipeline Handler abstracts and manages
> +the complex pipelines that kernel drivers expose through the `Linux
> +Media
> +Controller <https://www.kernel.org/doc/html/latest/media/uapi/mediactl/media-controller-intro.html>`__

Do we need two underscores at the end of links ? I thought a single one
was enough.

> +and `V4L2 <https://www.linuxtv.org/docs.php>`__ APIs, meaning that an
> +application doesn’t need to handle device or driver specifics.

As Umang mentioned, I think we should not mention pipeline handlers
here, as they're internal to libcamera and should be invisible to
applications. How about s/the libcamera Pipeline Handler/libcamera/

Behind the scenes, libcamera abstracts and manages the complex pipelines
that kernel drivers expose through the `Linux Media Controller
<https://www.kernel.org/doc/html/latest/media/uapi/mediactl/media-controller-intro.html>`_
and `V4L2 <https://www.linuxtv.org/docs.php>`_ APIs, meaning that an
application doesn’t need to handle device or driver specifics.

> +
> +To create and start a new Camera Manager, create a new pointer variable
> +to the instance, and then start it:

Creating a new pointer variable covers the declaration of the variable,
but not the instantiation of the camera manager. I would write it as

"To create and start a new Camera Manager, create and instance of the
class, and then start it:"

That's a bit of a repeat though. Maybe the following would be better ?

"The first operations we need to perform is to create a Camera Manager
instance, and then start it:"

> +
> +.. code:: cpp
> +
> +   CameraManager *cm = new CameraManager();
> +   cm->start();
> +
> +When you build the application, the Camera Manager identifies all
> +supported devices and creates cameras the application can interact with.

s/build/run/ ? We don't create cameras when compiling the application
:-)

Or maybe

"When the Camera Manager is started, it identifies all supported devices
and creates cameras the application can interact with."

I would also mention that there should be a single instance of the
camera manager by a adding a sentence to the paragraph.

"An application should only create one Camera Manager instance."

> +
> +The code below identifies all available cameras, and for this example,

I would write "lists" instead of "identifies", as the identication is
mentioned above as a task performed by the camera manager when started.

> +writes them to standard output:

s/writes them/writes their name/ ?

> +
> +.. code:: cpp
> +
> +   for (auto const &camera : cm->cameras())
> +       std::cout << camera->name() << std::endl;
> +
> +For example, the output on Ubuntu running in a VM on macOS is
> +``FaceTime HD Camera (Built-in):``, and for x y, etc.
> +
> +.. TODO: Better examples

It's not a bad example, but it could be considered as a bit of a corner
case :-) How about the following ?

"For example, the output on a Linux machine with a connected USB webcam
is ``UVC Camera (046d:080a)``. When running Ubuntu in a VM on macOS, the
output is ``FaceTime HD Camera (Built-in):``."

We should provide examples for systems based on embedded cameras (I'm
thinking about Raspberry Pi in particular), but today it would just
print the sensor name. Maybe this should be revisited when we'll have
better camera names (that's a planned task) ?

> +
> +Create and acquire a camera
> +---------------------------
> +
> +What libcamera considers a camera
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The libcamera library supports fixed and hot-pluggable cameras,
> +including cameras, plugged and unplugged after initializing the library.

Is there an extra comma after cameras ?

s/initializing the library/starting the Camera Manager/ (as we don't
explain above what "initializing the library" would be.

> +The libcamera library supports point-and-shoot still image and video

Maybe "It supports ..." to avoid the repetition in two consecutive
sentences ? You can also write "libcamera" instead of "the libcamera
library" as an alternative form.

> +capture, either controlled directly by the CPU or exposed through an
> +internal USB bus as a UVC device designed for video conferencing usage.
> +The libcamera library considers any device that includes independent
> +camera sensors, such as front and back sensors as multiple different

missing comma after "back sensors" ?

> +camera devices.
> +
> +Once you know what camera you want to use, your application needs to
> +acquire a lock to it so no other application can use it.

Maybe s/a lock/exclusive access/ ?

I'd split this in camera selection and locking. This paragraph could be
moved to [1].

> +
> +This example application uses a single camera that the Camera Manager

Should this be s/a single camera/the first camera/ ?

> +reports as available to applications.
> +
> +The code below creates the name of the first available camera as a
> +convenience variable, fetches that camera, and acquires the device for
> +exclusive access:

I think a word about the fact that we support getting cameras by index
and by name would be useful. Maybe along the lines of

"libcamera supports accessing cameras by index or by name. To
demonstrate both, the code below retrieves the name of the first
available camera, and then gets the camera by name from the Camera
Manager."

> +
> +.. code:: cpp
> +
> +   std::string cameraName = cm->cameras()[0]->name();
> +   camera = cm->get(cameraName);

[1] here, with an additional

.. code:: cpp

> +   camera->acquire();

to split the code in two parts.

> +
> +Configure the camera
> +--------------------
> +
> +Before the application can do anything with the camera, you need to know
> +what it’s capabilities are. These capabilities include scalars,

s/it's/its/

What does "scalars" refer to here ?

> +resolutions, supported formats and converters. The libcamera library

I'd write "pixel formats" instead of "formats", thats a term widely used
for cameras on Linux systems (it should be added to a glossary though).

I'm also not sure what "converters" refer to here.

> +uses ``StreamRole``\ s to define four predefined ways an application
> +intends to use a camera (`You can read the full list in the API
> +documentation <http://libcamera.org/api-html/stream_8h.html#a295d1f5e7828d95c0b0aabc0a8baac03>`__).

Before mentioning stream roles, I think we need to explain streams.

It's getting a bit late so I'm sending this already, I will continue
reviewing the rest ASAP.

> +
> +To find out if how your application wants to use the camera is possible,
> +generate a new configuration using a vector of ``StreamRole``\ s, and
> +send that vector to the camera. To do this, create a new configuration
> +variable and use the ``generateConfiguration`` function to produce a
> +``CameraConfiguration`` for it. If the camera can handle the
> +configuration it returns a full ``CameraConfiguration``, and if it
> +can’t, a null pointer.
> +
> +.. code:: cpp
> +
> +   std::unique_ptr<CameraConfiguration> config = camera->generateConfiguration( { StreamRole::Viewfinder } );
> +
> +A ``CameraConfiguration`` has a ``StreamConfiguration`` instance for
> +each ``StreamRole`` the application requested, and that the camera can
> +support. Each of these has a default size and format that the camera
> +assigned, depending on the ``StreamRole`` requested.
> +
> +The code below creates a new ``StreamConfiguration`` variable and
> +populates it with the value of the first (and only) ``StreamRole`` in
> +the camera configuration. It then outputs the value to standard out.
> +
> +.. code:: cpp
> +
> +   StreamConfiguration &streamConfig = config->at(0);
> +   std::cout << "Default viewfinder configuration is: " << streamConfig.toString() << std::endl;
> +
> +Change and validate the configuration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Once you have a ``StreamConfiguration``, your application can make
> +changes to the parameters it contains, for example, to change the width
> +and height you could use the following code:
> +
> +.. code:: cpp
> +
> +   streamConfig.size.width = 640;
> +   streamConfig.size.height = 480;
> +
> +If your application makes changes to any parameters, validate them
> +before applying them to the camera by using the ``validate`` function.
> +If the new value is invalid, the validation process adjusts the
> +parameter to what it considers a valid value. An application should
> +check that the adjusted configuration is something you expect (you can
> +use the
> +`Status <http://libcamera.org/api-html/classlibcamera_1_1CameraConfiguration.html#a64163f21db2fe1ce0a6af5a6f6847744>`_
> +method to check if the Pipeline Handler adjusted the configuration).
> +
> +For example, above you set the width and height to 640x480, but if the
> +camera cannot produce an image that large, it might return the
> +configuration with a new size of 320x240 and a status of ``Adjusted``.
> +
> +For this example application, the code below prints the adjusted values
> +to standard out.
> +
> +.. code:: cpp
> +
> +   config->validate();
> +   std::cout << "Validated viewfinder configuration is: " << streamConfig.toString() << std::endl;
> +
> +With a validated ``CameraConfiguration``, send it to the camera to
> +confirm the new configuration:
> +
> +.. code:: cpp
> +
> +   camera->configure(config.get());
> +
> +If you don’t first validate the configuration before calling
> +``configure``, there’s a chance that calling the function fails.
> +
> +Allocate FrameBuffers
> +---------------------
> +
> +The libcamera library consumes buffers provided by applications as
> +``FrameBuffer`` instances, which makes libcamera a consumer of buffers
> +exported by other devices (such as displays or video encoders), or
> +allocated from an external allocator (such as ION on Android).
> +
> +The libcamera library uses ``FrameBuffer`` instances to buffer frames of
> +data from memory, but first, your application should reserve enough
> +memory for the ``FrameBuffer``\ s your streams need based on the sizes
> +and formats you configured.
> +
> +In some situations, applications do not have any means to allocate or
> +get hold of suitable buffers, for instance, when no other device is
> +involved, or on Linux platforms that lack a centralized allocator. The
> +``FrameBufferAllocator`` class provides a buffer allocator that you can
> +use in these situations.
> +
> +An application doesn’t have to use the default ``FrameBufferAllocator``
> +that libcamera provides, and can instead allocate memory manually, and
> +pass the buffers in ``Request``\ s (read more about ``Request``\ s in
> +`the frame capture section <#frame-capture>`__ of this guide). The
> +example in this guide covers using the ``FrameBufferAllocator`` that
> +libcamera provides.
> +
> +Using the libcamera ``FrameBufferAllocator``
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +As the camera manager knows what configuration is available, it can
> +allocate all the resources for you with a single method, and de-allocate
> +with another.
> +
> +Applications create a ``FrameBufferAllocator`` for a Camera, and use it
> +to allocate buffers for streams of a ``CameraConfiguration`` with the
> +``allocate()`` function.
> +
> +.. code:: cpp
> +
> +   for (StreamConfiguration &cfg : *config) {
> +       int ret = allocator->allocate(cfg.stream());
> +       if (ret < 0) {
> +           std::cerr << "Can't allocate buffers" << std::endl;
> +           return -ENOMEM;
> +       }
> +
> +       unsigned int allocated = allocator->buffers(cfg.stream()).size();
> +       std::cout << "Allocated " << allocated << " buffers for stream" << std::endl;
> +   }
> +
> +Frame Capture
> +~~~~~~~~~~~~~
> +
> +The libcamera library follows a familiar streaming request model for
> +data (frames in this case). For each frame a camera captures, your
> +application must queue a request for it to the camera.
> +
> +In the case of libcamera, a ‘Request’ is at least one Stream (one source
> +from a Camera), with a FrameBuffer full of image data.
> +
> +First, create an instance of a ``Stream`` from the ``StreamConfig``,
> +assign a vector of ``FrameBuffer``\ s to the allocation created above,
> +and create a vector of the requests the application will make.
> +
> +.. code:: cpp
> +
> +   Stream *stream = streamConfig.stream();
> +   const std::vector<std::unique_ptr<FrameBuffer>> &buffers = allocator->buffers(stream);
> +   std::vector<Request *> requests;
> +
> +Create ``Request``\ s for the size of the ``FrameBuffer`` by using the
> +``createRequest`` function and adding all the requests created to a
> +vector. For each request add a buffer to it with the ``addBuffer``
> +function, passing the stream the buffer belongs to, and the ``FrameBuffer``.
> +
> +.. code:: cpp
> +
> +       for (unsigned int i = 0; i < buffers.size(); ++i) {
> +           Request *request = camera->createRequest();
> +           if (!request)
> +           {
> +               std::cerr << "Can't create request" << std::endl;
> +               return -ENOMEM;
> +           }
> +
> +           const std::unique_ptr<FrameBuffer> &buffer = buffers[i];
> +           int ret = request->addBuffer(stream, buffer.get());
> +           if (ret < 0)
> +           {
> +               std::cerr << "Can't set buffer for request"
> +                     << std::endl;
> +               return ret;
> +           }
> +
> +           requests.push_back(request);
> +
> +           /*
> +            * todo: Set controls
> +            *
> +            * ControlList &Request::controls();
> +            * controls.set(controls::Brightness, 255);
> +            */
> +       }
> +
> +.. TODO: Controls
> +.. TODO: A request can also have controls or parameters that you can apply to the image. -->
> +
> +Event handling and callbacks
> +----------------------------
> +
> +The libcamera library uses the concept of signals and slots (`similar to
> +Qt <https://doc.qt.io/qt-5/signalsandslots.html>`__) to connect events
> +with callbacks to handle those events.
> +
> +Signals
> +~~~~~~~
> +
> +Signals are emitted when the buffer has been completed (image data
> +written into), and because a Request can contain multiple buffers - the
> +Request completed signal is emitted when all buffers within the request
> +are completed.
> +
> +A camera class instance emits a completed request signal to report when
> +all the buffers in a request are complete with image data written to
> +them. To receive these signals, connect a slot function to the signal
> +you are interested in. For this example application, that’s when the
> +camera completes a request.
> +
> +.. code:: cpp
> +
> +   camera->requestCompleted.connect(requestComplete);
> +
> +Slots
> +~~~~~
> +
> +Every time the camera request completes, it emits a signal, and the
> +connected slot invoked, passing the Request as a parameter.
> +
> +For this example application, the ``requestComplete`` slot outputs
> +information about the ``FrameBuffer`` to standard out, but the callback is
> +typically where your application accesses the image data from the camera
> +and does something with it.
> +
> +Signals operate in the libcamera ``CameraManager`` thread context, so it
> +is important not to block the thread for a long time, as this blocks
> +internal processing of the camera pipelines, and can affect realtime
> +performances, leading to skipped frames etc.
> +
> +First, create the function that matches the slot:
> +
> +.. code:: cpp
> +
> +   static void requestComplete(Request *request)
> +   {
> +       // Code to follow
> +   }
> +
> +The signal/slot flow is the only way to pass requests and buffers from
> +libcamera back to the application. There are times when a request can
> +emit a ``requestComplete`` signal, but this request is actually
> +cancelled, for example by application shutdown. To avoid an application
> +processing image data that doesn’t exist, it’s worth checking that the
> +request is still in the state you expect (You can find `a full list of
> +the completion statuses in the
> +documentation <https://www.libcamera.org/api-html/classlibcamera_1_1Request.html#a2209ba8d51af8167b25f6e3e94d5c45b>`__).
> +
> +.. code:: cpp
> +
> +   if (request->status() == Request::RequestCancelled) return;
> +
> +When the request completes, you can access the buffers from the request
> +using the ``buffers()`` function which returns a map of each buffer, and
> +the stream it is associated with.
> +
> +.. code:: cpp
> +
> +   const std::map<Stream *, FrameBuffer *> &buffers = request->buffers();
> +
> +Iterating through the map allows you to inspect each buffer from each
> +stream completed in this request, and access the metadata for each frame
> +the camera captured. The buffer metadata contains information such as
> +capture status, a timestamp and the bytes used.
> +
> +.. code:: cpp
> +
> +   for (auto bufferPair : buffers) {
> +       FrameBuffer *buffer = bufferPair.second;
> +       const FrameMetadata &metadata = buffer->metadata();
> +   }
> +
> +The buffer describes the image data, but a buffer can consist of more
> +than one image plane that in memory hold the image data in the Frames.
> +For example, the Y, U, and V components of a YUV-encoded image are
> +described by a plane.
> +
> +For this example application, still inside the ``for`` loop from above,
> +print the Frame sequence number and details of the planes.
> +
> +.. code:: cpp
> +
> +   std::cout << " seq: " << std::setw(6) << std::setfill('0') << metadata.sequence << " bytesused: ";
> +
> +   unsigned int nplane = 0;
> +   for (const FrameMetadata::Plane &plane : metadata.planes)
> +   {
> +       std::cout << plane.bytesused;
> +       if (++nplane < metadata.planes.size()) std::cout << "/";
> +   }
> +
> +   std::cout << std::endl;
> +
> +With the handling of this request complete, reuse the buffer by adding
> +it back to the request with its matching stream, and create a new
> +request using the ``createRequest`` function.
> +
> +.. code:: cpp
> +
> +       request = camera->createRequest();
> +       if (!request)
> +       {
> +           std::cerr << "Can't create request" << std::endl;
> +           return;
> +       }
> +
> +       for (auto it = buffers.begin(); it != buffers.end(); ++it)
> +       {
> +           Stream *stream = it->first;
> +           FrameBuffer *buffer = it->second;
> +
> +           request->addBuffer(stream, buffer);
> +       }
> +
> +       camera->queueRequest(request);
> +
> +Start the camera and event loop
> +-------------------------------
> +
> +If you build and run the application at this point, none of the code in
> +the slot method above runs. While most of the code to handle processing
> +camera data is in place, you need first to start the camera to begin
> +capturing frames and queuing requests to it.
> +
> +.. code:: cpp
> +
> +   camera->start();
> +   for (Request *request : requests)
> +       camera->queueRequest(request);
> +
> +To emit signals that slots can respond to, your application needs an
> +event loop. You can use the ``EventDispatcher`` class as an event loop
> +for your application to listen to signals from resources libcamera
> +handles.
> +
> +The libcamera library does this by creating instances of the
> +``EventNotifier`` class, which models a file descriptor event source an
> +application can monitor and registers them with the ``EventDispatcher``.
> +Whenever the ``EventDispatcher`` detects an event it is monitoring, and
> +it emits an ``EventNotifier::activated signal``. The ``Timer`` class to
> +control the length event loops run for that you can register with a
> +dispatcher with the ``registerTimer`` function.
> +
> +The code below creates a new instance of the ``EventDispatcher`` class,
> +adds it to the camera manager, creates a timer to run for 3 seconds, and
> +during the length of that timer, the ``EventDispatcher`` processes
> +events that occur, and calls the relevant signals.
> +
> +.. code:: cpp
> +
> +   EventDispatcher *dispatcher = cm->eventDispatcher();
> +   Timer timer;
> +   timer.start(3000);
> +   while (timer.isRunning())
> +       dispatcher->processEvents();
> +
> +Clean up and stop application
> +-----------------------------
> +
> +The application is now finished with the camera and the resources the
> +camera uses, so you need to do the following:
> +
> +-  stop the camera
> +-  free the stream from the ``FrameBufferAllocator``
> +-  delete the ``FrameBufferAllocator``
> +-  release the lock on the camera and reset the pointer to it
> +-  stop the camera manager
> +-  exit the application
> +
> +.. code:: cpp
> +
> +   camera->stop();
> +   allocator->free(stream);
> +   delete allocator;
> +   camera->release();
> +   camera.reset();
> +   cm->stop();
> +
> +   return 0;
> +
> +Conclusion
> +----------

-- 
Regards,

Laurent Pinchart