[libcamera-devel] [PATCH 2/9] libcamera: delayed_controls: Add helper for controls that applies with a delay
Naushir Patuck
naush at raspberrypi.com
Tue Nov 10 16:45:45 CET 2020
Hi Niklas,
On Mon, 9 Nov 2020 at 22:54, Niklas Söderlund <niklas.soderlund at ragnatech.se>
wrote:
> Hi Naush,
>
> Thanks for your feedback.
>
> On 2020-11-03 10:27:06 +0000, Naushir Patuck wrote:
> > Hi Niklas,
> >
> > Thank you for your patch. Please see some initial comments below.
> >
> > Best regards,
> > Naush
> >
> >
> > On Wed, 28 Oct 2020 at 01:01, Niklas Söderlund <
> > niklas.soderlund at ragnatech.se> wrote:
> >
> > > Some sensor controls take effect with a delay as the sensor needs time
> > > to adjust, for example exposure. Add a optional helper DelayedControls
> > > to help pipelines deal with such controls.
> > >
> > > The idea is to provide a queue of controls towards the V4L2 device and
> > > apply individual controls with the specified delay with the aim to get
> > > predictable and retrievable control values for any given frame. To do
> > > this the queue of controls needs to be at least as deep as the control
> > > with the largest delay.
> > >
> > > The DelayedControls needs to be informed of every start of exposure.
> > > This can be emulated but the helper is designed to be used with this
> > > event being provide by the kernel thru V4L2 events.
> > >
> > > This helper is based on StaggeredCtrl from the Raspberry Pi pipeline
> > > handler but expands on its API. This helpers aims to replace the
> > > Raspberry Pi implementations and mimics it behavior perfectly.
> > >
> > > Signed-off-by: Niklas Söderlund <niklas.soderlund at ragnatech.se>
> > > ---
> > > include/libcamera/internal/delayed_controls.h | 87 ++++++
> > > src/libcamera/delayed_controls.cpp | 282 ++++++++++++++++++
> > > src/libcamera/meson.build | 1 +
> > > 3 files changed, 370 insertions(+)
> > > create mode 100644 include/libcamera/internal/delayed_controls.h
> > > create mode 100644 src/libcamera/delayed_controls.cpp
> > >
> > > diff --git a/include/libcamera/internal/delayed_controls.h
> > > b/include/libcamera/internal/delayed_controls.h
> > > new file mode 100644
> > > index 0000000000000000..df5520d240a54e4b
> > > --- /dev/null
> > > +++ b/include/libcamera/internal/delayed_controls.h
> > > @@ -0,0 +1,87 @@
> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> > > +/*
> > > + * Copyright (C) 2020, Google Inc.
> > > + *
> > > + * delayed_controls.h - Helper to deal with controls that are applied
> > > with a delay
> > > + */
> > >
> >
> > Given this is heavily derived from the staggered_write work, would it be
> ok
> > to keep the Raspberry Pi copyright in the header?
> >
> > Copyright (C) 2020, Raspberry Pi (Trading) Ltd.
>
> As discussed of-band this was always my intention, my templates got in
> the way. Sorry about that and thanks for pointing it out.
>
> >
> >
> > > +#ifndef __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__
> > > +#define __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__
> > > +
> > > +#include <mutex>
> > > +#include <stdint.h>
> > > +#include <unordered_map>
> > > +
> > > +#include <libcamera/controls.h>
> > > +
> > > +namespace libcamera {
> > > +
> > > +class V4L2Device;
> > > +
> > > +class DelayedControls
> > > +{
> > > +public:
> > > + DelayedControls(V4L2Device *device,
> > > + const std::unordered_map<uint32_t, unsigned
> int>
> > > &delays);
> > > +
> > > + void reset(ControlList *controls = nullptr);
> > > +
> > > + bool push(const ControlList &controls);
> > > + ControlList get(uint32_t sequence);
> > > +
> > > + void frameStart(uint32_t sequence);
> > > +
> > > +private:
> > > + class ControlInfo
> > > + {
> > > + public:
> > > + ControlInfo()
> > > + : updated(false)
> > > + {
> > > + }
> > > +
> > > + ControlInfo(const ControlValue &v)
> > > + : value(v), updated(true)
> > > + {
> > > + }
> > > +
> > > + ControlValue value;
> > > + bool updated;
> > > + };
> > > +
> > > + static constexpr int listSize = 16;
> > > + class ControlArray : public std::array<ControlInfo, listSize>
> > > + {
> > > + public:
> > > + ControlInfo &operator[](unsigned int index)
> > > + {
> > > + return std::array<ControlInfo,
> > > listSize>::operator[](index % listSize);
> > > + }
> > > +
> > > + const ControlInfo &operator[](unsigned int index) const
> > > + {
> > > + return std::array<ControlInfo,
> > > listSize>::operator[](index % listSize);
> > > + }
> > > + };
> > > +
> > > + using ControlsDelays = std::unordered_map<const ControlId *,
> > > unsigned int>;
> > > + using ControlsValues = std::unordered_map<const ControlId *,
> > > ControlArray>;
> > > +
> > > + bool queue(const ControlList &controls);
> > > +
> > > + std::mutex lock_;
> > > +
> > > + V4L2Device *device_;
> > > + ControlsDelays delays_;
> > > + unsigned int maxDelay_;
> > > +
> > > + bool running_;
> > > + uint32_t fistSequence_;
> > > +
> > > + uint32_t queueCount_;
> > > + uint32_t writeCount_;
> > > + ControlsValues ctrls_;
> > > +};
> > > +
> > > +} /* namespace libcamera */
> > > +
> > > +#endif /* __LIBCAMERA_INTERNAL_DELAYED_CONTROLS_H__ */
> > > diff --git a/src/libcamera/delayed_controls.cpp
> > > b/src/libcamera/delayed_controls.cpp
> > > new file mode 100644
> > > index 0000000000000000..0e32f417c5cc68b7
> > > --- /dev/null
> > > +++ b/src/libcamera/delayed_controls.cpp
> > > @@ -0,0 +1,282 @@
> > > +/* SPDX-License-Identifier: LGPL-2.1-or-later */
> > > +/*
> > > + * Copyright (C) 2020, Google Inc.
> > > + *
> > > + * delayed_controls.h - Helper to deal with controls that are applied
> > > with a delay
> > > + */
> > >
> >
> > Similar to above:
> > Copyright (C) 2020, Raspberry Pi (Trading) Ltd.
> >
> >
> > > +
> > > +#include "libcamera/internal/delayed_controls.h"
> > > +#include "libcamera/internal/v4l2_device.h"
> > > +
> > > +#include <libcamera/controls.h>
> > > +
> > > +#include "libcamera/internal/log.h"
> > > +
> > > +/**
> > > + * \file delayed_controls.h
> > > + * \brief Helper to deal with controls that are applied with a delay
> > > + */
> > > +
> > > +namespace libcamera {
> > > +
> > > +LOG_DEFINE_CATEGORY(DelayedControls)
> > > +
> > > +/**
> > > + * \class DelayedControls
> > > + * \brief Helper to deal with controls that takes effect with a delay
> > > + *
> > > + * Some sensor controls take effect with a delay as the sensor needs
> time
> > > to
> > > + * adjust, for example exposure and focus. This is an optional helper
> > > class to
> > > + * deal with such controls and the intended user is pipeline handlers.
> > > + *
> > > + * The idea is to extend the concept of the pipeline depth the users
> > > needs to
> > > + * maintain for buffers to controls. The depth is determined with by
> the
> > > control
> > > + * with the grates delay. As long as the pipeline keeps the control
> queue
> > > above
> > > + * this level the control values are guaranteed to be in effect at the
> > > specified
> > > + * point in time.
> > > + *
> > > + * The downside is of course that the pipeline needs to know what
> > > controls to
> > > + * set control depth frames in advance. But there really is no way
> around
> > > this
> > > + * as the delay is a consequence of the physical world. Compare this
> with
> > > + * parameter buffers where the buffer may be queued to the device
> while
> > > it's
> > > + * still being written to as long as it's ready before it's consumed,
> > > this is
> > > + * because the parameter buffer (usually) does not contain controls
> that
> > > + * requires time to take effect.
> > > + */
> > > +
> > > +
> > > +/**
> > > + * \brief Construct a DelayedControls
> > > + * \param[in] device The V4L2 device containing the delayed controls
> > > + * \param[in] delays Map of numerical V4L2 control id to its delay to
> take
> > > + * effect in frames
> > > + *
> > > + * Only controls specified in \a delays are handled by the
> DelayedControls
> > > + * instance. If it's desired to mix delayed controls and controls that
> > > takes
> > > + * effect immediately the immediate controls must be listed in the \a
> > > delays map
> > > + * with a delay value of 0.
> > > + */
> > > +DelayedControls::DelayedControls(V4L2Device *device,
> > > + const std::unordered_map<uint32_t,
> > > unsigned int> &delays)
> > > + : device_(device), maxDelay_(0)
> > > +{
> > > + const ControlInfoMap &controls = device_->controls();
> > > +
> > > + /*
> > > + * Sanity check that all controls where delays are requested
> are
> > > + * exposed byt the device.
> > > + */
> > > + for (auto const &delay : delays) {
> > > + unsigned int id = delay.first;
> > > +
> > > + if (controls.find(id) == controls.end())
> > > + LOG(DelayedControls, Error)
> > > + << "Delay request for control id "
> > > + << utils::hex(id)
> > > + << " but control is not exposed by
> device "
> > > + << device_->deviceNode();
> > > + }
> > > +
> > > + /*
> > > + * Create a map of control to delay for all controls exposed
> by the
> > > + * device. If no delay is specified assume the control applies
> > > directly.
> > > + */
> > > + for (auto const &control : controls) {
> > > + const ControlId *id = control.first;
> > > +
> > > + auto it = delays.find(id->id());
> > > + if (it == delays.end())
> > > + continue;
> > > +
> > > + delays_[id] = it->second;
> > > +
> > > + LOG(DelayedControls, Debug)
> > > + << "Set a delay of " << delays_[id]
> > > + << " for " << id->name();
> > > +
> > > + maxDelay_ = std::max(maxDelay_, delays_[id]);
> > > + }
> > >
> >
> > Could you combine the two loops here into one here? If you iterate over
> > "delays", you could do the error check as well as adding to the "delays_"
> > map.
>
> Good call!
>
> >
> > +
> > > + reset();
> > >
> >
> > Does this need to be called here? But my understanding of reset() may be
> > wrong... see below.
>
> It does as reset() initialize the internal state machine.
>
> >
> >
> > > +}
> > > +
> > > +/**
> > > + * \brief Reset the V4L2 device
> > > + * \param[in] controls List of controls to reset the device to or
> nullptr
> > > + *
> > > + * Resets the delayed controls state machine to its starting state.
> All
> > > controls
> > > + * are fetched from the V4L2 device to provide a good starting point
> for
> > > the
> > > + * first frames (length of control depth).
> > > + *
> > > + * Optionally \a controls can be specified to set some or all of the
> > > handled
> > > + * V4L2 controls prior to reading them back. If no controls needs to
> be
> > > set
> > > + * nullptr may be used.
> > > + */
> > > +void DelayedControls::reset(ControlList *controls)
> > > +{
> > > + std::lock_guard<std::mutex> lock(lock_);
> > > +
> > > + running_ = false;
> > > + fistSequence_ = 0;
> > > + queueCount_ = 0;
> > > + writeCount_ = 0;
> > > +
> > > + /* Set the controls on the device if requested. */
> > > + if (controls)
> > > + device_->setControls(controls);
> > >
> >
> > Should this happen here in reset()? The reset method to me indicates
> that
> > some internal state of the class is meant to be reset, and it does
> nothing
> > to send controls to the device. Writing to the device could happen from
> > the caller, if they desire.
>
> Maybe this should be renamed to init() and the argument to
> initialControls? As this function should only be called when we are not
> streaming I see no harm in optionally allowing a init value. But maybe
> this goes hand in had with your follow up comment below.
>
> >
> >
> > > +
> > > + /* Retrieve current control values reported by the device. */
> > > + std::vector<uint32_t> ids;
> > > + for (auto const &delay : delays_)
> > > + ids.push_back(delay.first->id());
> > > +
> > > + ControlList devCtrls = device_->getControls(ids);
> > > +
> > > + /* Seed the control queue with the controls reported by the
> > > device. */
> > > + ctrls_.clear();
> > > + for (const auto &ctrl : devCtrls) {
> > > + const ControlId *id =
> > > devCtrls.infoMap()->idmap().at(ctrl.first);
> > > + ctrls_[id][queueCount_] = ControlInfo(ctrl.second);
> > > + }
> > > +
> > > + queueCount_++;
> > > +}
> > >
> >
> > I'm a bit unclear what reset() does here. In staggered_writer, reset()
> > would essentially clear the queues and re-initialise with the last value
> > that was set. This seems to pull the value from the device, and put it
> > into the queue as a starting point. When we start streaming a sensor,
> the
> > settings will be applied without delay. So if we want to intialise the
> > sensor with a new set of controls, it would not work here. Also, if we
> > were running a use case where we needed to do a mode switch (e.g. preview
> > then capture), then the last requested preview controls would not be used
> > to seed the list, and would be lost.
>
> Is this not the same thing but without having to cache it? What ever
> controls are set on the device when we stop streaming are our starting
> point. IIRC in StaggerdCtrls the initial value may not have been put to
> the device but only been at the top of the queue when the streaming
> stopped.
That is indeed what the StaggeredCtrl does in the reset(). If the IPA had
queued up a ctrl to write, that would sit at the end of the delay queue.
On a reset() call (perhaps through a mode switch operation), I want to
write that latest given values to the sensor (i.e. the end of the delayed
queue), not the last used sensor value. The StaggeredCtrl code would do
the former, and the new code above would do the latter correct?
> Another advantage of reading it back here is that we will catch
> any control changes done elsewhere in the pipeline perhaps as part of
> configure().
>
I do see that it might be possible for other parts of the system to do
this. However, I would argue that it is incorrect. Only one module (i.e.
the IPA in our case) should be responsible for arbitrating and setting
these ctrl values. Any other modules that want to go and modify, e.g. gain
and exposure, must go through that one module to avoid such conflicts.
Regards,
Naush
>
> >
> >
> > > +
> > > +/**
> > > + * \brief Push a set of controls on the queue
> > > + * \param[in] controls List of controls to add the device queue
> > > + *
> > > + * Push a set of controls to the control queue. This increases the
> > > control queue
> > > + * depth by one.
> > > + *
> > > + * \returns true if \a controls are accepted, or false otherwise
> > > + */
> > > +bool DelayedControls::push(const ControlList &controls)
> > > +{
> > > + std::lock_guard<std::mutex> lock(lock_);
> > > +
> > > + return queue(controls);
> > > +}
> > > +
> > > +bool DelayedControls::queue(const ControlList &controls)
> > > +{
> > > + /* Copy state from previous frame. */
> > > + for (auto &ctrl : ctrls_) {
> > > + ControlInfo &info = ctrls_[ctrl.first][queueCount_];
> > > + info.value = ctrls_[ctrl.first][queueCount_ - 1].value;
> > > + info.updated = false;
> > > + }
> > > +
> > > + /* Update with new controls. */
> > > + for (const auto &control : controls) {
> > > + const ControlId *id =
> > > device_->controls().idmap().at(control.first);
> > > +
> > > + if (delays_.find(id) == delays_.end())
> > > + return false;
> > >
> >
> > Could you add an error log message here? Do you think this should be a
> > continue, my inclination is that return false is the right thing?
>
> I think this should fail as a control was queued which delay is unknown.
>
> >
> >
> > > +
> > > + ControlInfo &info = ctrls_[id][queueCount_];
> > > +
> > > + info.value = control.second;
> > > + info.updated = true;
> > > +
> > > + LOG(DelayedControls, Debug)
> > > + << "Queuing " << id->name()
> > > + << " to " << info.value.toString()
> > > + << " at index " << queueCount_;
> > > + }
> > > +
> > > + queueCount_++;
> > > +
> > > + return true;
> > > +}
> > > +
> > > +/**
> > > + * \brief Read back controls in effect at a specific sequence number
> > > + * \param[in] sequence Sequence number to get read back controls for
> > > + *
> > > + * Read back what controls where in effect at a specific sequence
> number.
> > > The
> > > + * history is a ring buffer of 16 entries where new and old values
> > > coexist. It's
> > > + * the callers responsibility to not read to old sequence numbers that
> > > have been
> > > + * pushed out of the history.
> > > + *
> > > + * Historic values are evicted by pushing new values onto the queue
> using
> > > + * push(). The max history from the current sequence number that
> yields
> > > valid
> > > + * values are thus 16 minus number of controls pushed.
> > > + *
> > > + * \returns List of controls in effect at \a sequence
> > > + */
> > > +ControlList DelayedControls::get(uint32_t sequence)
> > > +{
> > > + std::lock_guard<std::mutex> lock(lock_);
> > > +
> > > + uint32_t adjustedSeq = sequence - fistSequence_ + 1;
> > > + unsigned int index = std::max<int>(0, adjustedSeq - maxDelay_);
> > >
> >
> > Probably my misunderstanding, but will firstSequence ever be anything
> but 0?
>
> Unfortunately yes. Some V4L2 drivers does not reset their sequence
> number to 0 when starting to stream. So reputing start/stop may have the
> first frame seq+1 from the previous stop.
>
> >
> >
> > > +
> > > + ControlList out(device_->controls());
> > > + for (const auto &ctrl : ctrls_) {
> > > + const ControlId *id = ctrl.first;
> > > + const ControlInfo &info = ctrl.second[index];
> > > +
> > > + out.set(id->id(), info.value);
> > > +
> > > + LOG(DelayedControls, Debug)
> > > + << "Reading " << id->name()
> > > + << " to " << info.value.toString()
> > > + << " at index " << index;
> > > + }
> > > +
> > > + return out;
> > > +}
> > > +
> > > +/**
> > > + * \brief Inform DelayedControls of a start of a new frame
> > > + * \param[in] sequence Sequence number of the frame that started
> > > + *
> > > + * Inform the state machine that a new frame have started and it's
> > > sequence
> > > + * number. It's user of this helpers responsibility to inform the
> helper
> > > + * at the start of every frame. This can with ease be connected to the
> > > start
> > > + * of exposure (SOE) V4L2 event.
> > > + */
> > > +void DelayedControls::frameStart(uint32_t sequence)
> > > +{
> > > + LOG(DelayedControls, Debug) << "frame " << sequence << "
> started";
> > > +
> > > + std::lock_guard<std::mutex> lock(lock_);
> > > +
> > > + if (!running_) {
> > > + fistSequence_ = sequence;
> > > + running_ = true;
> > > + }
> > >
> >
> > As above, can firstSequence_ be anything but 0?
> >
> >
> > > +
> > > + /*
> > > + * Create control list peaking ahead in the value queue to
> ensure
> > > + * values are set in time to satisfy the sensor delay.
> > > + */
> > > + ControlList out(device_->controls());
> > > + for (const auto &ctrl : ctrls_) {
> > > + const ControlId *id = ctrl.first;
> > > + unsigned int delayDiff = maxDelay_ - delays_[id];
> > > + unsigned int index = std::max<int>(0, writeCount_ -
> > > delayDiff);
> > > + const ControlInfo &info = ctrl.second[index];
> > > +
> > > + if (info.updated) {
> > > + out.set(id->id(), info.value);
> > > + LOG(DelayedControls, Debug)
> > > + << "Setting " << id->name()
> > > + << " to " << info.value.toString()
> > > + << " at index " << index;
> > > + }
> > > + }
> > > +
> > > + writeCount_++;
> > > +
> > > + while (writeCount_ >= queueCount_) {
> > > + LOG(DelayedControls, Debug)
> > > + << "Queue is empty, auto queue no-op.";
> > > + queue({});
> > > + }
> > > +
> > > + device_->setControls(&out);
> > > +}
> > > +
> > > +} /* namespace libcamera */
> > > diff --git a/src/libcamera/meson.build b/src/libcamera/meson.build
> > > index 07711b5f93bcc921..19f22f9c94e1d64d 100644
> > > --- a/src/libcamera/meson.build
> > > +++ b/src/libcamera/meson.build
> > > @@ -12,6 +12,7 @@ libcamera_sources = files([
> > > 'controls.cpp',
> > > 'control_serializer.cpp',
> > > 'control_validator.cpp',
> > > + 'delayed_controls.cpp',
> > > 'device_enumerator.cpp',
> > > 'device_enumerator_sysfs.cpp',
> > > 'event_dispatcher.cpp',
> > > --
> > > 2.29.1
> > >
> > >
>
> --
> Regards,
> Niklas Söderlund
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20201110/1f675ea7/attachment-0001.htm>
More information about the libcamera-devel
mailing list