[RFC] Per-stream controls

[David Plowman]

Hi Naush, everyone

On Wed, 11 Sept 2024 at 09:02, Naushir Patuck <naush at raspberrypi.com> wrote:
>
> Hi,
>
> This RFC introduces the idea of libcamera controls that can be applied
> to one or more specific streams that have been configured via
> camera::configure().  One typical use case would be for applying
> different ScalerCrop controls for individual streams.
>
> To start with, each StreamConfiguration created from
> camera::generateConfiguration() will have a unique id field
> (StreamConfiguration::id_) populated by the pipeline handler.  This id
> will be used to associate a control with a specific stream.
>
> Next, for setting controls ControlValue::set(const T &value) has a new
> overload which accepts a vector of controls and matching vector of
> stream ids:
>
> template<typename T>
> struct StreamValue {
>     unsigned int streamId;
>     T value;
> };
>
> template<typename T>
> void ControlValue::set(const std::vector<StreamValue<T>> &values);
>
> The ControlValue class will need to be updated appropriately to store
> the array of values from the above, including ControlValue::reserve().
> The existing ControlValue::set(const T &value) implies this control is
> set for all streams, exactly the same behavior as we have today.
> Similarly, ControlList::set() will have a new prototype:
>
> template<typename T, typename V>
> void set(const Control<T> &ctrl, const std::vector<StreamValue <V>> &values)
>
> and ditto for the other set() variant:
>
> template<typename T, typename V, size_t Size>
> void set(const Control<Span<T, Size>> &ctrl, const
> std::initializer_list<StreamValue<V>> &values)
>
> There should probably be a helper to test if the ControlValue has been
> set with one of the new prototypes above:
>
> bool ControlValue::isStreamValue();
>
> Next, on the getting controls T ControlValue::get() is updated with a
> new prototype:
>
> template<typename T>
> T ControlValue::get(unsigned int streamId = 0)
> [maybe this should return std::optional<T> in case streamId is not valid?]
>
> or we could also have
>
> std::StreamValue<T> ControlValue::getSreamValues() that the caller
> would have to use if ControlValue::isStreamValue() == true.
>
> The advantage of liking stream ids into the ControlValue directly is
> that it propagates nicely into ControlInfo, and we can get
> min()/max()/def() values per-stream if the pipeline handler were to
> report in such a way.
>
> There are still some questions that I don't have any answers to:
> - Should we tag specific controls as having a per-stream value in the
> yaml file?  The ControlInfo should hopefully inform the application if
> a control is per-stream, but do we need to be more explicit?
> - Error handling, what if the user sets a control on an invalid
> stream.  How is the error propagated?
>
> There's still quite a bit of implementation detail that is missing
> from this proposal, but I think it's a good start for some
> discussions.  Let me know your thoughts.
>
> Regards,
> Naush

Thanks for the proposal, I think it sounds good. Just one significant
point I'd like to add to the discussion.

For a long time, and I talked about this at the time, I've wanted to
run more than one instance of a particular algorithm, and have a
method to direct control values at one or more of these algorithm
instances. The classic case for this is AGC/AEC, when you're doing
multi-exposure HDR. This may seem slightly outlandish to folks who
haven't thought about this before, though I note that at a recent
Karamos face-to-face we learnt that Raspberry Pi are not the only
people who are doing this kind of thing.

The observation is that adding an "id number" when you set controls is
exactly what I want for this purpose. Of course, the id is no longer a
stream number - so you'd have to have a scheme that explains how you
know what the number means.

Obviously I see some difficulties here, many relating to describing
what the behaviour is, whether we're overloading this mechanism in a
confusing way. To be fair, you could regard the "stream id" as being
equivalent to running another instance of an "algorithm" which happens
to apply to another stream. Or something. But there's clearly stuff to
think about. On the other hand, is this a useful feature and do we
have a plan B? I understand this is slated for discussion at the
libcamera face-to-face next week, but unfortunately I'm not able to
attend. But I'd appreciate some opinions on the matter!

Thanks!
David