Using libcamera in New Open Source Wildlife Camera System :-)

[Jacopo Mondi]

Hi Will

On Sat, Dec 21, 2024 at 02:22:35PM +0000, w.robertson at cairnwater.com wrote:
> Hi Jacopo
>
> Thank you very much!
>
> Thank you - that's great to know that I'm along the right lines with the RPi
> V3 cameras - with 75°, 120° and NoIR I think they should cover the needs of
> most conservationists, zoologists and biologists while those with specialist
> needs can hopefully use the RPi High Quality camera (IMX477 sensor) and
> mount custom optics.
>
> The only other requirement that I can think of is a sensitive monochrome
> image sensor with no IR filter for night time use - I think the ST VD56G3
> sensor or similar might cover this.
>
> I looked in detail at using a Raspberry Pi but I couldn't work out any way
> of getting a Raspberry Pi into and out of a low power sleep mode -
> eventually I concluded that the way the RPi hardware is designed makes it
> impossible for it to do this - could I have been wrong in this conclusion?

That's something I can't tell. The RPi camera team reads the mailing
list (maybe not exactly every day during this period) but I presume
you would have more luck asking on the rpi forums.

>
> To conserve power, CritterCam would have to be able to waken from a low
> power sleep mode or to boot Linux when sensors (usually PIR) detect an
> animal approaching - I reckoned that even with aggressive optimisation it
> likely wouldn't be possible to boot Linux fast enough so a low power sleep
> mode seemed the best option.
>
> Conservationists working on tree dwelling mammals currently risk their lives
> on top of wobbly ladders fiddling to change AA batteries and SD cards in
> commercial wildlife cameras so CritterCam aims to be much more independent
> in the field by using solar panels and a flexible choice of rechargeable
> battery chemistries.
>
> > The algorithms that control the
> > image processing are called "IPA" in libcamera lingo. Those algorithms
> > are platform-specific (becase ISPs are different on each platform)
>
> Are those IPA algorithms usually provided by the manufacturer of the image
> sensor or by the manufacturer of the μP? I guess optimising the ISP might

ISPs produce statistics and accept parameters in a platform-specific
format, hence algorithms are platform-specific. The implementation of
the algorithms, how they use and combine features of different IPs on
the platforms (gpu, other accelerators etc) and the tuning of the whole
camera pipeline is usually what SoC vendors compete on.

In mainline libcamera we have fully open source algorithms
implementations contributed by vendors (the RPi ones) and fully
open-source algorithms implementations provided and maintained by the
community and by Ideas On Board for the other platforms. Some vendors
decide to distribute their "advanced" algorithms as binaries through
different channels and support their platforms in mainline libcamera with
a reduced number of features in the source implementation.

Sensor driver manufacturers are not involved when it comes to ISP
algorithms development. Knowledge of the ISP, its design and
documentation are usually only accessible to the SoC vendors (again
apart from RPi that fully documents their ISP publicly).

> need knowledge of both the image sensor and the ISP hardware? (Sorry I'm
> using the term "μP" in a very inexact way - I mean "the microprocessor, ISP
> and other hardware accelerators on the chip".)

Yes, to realize and tune a camera system you need knowledge of both
the ISP, the image sensor and the optics. That's not strictly related
to the algorithm's implementation as algorithms need tuning data
to work correctly with a sensor but do not (or should not) require any
code specific to a single sensor.

>
> Laurent mentioned that ST seem to be hoping to add support for STM32MP2 to
> libcamera in 2025 - ST make both μPs and image sensors and seems to be
> pushing aggressively into both markets so could there be an advantage in
> having the μP and image sensor made by the same company in that the IPA
> could be written by a team with detailed knowledge of both the image sensor
> and the ISP?
>

Producing both the SoC and the image sensor it's not something that
gives you any advantage by itself.

If a vendor instead provide tuning data pre-calculated for
a specific set of camera modules designed to work with their platforms
then yes, you would have a choice of pre-tuned cameras to pick from.
This is no different from what RPi does with their official camera
modules which come with pre-tuned and validated tuning files.

> I'm a bit unsure about whether it will be possible to process the vast
> volumes of data from 12 MP image sensors so I was wondering if anyone had
> had success with pixel binning on the RPi cameras (telling an image sensor
> or the ISP hardware to average together neighboring pixels to reduce the
> amount of data sent (e.g. averaging together 4 neighbouring pixels might
> reduce 12 MP worth of data to 3 MP worth of data to be processed)) or by
> telling the ISP to consider only the most significant bits and leave out one
> or two of the least significant bits?

The possible configurations of the camera sensor depend on a few
things, mostly by the Linux driver implementation and by the configuration
selection logic implemented in the libcamera pipeline handler.

To make an example, the imx708 driver in the RPi kernel tree supports
three modes (full FOV, 2x2 binned, 2x2 binned + crop):
https://github.com/raspberrypi/linux/blob/rpi-6.6.y/drivers/media/i2c/imx708.c#L674

The RPi pipeline handler implements a selection logic to pick the "best"
sensor configuration given a desired output streams configuration:
https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/pipeline/rpi/common/pipeline_base.cpp#n937

libcamera offers an API to override the pipeline handler choice, so I
would say your application is free to chose any configuration
offered by the sensor's driver.

>
> I'll try to do a deep delve into the documentation over the coming days...
>

Thanks, I suggest you to just play around a bit with your platforms
and see what better fits your need!

> Will
>
> On 2024-12-21 11:51, Jacopo Mondi wrote:
> > Hi Will
> >
> > On Fri, Dec 20, 2024 at 10:53:58PM +0000, w.robertson at cairnwater.com
> > wrote:
> > > Hi all,
> > >
> > > It's wonderful to learn about your work and I've enormous respect
> > > for the
> > > way you've brought a brilliantly structured solution to the
> > > technically and
> > > commercially extremely difficult and complex problem of bringing
> > > camera
> > > support to Linux!
> > >
> >
> > Thank you so much for your kind words :)
> >
> > > I'm hoping to use libcamera to develop a new open source wildlife
> > > camera
> > > system that will be much better, much more flexible and much cheaper
> > > than
> > > anything available commercially at the moment.
> > >
> > > I was wondering if anyone could give me any pointers on what I
> > > should study
> > > to get started?
> > >
> > > I've got an RPi 4 and I've ordered a DEBIX Model A, DEBIX I/O Board
> > > EMB-AS-E01, 3 variants of the RPi Camera V3 (standard, NoIR, wide)
> > > and some
> > > of the new connector leads for the RPi Camera. Is there anything
> > > else that I
> > > should order for prototyping (maybe a DEBIX Camera 200A, 500A or
> > > 1300A)?
> >
> > A few notes: The debix I/O Board camera connector is compatible with the
> > RPi
> > cameras cables. The RPi cameras are the defintely the best supported
> > ones in the libcamera ecosystem and I suggest you to use those (the v3
> > camera is an excellenct choice, but if you have very specific needs in
> > terms of optical performances or specific lighting conditions, you
> > could consider alternatives).
> >
> > Now, that's only the "camera sensor", the part of the system that
> > produces RAW (==unusable) frames that gets processed by libcamera
> > using the ISP hardware accellerator. The algorithms that control the
> > image processing are called "IPA" in libcamera lingo. Those algorithms
> > are platform-specific (becase ISPs are different on each platform) and
> > here as well the RPi implementation is the most complete and
> > performing one we have in our ecosystem. Debix-A is an NXP iMX8MP
> > platform, controlled by the 'rkisp1' (historical reasons for the name)
> > algorithms, which are catching up with RPi but aren't as complete and
> > performing yet.
> >
> > RPi in addition has an incredible amount of documentation about the
> > setup process, tuning and algorithms control, and a huge use base that
> > plays with cameras and interact on forums. rkisp1/imx8mp is catching
> > up there as well on the documentation part, specifically about the
> > tuning
> > guides.
> >
> > If you already have both platforms, I suggest you play with both.
> >
> > When it comes to documentation as said RPi has lenghty docs about the
> > setup and configuration of cameras on their platforms (you can start
> > from here
> > https://www.raspberrypi.com/documentation/accessories/camera.html)
> >
> > Our website has guides for both applications and system developers
> > which could provide you details on the libcamera stack, most of those
> > are probably not necessary to you but might help better framing the
> > architecture.
> >
> > >
> > > My work is as a climbing arborist and my research is mostly unfunded
> > > research into new minimally invasive techniques to carve precision
> > > nest
> > > holes and roosting spaces for endangered dormouse, squirrel and bat
> > > species.
> >
> > Amazing!
> >
> > >
> > > This is a short summary that I wrote in English and German:
> > >
> > > https://drive.google.com/file/d/1jkFA-PgDr-O9-nGSgODCUFkukj63F4fc/view?usp=sharing
> > >
> > > This is an article with some videos of Garden Dormice exploring our
> > > carved
> > > nest holes published in Flemish yesterday by Goedele Verbeylen (I'll
> > > try to
> > > get it translated soon):
> > >
> > > https://www.natuurpunt.be/nieuws/boomholtes-kerven-voor-bedreigde-eikelmuizen
> > >
> > > We have a small website at:
> > >
> > > new-homes-for-old-friends.cairnwater.com
> >
> > This is all very nice! I wish you all the best and I'm happy to know
> > libcamera is helpful to you!
> >
> > >
> > > Will