Using libcamera in New Open Source Wildlife Camera System :-)

[w.robertson at cairnwater.com]

Hi Jacopo

Thank you very much!

Thank you - that's great to know that I'm along the right lines with the 
RPi V3 cameras - with 75°, 120° and NoIR I think they should cover the 
needs of most conservationists, zoologists and biologists while those 
with specialist needs can hopefully use the RPi High Quality camera 
(IMX477 sensor) and mount custom optics.

The only other requirement that I can think of is a sensitive monochrome 
image sensor with no IR filter for night time use - I think the ST 
VD56G3 sensor or similar might cover this.

I looked in detail at using a Raspberry Pi but I couldn't work out any 
way of getting a Raspberry Pi into and out of a low power sleep mode - 
eventually I concluded that the way the RPi hardware is designed makes 
it impossible for it to do this - could I have been wrong in this 
conclusion?

To conserve power, CritterCam would have to be able to waken from a low 
power sleep mode or to boot Linux when sensors (usually PIR) detect an 
animal approaching - I reckoned that even with aggressive optimisation 
it likely wouldn't be possible to boot Linux fast enough so a low power 
sleep mode seemed the best option.

Conservationists working on tree dwelling mammals currently risk their 
lives on top of wobbly ladders fiddling to change AA batteries and SD 
cards in commercial wildlife cameras so CritterCam aims to be much more 
independent in the field by using solar panels and a flexible choice of 
rechargeable battery chemistries.

> The algorithms that control the
> image processing are called "IPA" in libcamera lingo. Those algorithms
> are platform-specific (becase ISPs are different on each platform)

Are those IPA algorithms usually provided by the manufacturer of the 
image sensor or by the manufacturer of the μP? I guess optimising the 
ISP might need knowledge of both the image sensor and the ISP hardware? 
(Sorry I'm using the term "μP" in a very inexact way - I mean "the 
microprocessor, ISP and other hardware accelerators on the chip".)

Laurent mentioned that ST seem to be hoping to add support for STM32MP2 
to libcamera in 2025 - ST make both μPs and image sensors and seems to 
be pushing aggressively into both markets so could there be an advantage 
in having the μP and image sensor made by the same company in that the 
IPA could be written by a team with detailed knowledge of both the image 
sensor and the ISP?

I'm a bit unsure about whether it will be possible to process the vast 
volumes of data from 12 MP image sensors so I was wondering if anyone 
had had success with pixel binning on the RPi cameras (telling an image 
sensor or the ISP hardware to average together neighboring pixels to 
reduce the amount of data sent (e.g. averaging together 4 neighbouring 
pixels might reduce 12 MP worth of data to 3 MP worth of data to be 
processed)) or by telling the ISP to consider only the most significant 
bits and leave out one or two of the least significant bits?

I'll try to do a deep delve into the documentation over the coming 
days...

Will

On 2024-12-21 11:51, Jacopo Mondi wrote:
> Hi Will
> 
> On Fri, Dec 20, 2024 at 10:53:58PM +0000, w.robertson at cairnwater.com 
> wrote:
>> Hi all,
>> 
>> It's wonderful to learn about your work and I've enormous respect for 
>> the
>> way you've brought a brilliantly structured solution to the 
>> technically and
>> commercially extremely difficult and complex problem of bringing 
>> camera
>> support to Linux!
>> 
> 
> Thank you so much for your kind words :)
> 
>> I'm hoping to use libcamera to develop a new open source wildlife 
>> camera
>> system that will be much better, much more flexible and much cheaper 
>> than
>> anything available commercially at the moment.
>> 
>> I was wondering if anyone could give me any pointers on what I should 
>> study
>> to get started?
>> 
>> I've got an RPi 4 and I've ordered a DEBIX Model A, DEBIX I/O Board
>> EMB-AS-E01, 3 variants of the RPi Camera V3 (standard, NoIR, wide) and 
>> some
>> of the new connector leads for the RPi Camera. Is there anything else 
>> that I
>> should order for prototyping (maybe a DEBIX Camera 200A, 500A or 
>> 1300A)?
> 
> A few notes: The debix I/O Board camera connector is compatible with 
> the RPi
> cameras cables. The RPi cameras are the defintely the best supported
> ones in the libcamera ecosystem and I suggest you to use those (the v3
> camera is an excellenct choice, but if you have very specific needs in
> terms of optical performances or specific lighting conditions, you
> could consider alternatives).
> 
> Now, that's only the "camera sensor", the part of the system that
> produces RAW (==unusable) frames that gets processed by libcamera
> using the ISP hardware accellerator. The algorithms that control the
> image processing are called "IPA" in libcamera lingo. Those algorithms
> are platform-specific (becase ISPs are different on each platform) and
> here as well the RPi implementation is the most complete and
> performing one we have in our ecosystem. Debix-A is an NXP iMX8MP
> platform, controlled by the 'rkisp1' (historical reasons for the name)
> algorithms, which are catching up with RPi but aren't as complete and
> performing yet.
> 
> RPi in addition has an incredible amount of documentation about the
> setup process, tuning and algorithms control, and a huge use base that
> plays with cameras and interact on forums. rkisp1/imx8mp is catching
> up there as well on the documentation part, specifically about the 
> tuning
> guides.
> 
> If you already have both platforms, I suggest you play with both.
> 
> When it comes to documentation as said RPi has lenghty docs about the
> setup and configuration of cameras on their platforms (you can start
> from here 
> https://www.raspberrypi.com/documentation/accessories/camera.html)
> 
> Our website has guides for both applications and system developers
> which could provide you details on the libcamera stack, most of those
> are probably not necessary to you but might help better framing the
> architecture.
> 
>> 
>> My work is as a climbing arborist and my research is mostly unfunded
>> research into new minimally invasive techniques to carve precision 
>> nest
>> holes and roosting spaces for endangered dormouse, squirrel and bat 
>> species.
> 
> Amazing!
> 
>> 
>> This is a short summary that I wrote in English and German:
>> 
>> https://drive.google.com/file/d/1jkFA-PgDr-O9-nGSgODCUFkukj63F4fc/view?usp=sharing
>> 
>> This is an article with some videos of Garden Dormice exploring our 
>> carved
>> nest holes published in Flemish yesterday by Goedele Verbeylen (I'll 
>> try to
>> get it translated soon):
>> 
>> https://www.natuurpunt.be/nieuws/boomholtes-kerven-voor-bedreigde-eikelmuizen
>> 
>> We have a small website at:
>> 
>> new-homes-for-old-friends.cairnwater.com
> 
> This is all very nice! I wish you all the best and I'm happy to know
> libcamera is helpful to you!
> 
>> 
>> Will