libcamera - Development of Open Source Embedded Linux Wildlife Camera System
Laurent Pinchart
laurent.pinchart at ideasonboard.com
Mon Jan 6 09:20:01 CET 2025
Hi Will,
On Fri, Jan 03, 2025 at 09:52:13PM +0000, w.robertson at cairnwater.com wrote:
> Hi Laurent,
>
> Thank you very much. I've redrafted the README.md file to get rid of the
> largely irrelevant information about CPU cores on SoCs and include the
> much more relevant information about ISP instead. Also reduced down the
> list of SoCs to SoCs with current or planned libcamera support. In the
> low power category I got rid of the NXP i.MX RT1170 and replaced it with
> the S32V2 which has an ISP. Let me know if I got it right and I'll
> redraft the 6. Development & Prototyping section:
>
> - **Candidate Processors**
> - **High Quality, High Flexibility**
> - **Supported by libcamera**
> - **ST STM32MP2 Series** CSI-2 #1 (5 Mpixels @30 fps with ISP), #2 (1
> Mpixels @15 fps no ISP)
> - **NXP i.MX 8M Plus** Dual ISPs: up to 12MP and 375MPixels/s
> - **NXP i.MX95** Preproduction: 500 Mpixel/s MIPI-CSI and ISP (2x
> 4-lane, 2.5 Gbps/lane)
> - **Chips Integrating Mali-C55 ISP**
> - **Renesas RZ/V2H** Mali-C55 ISPC
> - Chips Integrating **Mali C52 ISP**
> - Possibly in the longer therm
> - **Chips Integrating Amlogic C3 ISP**
> - Currently in development by Amlogic
That looks fine.
> - **Low Power**
> - **ST STM32N6 Series μCs** ISP, MIPI CSI-2, H264 hardware
> - **NXP S32V2** Embedded ISP for HDR, color conversion, tone mapping,
> etc. enabled by ISP graph tool
I have no experience with those chips, so I can't comment on this.
> > I think that would be a good first target. Using a raw sensor with an
> > SoC that integrates an ISP gives you a very wide range of usable camera
> > sensors, with a wide range of prices and performances. It is the most
> > flexible solution, especially if you want a modular system that allows
> > users to pick a sensor based on their needs and budget. Adding support
> > for a new sensor is relative easy. The downside is the limited choice of
> > SoCs, and the cost of bringing up a new ISP. As libcamera evolves, you
> > can expect support for more ISPs and SoCs, so the situation will improve
> > over time.
>
> Thank you very much - that's an enormous help - I'd got mixed up and had
> thought that the big cost was in the image sensor driver software not in
> the software for the ISP - it's an enormous help to me to know that the
> software for the ISP is the expensive part.
>
> Commercial wildlife camera manufacturers often seem to try to make money
> by price gouging researchers and conservationists but between NXP and ST
> there seems to be healthy competition and they seem to try to aim to
> make money through excellence and by winning market share in large,
> highly competitive and fast-growing markets - they've no interest in
> trying to price gouge conservationists.
>
> All the real intellectual heavy lifting in CritterCam is done by the
> libcamera team and open source so safe into the future.
>
> I don't have the specialist experience or funding needed to design a
> board for a modern SoC but prototyping using an evaluation board for a
> SoM or SBC then designing a carrier board for a SoM or SBC seems
> feasible. Maybe if CritterCam becomes widely used someone could find
> separate funding to reduce production costs by designing a board based
> on a reference design for a SoC without a SoM but that would be some
> time in the future.
>
> > Using a smart sensor with a simpler SoC gives you a much wider range of
> > SoCs to choose from, and a lower price point, and possibly with better
> > low power capabilities. The drawback is the limited choice of camera
> > sensors. If you want to provide very very low power consumption on some
> > models, this is an architecture that may still be worth investigating.
> >
> > In any case, I would recommend picking one of the two options to start
> > with, and exploring the second option later as an extension to the
> > product range if needed.
>
> It's difficult. I think fundamentally solar panels are becoming fairly
> cheap now but a software engineer's time is very expensive so there's an
> economic case for having the flexibility of an OS there and keeping
> firmware development costs to a minimum even if it comes at the price of
> relatively higher power consumption and relatively large solar panels.
>
> I kind-of get the feeling that a highly power efficient μC solution
> might be something to look at separately in the future if funding
> becomes available - STM32N6 and NXP S32V2 were the only μCs I could find
> with an ISP and STM32N6 was only launched in mid-December so the
> technology seems very new.
>
> > Yes, the list of pipeline handlers is the best place to check. STM32MP2
> > support will come (I don't know ST's roadmap, so I can't tell when).
>
> Do ST expect customers to use the V4L2 and Media Controller APIs for the
> time being until libcamera support is implemented?
I'm not sure what they expect their customers to use for the time being,
I believe (and hope) libcamera is the solution they will push forward.
> > Amlogic is working on support for their C3 ISP
>
> Is there a way of working out which of the Amlogic chips use their C3
> ISP or do all Amlogic chips with an ISP use either their C3 ISP or an
> ARM Mali ISP? (I was trying to work out if the Amlogic C308X's ISP is a
> C3 ISP but I couldn't work it out 🙈)
Kieran, do you have more information about this ?
> > There are V2H versions without the C55 ISP but with a simpler "ISP
> > lite" processing pipeline that could also be considered. They should be
> > cheaper, may consume less power, and could still provide acceptable
> > image quality.
>
> I don't think much about minimising the cost of the SoC - I reckon a lot
> of the modern SoCs with a camera pipeline offer exceptional value for
> money - there seems to be a big price increase and increase in power
> consumption between the SoC and the SoM or SBC so I tend to worry more
> about the cost and power consumption of the SoM or SBC. I tend to worry
> about power consumption in sleep mode a lot more than power consumption
> in active mode because wildlife cameras only tend to spend a very small
> proportion of their time in an active state.
>
> > Note that, to use an SoC without an ISP, you will need a full ISP in the
> > camera sensor. As far as I can tell from the data brief, the ISP in the
> > VD1940 doesn't implement a full processing pipeline, and still outputs
> > bayer data.
>
> Thanks - I should be cautious of that.
>
> > That being said, it also depends what you want to record. If your goal
> > is to capture still images, you could possibly get away without a real
> > ISP and use software processing on still images. You would still need to
> > implement AGC and AWB, and some SoCs (in the i.MX7 family for instance)
> > include a statistics computation engine that should be enough to do so.
> > Some camera sensors also implement AGC and AWB in the sensor itself,
> > albeit with less control on the algorithm, so it may or may not work for
> > your use cases.
>
> Goedele Verbeylen strongly prefers videos to still images.
>
> Would it be possible to stream raw data from an image sensor to a raw
> data file in real time then for ISP tasks to be carried out on the raw
> data at a later date when handling data in real time isn't important
> (this could be done after download to a more powerful computer or by a
> μC core gradually during the day when solar energy is plentiful)?
It's theoretically possible, assuming again that auto-exposure is
implemented on the device. That, and auto-focus if needed, are the
only parts that can't be implemented in post-processing. You would
however need to store the data uncompressed, which will require a large
amount of storage, and high bandwidth to the storage medium.
> On 2025-01-02 22:55, Laurent Pinchart wrote:
> > On Thu, Jan 02, 2025 at 11:53:07AM +0000, w.robertson at cairnwater.com wrote:
> >> Hi Laurent,
> >>
> >> Yes - I'm very worried about the ambitiousness of it and the risk of
> >> scope creep wrecking the project - I was thinking about limiting the
> >> scope to SoCs and image sensors supported by libcamera - do you think
> >> that would work? Anyone wanting a SoC or image sensor added that isn't
> >> supported by libcamera would have to estimate feasibility and cost for
> >> that separately.
> >
> > I think that would be a good first target. Using a raw sensor with an
> > SoC that integrates an ISP gives you a very wide range of usable camera
> > sensors, with a wide range of prices and performances. It is the most
> > flexible solution, especially if you want a modular system that allows
> > users to pick a sensor based on their needs and budget. Adding support
> > for a new sensor is relative easy. The downside is the limited choice of
> > SoCs, and the cost of bringing up a new ISP. As libcamera evolves, you
> > can expect support for more ISPs and SoCs, so the situation will improve
> > over time.
> >
> > Using a smart sensor with a simpler SoC gives you a much wider range of
> > SoCs to choose from, and a lower price point, and possibly with better
> > low power capabilities. The drawback is the limited choice of camera
> > sensors. If you want to provide very very low power consumption on some
> > models, this is an architecture that may still be worth investigating.
> >
> > In any case, I would recommend picking one of the two options to start
> > with, and exploring the second option later as an extension to the
> > product range if needed.
> >
> >> > Most (if not all) of the camera sensors you list are raw camera sensors,
> >> > and most of the SoCs you list either don't include an ISP (SAMA7G54,
> >> > R2/G2, AM625, STM32N6, RT1170) or have an ISP that is not supported yet
> >> > but upstream kernel drivers and by libcamera (AM67A, TDA4VM, Genio 510).
> >> > It isn't clear to me how you plan to address that issue.
> >>
> >> Thank you very much. That's an extremely important point.
> >>
> >> What's the best way to work out which SoCs are supported by libcamera?
> >> Is it best to look in the git repository here?:
> >>
> >> https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/pipeline
> >>
> >> with NXP i.MX8M Plus SoC having full support and that ST planning to add
> >> full support for STM32MP2 Series during 2025 ( currently partial support
> >> https://wiki.st.com/stm32mpu/wiki/How_to_use_libcamera )?
> >
> > Yes, the list of pipeline handlers is the best place to check. STM32MP2
> > support will come (I don't know ST's roadmap, so I can't tell when).
> > There's work underway by NXP on i.MX95 support too. We are working on
> > the Mali C55 ISP, which is integrated in the Renesas RZ/V2H. As a
> > stretch goal we want to support the C52, found in at least some Amlogic
> > SoCs, but that's more of a spare time project at this point. Amlogic is
> > working on support for their C3 ISP (see
> > https://lore.kernel.org/r/20241227-c3isp-v5-0-c7124e762ff6@amlogic.com
> > and
> > https://patchwork.libcamera.org/project/libcamera/list/?series=4926), I
> > don't know when it will be ready for production. There will be more.
> >
> >> I think it would be an enormous help for CritterCam to focus only on
> >> SoCs and image sensors supported by libcamera with any SoCs and image
> >> sensors that aren't supported by libcamera being treated as optional
> >> add-on functionality for which costs and feasibility would have to be
> >> estimated separately.
> >>
> >> If I abandon all the following because they do not have a hardware ISP:
> >>
> >> SAMA7G54
> >> i.MX RT1170
> >> Renesas RZ/G2
> >>
> >> However Renesas RZ/G2 could be replaced by Renesas RZ/V2H which has an
> >> ARM Mali-C55 ISP since libcamera is working on Mali-C55 support
> >> https://www.renesas.com/en/software-tool/rzv2h-isp-support-package
> >> https://libcamera.org/entries/2024-01-31.html
> >
> > Indeed. I should have read your mail fully before writing the text above
> > :-) There are V2H versions without the C55 ISP but with a simpler "ISP
> > lite" processing pipeline that could also be considered. They should be
> > cheaper, may consume less power, and could still provide acceptable
> > image quality.
> >
> >> AM67A, TDA4VM and Genio 510 would all be put out of scope until they
> >> support libcamera.
> >
> > Those are SoCs we would really like to support.
> >
> >> STM32N6 can't run Linux but does have a hardware ISP "A dedicated
> >> computer vision pipeline with a MIPI CSI-2 interface and image signal
> >> processing (ISP) ensures compatibility with a wide range of cameras. The
> >> STM32N6 also features an H264 hardware encoder".
> >>
> >> I put together a list of the ST BrightSense image sensors and whether
> >> they do or do not have an integrated ISP or similar (Google Sheets
> >> https://docs.google.com/spreadsheets/d/1S1-Ji6rsUPARtQFwsfhblWhzQ-97r-zk/edit?usp=sharing&ouid=102590517794814819664&rtpof=true&sd=true
> >> or tab separated below):
> >>
> >> VG5761, VG5661, VD5761, VD5661 data sheet
> >> Block diagram of Vx5y61 Shows ISP however web pages don't all seem to
> >> reflect this
> >>
> >> VB56G4A No ISP Embedded autoexposure
> >> VD16GZ No ISP in-sensor AE and various corrections.
> >> VD1940 On-chip bayerization ISP
> >
> > Note that, to use an SoC without an ISP, you will need a full ISP in the
> > camera sensor. As far as I can tell from the data brief, the ISP in the
> > VD1940 doesn't implement a full processing pipeline, and still outputs
> > bayer data.
> >
> > That being said, it also depends what you want to record. If your goal
> > is to capture still images, you could possibly get away without a real
> > ISP and use software processing on still images. You would still need to
> > implement AGC and AWB, and some SoCs (in the i.MX7 family for instance)
> > include a statistics computation engine that should be enough to do so.
> > Some camera sensors also implement AGC and AWB in the sensor itself,
> > albeit with less control on the algorithm, so it may or may not work for
> > your use cases.
> >
> > Lots of options...
> >
> >> VD55G0 No ISP Embedded auto-exposure
> >> VD55G1 No ISP Various on-chip features to optimize image quality, such
> >> as autoexposure, noise reduction, or defect correction
> >> VD5661 No ISP Embedded 16-bit video processing pipe with pixel defect
> >> correction, high dynamic range (HDR) merge with ghost removal, and
> >> programmable compression of dynamic
> >> VD56G3 No ISP in-sensor autoexposure and various corrections.
> >> VD5761 No ISP Embedded 16-bit video processing pipe with pixel defect
> >> correction, high dynamic range (HDR) merge with ghost removal, and
> >> programmable compression of dynamic
> >> VD66GY No ISP in-sensor autoexposure and various corrections.
> >> VG5761 No ISP Embedded 16-bit video processing pipe with pixel defect
> >> correction, high dynamic range (HDR) merge with ghost removal, and
> >> programmable compression of dynamic
> >> VG6640 No ISP An embedded Bayer and monochrome data pre-processor
> >> VB1740 On-chip bayerization ISP
> >> VB1940 On-chip bayerization ISP
> >>
> >> For Sony I couldn't find the data but as far as I could work out the
> >> IMX708 doesn't have an integrated ISP.
> >>
> >> > Wake up latency is also something that will need to be considered very
> >> > carefully. If your power consumption requirements require powering down
> >> > the SoC and DRAM completely, the system will need to cold-boot when
> >> > woken up, which will probably take too long. Some workarounds are
> >> > possible but may require very extensive software development efforts,
> >> > and those workaround may not be portable across different SoCs.
> >>
> >> Power consumption is a major factor - at the moment I'm looking at using
> >> a 120 W nominal solar panel - guessing that real output would be 5 % of
> >> that on average would give 6 W - using that to charge a c. 500 Wh PBa
> >> battery to run a SoM that consumes several hundred mW in sleep / suspend
> >> mode should give enough margin for safety during spring, summer and
> >> autumn - and perhaps winter - the measurement equipment hasn't arrived
> >> yet to test this though.
> >>
> >> > One last (smaller) technical comment: you mention ACPI system states (S0
> >> > to S4), while none the the SoCs you list use ACPI.
> >>
> >> Thanks - I'd just been asking the SoM and SoC manufacturers about a "low
> >> power sleep / suspend mode" not about ACPI specifically and I'd mixed
> >> the two up. I'm cautious about committing to this until I've checked
> >> whether the measured power consumption matches the manufacturer's stated
> >> power consumption.
> >
> > Real life measurement are definitely needed.
> >
> >> On 2025-01-01 23:08, Laurent Pinchart wrote:
> >> > On Tue, Dec 31, 2024 at 02:45:44PM +0000, w.robertson at cairnwater.com
> >> > wrote:
> >> >> Hi Laurent, Jacopo and Kieran,
> >> >>
> >> >> All my best wishes to you for 2025!
> >> >>
> >> >> I put together this overview summary of CritterCam that hopefully
> >> >> explains how the engineering and economics of the situation make it
> >> >> feasible - let me know if it would be of any help to you as an example
> >> >> use case (it might be a good one because I'm not trying to sell anything
> >> >> or make a sales pitch but I am trying to take the speed and aggression
> >> >> of modern software engineering to an until-now far-too-sleepy area of
> >> >> electronics):
> >> >>
> >> >> https://bitbucket.org/WillRobertRobertson/crittercam/src/main/README.md
> >> >
> >> > I went through the document. The project sounds very ambitious, I think
> >> > you will need to carefully plan intermediate steps on your way to the
> >> > end goal.
> >> >
> >> > One point that you may have overlooked is the design of the camera
> >> > pipeline. Roughly speaking, to capture processed image, you can use
> >> >
> >> > - A raw camera sensor connected to an SoC with an ISP (ideally hardware,
> >> > possibly software with GPU offload at the cost of higher power
> >> > consumption).
> >> >
> >> > - A raw camera sensor connected to a standalone ISP, itself connected to
> >> > an SoC without an ISP (but with a camera capture interface, typically
> >> > a CSI-2 or parallel receiver). This increases the cost and power
> >> > consumption due to the external ISP. Wake up latencies can also be
> >> > affected.
> >> >
> >> > - A camera sensor that integrates an ISP (a.k.a. smart sensor or YUV
> >> > sensor) connected to an SoC without an ISP (but with a camera capture
> >> > interface, typically a CSI-2 or parallel receiver).
> >> >
> >> > Most (if not all) of the camera sensors you list are raw camera sensors,
> >> > and most of the SoCs you list either don't include an ISP (SAMA7G54,
> >> > R2/G2, AM625, STM32N6, RT1170) or have an ISP that is not supported yet
> >> > but upstream kernel drivers and by libcamera (AM67A, TDA4VM, Genio510).
> >> > It isn't clear to me how you plan to address that issue.
> >> >
> >> > Wake up latency is also something that will need to be considered very
> >> > carefully. If your power consumption requirements require powering down
> >> > the SoC and DRAM completely, the system will need to cold-boot when
> >> > woken up, which will probably take too long. Some workarounds are
> >> > possible but may require very extensive software development efforts,
> >> > and those workaround may not be portable across different SoCs.
> >> >
> >> > One last (smaller) technical comment: you mention ACPI system states (S0
> >> > to S4), while none the the SoCs you list use ACPI.
> >> >
> >> > Please see below for more comments.
> >> >
> >> >> The in-depth analysis behind that is in separate files (far too long to
> >> >> include in the summary) that're in a private repository for now because
> >> >> they have email addresses, etc. (Think I'm developing a preference for
> >> >> GitHub over BitBucket...)
> >> >>
> >> >> Also translated Goedele's most recent article to English :-)
> >> >>
> >> >> https://new-homes-for-old-friends.cairnwater.com/carved-tree-cavities-for-endangered-dormice/
> >> >>
> >> >> All the best for 2025!
> >> >>
> >> >> Will
> >> >>
> >> >> On 2024-12-22 19:22, w.robertson at cairnwater.com wrote:
> >> >> > Hi Laurent, Jacopo and Kieran,
> >> >> >
> >> >> > I was trying to think of ways that I could help out. All my current
> >> >> > experience is server side Python/.Net/SQL ( www.faclair.com is one
> >> >> > project) and as a climbing arborist so I don't have any kernel
> >> >> > experience.
> >> >> >
> >> >> > I'm compiling a list of SoMs, SiPs and SBCs that should in theory work
> >> >> > with libcamera with relevant information from their technical data
> >> >> > sheets and questions in emails to their manufacturers - at the moment
> >> >> > this is in CritterCam's git repository but I could make this available
> >> >> > for other libcamera users and testers if this would be a help?
> >> >> >
> >> >> > I looked at using a μC with a MIPI CSI-2 interface.
> >> >> >
> >> >> > From a conservation perspective, the research has been very successful
> >> >> > but trying to fund it from my commercial work has been financially
> >> >> > disastrous - I'm moving out of my flat and into my van to save money in
> >> >> > the hope that enough funding to survive might become available next
> >> >> > year - if I tried to port libcamera to a μC I'd go bankrupt long before
> >> >> > the work could be completed.
> >> >> >
> >> >> > The need for CritterCam to outlast all currently available silicon -
> >> >> > keeping up with very rapidly evolving processor and sensor technology -
> >> >> > and to support multiple contributors contributing both software
> >> >> > features and add-on off-the-shelf and custom electronic modules to a
> >> >> > single open source repository also encouraged me to look at a μP and
> >> >> > embedded Linux instead of a μC.
> >> >> >
> >> >> > The only ways I can see at the moment of having any possibility of
> >> >> > getting a high quality camera working on a μC would be to work together
> >> >> > with Raspberry Pi or ST.
> >> >> >
> >> >> > A talk by Naushir Patuck from Raspberry Pi in November 2024 mentioned
> >> >> > that Raspberry Pi were considering the possibility of a μC with MIPI
> >> >> > CSI-2 support:
> >> >> >
> >> >> > https://www.digikey.com/en/blog/webinar-introduction-raspberry-pi-imaging-and-computer-vision-tools
> >> >> >
> >> >> > ST seem to have very recently unveiled the STM32N6x7 (datasheet v1
> >> >> > November 2024) and STM32N6x5 (datasheet link gives 404 error)
> >> >> > Cortex-M55 μC with MIPI CSI-2 support:
> >> >> >
> >> >> > https://www.st.com/en/microcontrollers-microprocessors/stm32n6-series.html
> >> >> >
> >> >> >> the i.MX8MP, for instance, has a Cortex M core
> >> >> >
> >> >> > The STM32MP25 also includes a Cortex M33 - so the i.MX8MP and STM32MP25
> >> >> > raise the theoretical possibility of having both a low-power μC
> >> >> > solution and a more power hungry but much more flexible μP solution
> >> >> > that both use the same silicon - avoiding the potentially costly
> >> >> > headache of CritterCam maintaining seperate low power μC and high
> >> >> > flexibility μP forks - Cortex M can't support CSI-2 so would be limited
> >> >> > to simpler camera modules.
> >> >
> >> > Whether or not you can control the camera hardware (CSI-2 receiver, ISP,
> >> > DMA engines, ...) from a Cortex M core depends on the integration of
> >> > those peripherals in a particular SoC, and thus varies between SoC
> >> > models. It's not an intrinsic property of Cortex M that it can't support
> >> > CSI-2.
> >> >
> >> >> >> I presume you would have more luck asking on the rpi forums.
> >> >> >
> >> >> > Every time the need for a low power sleep or suspend mode has been
> >> >> > raised on the RPi forums the answer has been a very decisive "No.
> >> >> > Switch it off and reboot." - this is a massive problem that renders RPi
> >> >> > SBCs effectively useless for any power sensitive application - it lead
> >> >> > me to look at silicon from ST and NXP instead. I keep hoping that
> >> >> > something might appear from RPi but nothing has.
> >> >> >
> >> >> >> Yes, to realize and tune a camera system you need knowledge of both
> >> >> >> the ISP, the image sensor and the optics.
> >> >> >>
> >> >> >> If a vendor instead provide tuning data pre-calculated for
> >> >> >> a specific set of camera modules designed to work with their platforms
> >> >> >> then yes, you would have a choice of pre-tuned cameras to pick from.
> >> >> >
> >> >> > Thinking about this - I used to work in the physics, chemistry and
> >> >> > nonlinear optics of optoelectronic materials - I don't know if this
> >> >> > could be any help or not - the thought went through my mind of
> >> >> > developing a small optical bench on which an image sensor and any
> >> >> > associated optics could be placed then red, green, blue and IR lasers
> >> >> > modulated and scanned over both it's surface and possibly a known
> >> >> > reference image sensor while a computer reads data from it so that
> >> >> > precise calibration and tuning data for that particular make and model
> >> >> > of image sensor and any associated optics could be determined and open
> >> >> > sourced?
> >> >
> >> > Calibration and tuning for camera modules requires a test bench, but I
> >> > don't think lasers would help. See chapter 6 of
> >> > https://datasheets.raspberrypi.com/camera/raspberry-pi-camera-guide.pdf
> >> > for an example of a tuning procedure and the corresponding tools and
> >> > environment.
> >> >
> >> >> > I don't know if this would be of use or not (perhaps using
> >> >> > this data to train a simple NN could be a way of implementing
> >> >> > calibration and tuning algorithms if NPUs or GPUs become fast enough -
> >> >> > NPUs and GPUs might be good at handling the simple but highly parallel
> >> >> > tensor operations required without proprietory tie-in)?
> >> >> >
> >> >> > This might allow some flaws and limitations inherent in the
> >> >> > manufacturing of the sensor and optics to be compensated for through
> >> >> > firmware tuning and calibration?
> >> >
> >> > Flaws such as defective pixels, or intrinsic physical properties of the
> >> > camera module that degrate the image quality (noise, lens shading, ...)
> >> > are typically corrected by the ISP, based on calibration and tuning
> >> > data.
> >> >
> >> >> >> The implementation of
> >> >> >> the algorithms, how they use and combine features of different IPs on
> >> >> >> the platforms (gpu, other accelerators etc) and the tuning of the whole
> >> >> >> camera pipeline is usually what SoC vendors compete on.
> >> >> >
> >> >> >> Sensor driver manufacturers are not involved when it comes to ISP
> >> >> >> algorithms development. Knowledge of the ISP, its design and
> >> >> >> documentation are usually only accessible to the SoC vendors (again
> >> >> >> apart from RPi that fully documents their ISP publicly).
> >> >> >
> >> >> > Thank you ver much - it's a big help for me to understand that - so
> >> >> > basically one way that businesses like ST and NXP compete with each
> >> >> > other is by competing to provide the best ISP algorithms as well as the
> >> >> > best silicon?
> >> >> >
> >> >> >> The RPi pipeline handler implements a selection logic to pick the
> >> >> >> "best"
> >> >> > sensor configuration given a desired output streams configuration:
> >> >> > https://git.libcamera.org/libcamera/libcamera.git/tree/src/libcamera/pipeline/rpi/common/pipeline_base.cpp#n937
> >> >> >
> >> >> >> libcamera offers an API to override the pipeline handler choice, so I
> >> >> >> would say your application is free to chose any configuration
> >> >> >> offered by the sensor's driver.
> >> >> >
> >> >> > Thank you very much - that's an enormous help - I'd been confused by
> >> >> > that. Typically commercial wildlife cameras boast the number of pixels
> >> >> > - often including nonexistent "interpolated pixels" - whereas the big
> >> >> > limiting factor on the image quality is the signal to noise ratio on
> >> >> > those pixels so binning may be very valuable - particularly in low
> >> >> > light conditions or when processing power and storage are limited.
> >> >> >
> >> >> >> Some vendors
> >> >> >> decide to distribute their "advanced" algorithms as binaries through
> >> >> >> different channels and support their platforms in mainline libcamera with
> >> >> >> a reduced number of features in the source implementation.
> >> >> >
> >> >> > Why don't they contribute their "advanced" proprietory features as
> >> >> > obfuscated and compiled binary plugins for libcamera?
> >> >
> >> > They can do that, and some vendors are looking at this for their newest
> >> > SoCs. I believe ST will have a closed-source plugin (we call those IPA
> >> > modules in libcamera, for Image Processing Algorithms) for the STM32MP2
> >> > (don't quote me on that though). There will be more vendors adopting
> >> > that approach.
> >> >
> >> >> >> Maybe that's why I've never seen a flying squirrel in Nuuksio :-(
> >> >> >
> >> >> > The bad news is that their numbers have been dropping very rapidly. The
> >> >> > good news is that research into Pteromys volans by Ralf Wistbacka and
> >> >> > others and into Red Squirrels, Hazel Dormice and Garden Dormice by
> >> >> > Goedele Verbeylen, Thomas Briner and others has identified lack of
> >> >> > suitable nesting holes as a significant cause for this decline and that
> >> >> > we now have very fast, precise and scalable chainsaw and rotary carving
> >> >> > techniques to create nest holes for them. The other good news is that
> >> >> > Goedele Verbeylen and her colleagues have brought Garden Dormice back
> >> >> > from the brink of extinction in Belgium, proving that it can be done.
> >> >> >
> >> >> >> If you need to
> >> >> >> operate with really low power, keeping the Cortex A cores and DRAM
> >> >> >> powered, even in low-power mode, may go over your power budget.
> >> >> >
> >> >> > Yup. I'm keeping a close eye on reported power consumption for SoMs and
> >> >> > I'll measure power consumption. I'm also trying to keep everything
> >> >> > small to fit into a small amount of RAM and tentatively rejecting any
> >> >> > board that uses DDR4 instead of LPDDR4.
> >> >> >
> >> >> > The power available is limited by the amount of sun, size of solar
> >> >> > panels, size of rechargeable batteries and battery chemistry (Li and
> >> >> > LiFePO4 batteries work well in summer but only PbA batteries work below
> >> >> > a few °C). It may be that tiny wind generators could be added in
> >> >> > difficult conditions. Like any tree dwelling creature, CritterCam has
> >> >> > to be very light, flexible and adaptable to be successful.
> >> >> >
> >> >> > Simply by allowing the battery, solar panel and the μP to be in
> >> >> > seperate boxes, allowing videos and photos to be optionally compressed
> >> >> > and allowing WiFi or Ethernet to be switched on to download data,
> >> >> > CritterCam will remove the need for conservationists and researchers to
> >> >> > risk their lives on wobbly ladders changing batteries and SD cards.
> >> >
> >> > Modular designs are always tempting, but they come at a cost. I would
> >> > advice, for the first version, focussing on the components where
> >> > modularity would bring the best added value, and keeping the other
> >> > components in a more monolithic design. An external solar panel seems to
> >> > make sense, and making the connection waterproof shouldn't be too
> >> > difficult. An external camera module, on the other hand, would be more
> >> > difficult, with (in my opinion) less added value. You could still make
> >> > the design within the main enclosure modular to let users pick among
> >> > different camera sensors, or decide what communication module(s) they
> >> > want to install.
> >> >
> >> >> > An energy and power meter I ordered should arrive in a week or two and
> >> >> > I'll start measuring how much power a solar panel can give in the
> >> >> > typical worst case of a forest floor.
> >> >> >
> >> >> > When things get really desperate an optional GSM module could be
> >> >> > switched on to send a simple SMS or MQTT message to ask someone to come
> >> >> > and change a low battery for a fully charged one and consider adding
> >> >> > more solar panels.
> >> >> >
> >> >> >> C-PHY is more recent, it can provide the same bandwidth with a lower
> >> >> >> number of signals, but isn't as widely supported (I expect that to
> >> >> >> improve over time).
> >> >> >
> >> >> > That's good to know - C-PHY's 3 level signalling must be giving big
> >> >> > headaches for silicon designers used to working with digital signals.
> >> >> > Good to know that for the time being D-PHY is the best supported.
> >> >> >
> >> >> >> CSI-3 was killed by Intel and Qualcomm before it reached the market
> >> >> >
> >> >> > That's good to know. One less thing to keep track of. CSI-1 and CSI-3
> >> >> > RIP.
> >> >> >
> >> >> > Will
> >> >> >
> >> >> > On 2024-12-22 10:30, Laurent Pinchart wrote:
> >> >> >> Hi Will,
> >> >> >>
> >> >> >> On Fri, Dec 20, 2024 at 11:04:09AM +0000, w.robertson at cairnwater.com wrote:
> >> >> >>> Hello Laurent and Kieran,
> >> >> >>>
> >> >> >>> > This sounds very interesting. We rarely get contacted by people working
> >> >> >>> > on projects related to animals or nature. I live in Finland, so I would
> >> >> >>> > be happy to help the flying squirrels from Nuuksio :-)
> >> >> >>>
> >> >> >>> That's wonderful! Dormouse species hibernate but the flying squirrels
> >> >> >>> can't hibernate and so must have nest holes which provide protection
> >> >> >>> from weather and allow them to build well insulated nests to survive the
> >> >> >>> winter, they also ideally need nest holes with entrances which are the
> >> >> >>> right size to let them in but keep the larger red squirrels out. Past
> >> >> >>> forestry practice was to remove dead trees and trees with holes (which
> >> >> >>> were seen as unsightly or unstable) and this has left a drastic shortage
> >> >> >>> of suitable nest holes for the flying squirrels.
> >> >> >>
> >> >> >> Maybe that's why I've never seen a flying squirrel in Nuuksio :-(
> >> >> >>
> >> >> >>> Most of my research is unpaid (with occasional small sponsorship here
> >> >> >>> and there) so resources have to be used very efficiently.
> >> >> >>>
> >> >> >>> By developing new rotary boring and precision chainsaw carving
> >> >> >>> techniques we can carve nest holes entirely through a single narrow,
> >> >> >>> very precise entrance with a much larger nesting chamber behind.
> >> >> >>>
> >> >> >>> > I'll try to help, in my (unfortunately limited) free time. The best
> >> >> >>> > option, if compatible with your needs, would be to have open
> >> >> >>> > discussions
> >> >> >>> > either on the libcamera development mailing list, or on the project's
> >> >> >>> > IRC channel (you can find contact information in
> >> >> >>> > https://libcamera.org/contributing.html). That way other developers
> >> >> >>> > will also be able to chime in.
> >> >> >>>
> >> >> >>> That would be wonderful!
> >> >> >>>
> >> >> >>> > libcamera doesn't currently support the STM32MP2, but ST is working on
> >> >> >>> > it. They showcased a working prototype at Embedded World in April this
> >> >> >>> > year, and I assume they will provide a version usable in production
> >> >> >>> > soon. While they haven't started upstreaming much of their code in
> >> >> >>> > libcamera, I believe they plan to do so. The platform is therefore
> >> >> >>> > interesting for low-power applications.
> >> >> >>>
> >> >> >>> Thank you very much - I'd been unsure about that - that's very helpful
> >> >> >>> to know - I'll maybe try to find a way to give ST some gentle
> >> >> >>> encouragement in the spring - they're pushing aggressively to sell both
> >> >> >>> their image sensors and STM32MP2 so libcamera support should be a high
> >> >> >>> strategic priority for them. I get the feeling that having launched
> >> >> >>> STM32MP2 into both industrial control and domestic appliance markets
> >> >> >>> they've got a fairly heavy workload at the moment and the STM32MP257F-DK
> >> >> >>> dev. board is due to be launched later than planned.
> >> >> >>>
> >> >> >>> > The i.MX8MP is much better supported in libcamera at the moment, and we
> >> >> >>> > are actively improving its support. There are low-cost development
> >> >> >>> > boards, such as the Debix Model A that we use for development (note that
> >> >> >>> > its camera connector is not standard, but they have a Debix I/O board
> >> >> >>> > that can interface the Model A camera connector to a Raspberry Pi 15
> >> >> >>> > pins connector).
> >> >> >>>
> >> >> >>> Thank you very much! That's wonderful to have a good dev. board! I'd
> >> >> >>> been looking at i.MX8MP SoMs and SiPs but some of them still seem to be
> >> >> >>> in the design stage with some technical information missing from data
> >> >> >>> sheets. Will Debix Model A's LPDDR4 allow it to quickly enter and waken
> >> >> >>> from a low power sleep mode?
> >> >> >>
> >> >> >> I haven't experienced with low power sleep and wake up time myself, so I
> >> >> >> can't tell.
> >> >> >>
> >> >> >> Generally speaking, it all depends on your power budget. If you need to
> >> >> >> operate with really low power, keeping the Cortex A cores and DRAM
> >> >> >> powered, even in low-power mode, may go over your power budget. Other
> >> >> >> options are possible, such as turning power to the Cortex A and DRAM
> >> >> >> off, and handling fast wake up tasks in a smaller core (the i.MX8MP, for
> >> >> >> instance, has a Cortex M core that could be used for this). The wake up
> >> >> >> time gets significantly increased, as Linux will need to boot from
> >> >> >> scratch. You may be able to start powering up the camera sensor in
> >> >> >> parallel from the Cortex M core to save some time, or even drive the
> >> >> >> camera completely from the Cortex M, but that would require porting
> >> >> >> libcamera (and drivers) to whatever OS you would be running there, which
> >> >> >> would require really significant effort.
> >> >> >>
> >> >> >>> The 4 lane MIPI CSI seems potentially more
> >> >> >>> flexible than the STM32MP2's 2 lane CSI-2 interface.
> >> >> >>
> >> >> >> 4 lanes will give you more bandwidth, but unless you target really high
> >> >> >> resolutions at high frame rates, that shouldn't be necessary.
> >> >> >>
> >> >> >>> I guess the best way for me to get a Debix Model A or B is just to order
> >> >> >>> it from RS?
> >> >> >>
> >> >> >> I believe so.
> >> >> >>
> >> >> >>> (One thing that's confusing me - when modern data sheets like the Debix
> >> >> >>> Model A data sheet say "MIPI CSI 4-lane" I guess they mean "MIPI CSI-2"?
> >> >> >>> I'd heard of CSI-1 which supports only 1 lane and seems old and rarely
> >> >> >>> used now, of CSI-2 which supports multiple lanes and theoretically a
> >> >> >>> choice of C-PHY of D-PHY physical layers (though I guess many CSI-2
> >> >> >>> implementations maybe don't implement C-PHY because of its complexity?)
> >> >> >>> for the physical layer and CSI-3 which is based on UniPro and rarely
> >> >> >>> implemented at the moment.)
> >> >> >>
> >> >> >> Today the terms "MIPI CSI" (or sometimes just "MIPI camera") refers to
> >> >> >> CSI-2. CSI-1 has retired a very long time ago, and CSI-3 was killed by
> >> >> >> Intel and Qualcomm before it reached the market (there were some
> >> >> >> prototype implementations, but nothing in mass production).
> >> >> >>
> >> >> >> D-PHY is the most common PHY for CSI-2, and should be supported
> >> >> >> everywhere. C-PHY is more recent, it can provide the same bandwidth with
> >> >> >> a lower number of signals, but isn't as widely supported (I expect that
> >> >> >> to improve over time). I don't think the choice of PHY matters too much
> >> >> >> in your case, whatever is supported by both the camera sensor and the
> >> >> >> SoC will be fine.
> >> >> >>
> >> >> >>> Thank you very much for all your help!
> >> >> >>>
> >> >> >>> Will
> >> >> >>>
> >> >> >>> On 2024-12-19 23:17, Laurent Pinchart wrote:
> >> >> >>> > Hello Will,
> >> >> >>> >
> >> >> >>> > (CC'ing my colleague Kieran)
> >> >> >>> >
> >> >> >>> > On Thu, Dec 19, 2024 at 07:46:53PM +0000, w.robertson at cairnwater.com wrote:
> >> >> >>> >> Hi Laurent
> >> >> >>> >>
> >> >> >>> >> Thank you very much for your video "Giving Linux a Camera Stack:
> >> >> >>> >> libcamera's 3 Years Journey and Exciting Future" and the wonderful work
> >> >> >>> >> of you and the libcamera developers in bringing such a powerful and
> >> >> >>> >> structured approach to such an extremely difficult and complex
> >> >> >>> >> problem.
> >> >> >>> >
> >> >> >>> > You're more than welcome.
> >> >> >>> >
> >> >> >>> >> I work as a climbing arborist in practice and research and software
> >> >> >>> >> engineer. My research is mostly voluntary and focused on new minimally
> >> >> >>> >> invasive techniques to carve critically needed nest holes for endangered
> >> >> >>> >> tree dwelling dormouse species (Hazel Dormouse, Garden Dormouse, Forest
> >> >> >>> >> Dormouse), bat species and in Finland the European Flying Squirrel
> >> >> >>> >> (Pteromys volans).
> >> >> >>> >>
> >> >> >>> >> To overcome problems and limitations with commercial wildlife cameras
> >> >> >>> >> for small mammals I started work on CritterCam which - if I can get it
> >> >> >>> >> to work - will be a fully open source wildlife camera system which is
> >> >> >>> >> vastly better, vastly more flexible, vastly more independent in the
> >> >> >>> >> field and significantly cheaper than anything commercially available at
> >> >> >>> >> the moment. Working with embedded Linux and with several different
> >> >> >>> >> manufacturers for the microprocessor and image sensors for CritterCam
> >> >> >>> >> means that CritterCam users can never be price gouged and CritterCam
> >> >> >>> >> will be able to adapt to update all of its critical components with
> >> >> >>> >> newer alternatives well into the future.
> >> >> >>> >>
> >> >> >>> >> This is an article with some videos of Garden Dormice exploring our
> >> >> >>> >> carved nest holes published in Flemish yesterday by Goedele Verbeylen:
> >> >> >>> >>
> >> >> >>> >> https://www.natuurpunt.be/nieuws/boomholtes-kerven-voor-bedreigde-eikelmuizen
> >> >> >>> >>
> >> >> >>> >> This is a short summary that I wrote in English and German:
> >> >> >>> >>
> >> >> >>> >> https://drive.google.com/file/d/1jkFA-PgDr-O9-nGSgODCUFkukj63F4fc/view?usp=sharing
> >> >> >>> >>
> >> >> >>> >> We have a small website at:
> >> >> >>> >>
> >> >> >>> >> new-homes-for-old-friends.cairnwater.com
> >> >> >>> >
> >> >> >>> > This sounds very interesting. We rarely get contacted by people working
> >> >> >>> > on projects related to animals or nature. I live in Finland, so I would
> >> >> >>> > be happy to help the flying squirrels from Nuuksio :-)
> >> >> >>> >
> >> >> >>> >> My background is mainly server side so I don't have kernel experience
> >> >> >>> >> and I was wondering if you or another libcamera might be willing to
> >> >> >>> >> give
> >> >> >>> >> me any pointers to help me get started using supported MIPI CSI-2
> >> >> >>> >> cameras with embedded Linux or ANdroid?
> >> >> >>> >
> >> >> >>> > I'll try to help, in my (unfortunately limited) free time. The best
> >> >> >>> > option, if compatible with your needs, would be to have open
> >> >> >>> > discussions
> >> >> >>> > either on the libcamera development mailing list, or on the project's
> >> >> >>> > IRC channel (you can find contact information in
> >> >> >>> > https://libcamera.org/contributing.html). That way other developers
> >> >> >>> > will
> >> >> >>> > also be able to chime in.
> >> >> >>> >
> >> >> >>> >> I'm hoping to build a first prototype using the STM32MP257F-DK dev.
> >> >> >>> >> board based on the STM32MP257 μP when it becomes available in January
> >> >> >>> >> but I'm also looking at SoMs and SiPs based on other μPs like the NXP
> >> >> >>> >> i.MX 8M Plus that support a CSI-2 interface and a low power sleep mode
> >> >> >>> >> -
> >> >> >>> >> I can also do some initial testing using a Raspberry Pi SBC.
> >> >> >>> >
> >> >> >>> > libcamera doesn't currently support the STM32MP2, but ST is working on
> >> >> >>> > it. They showcased a working prototype at Embedded World in April this
> >> >> >>> > year, and I assume they will provide a version usable in production
> >> >> >>> > soon. While they haven't started upstreaming much of their code in
> >> >> >>> > libcamera, I believe they plan to do so. The platform is therefore
> >> >> >>> > interesting for low-power applications.
> >> >> >>> >
> >> >> >>> > The i.MX8MP is much better supported in libcamera at the moment, and we
> >> >> >>> > are actively improving its support. There are low-cost development
> >> >> >>> > boards, such as the Debix Model A that we use for development (note
> >> >> >>> > that
> >> >> >>> > its camera connector is not standard, but they have a Debix I/O board
> >> >> >>> > that can interface the Model A camera connector to a Raspberry Pi 15
> >> >> >>> > pins connector).
> >> >> >>> >
> >> >> >>> > Raspberry Pi platforms are also well supported, with the work being
> >> >> >>> > performed by Raspberry Pi.
--
Regards,
Laurent Pinchart
More information about the libcamera-devel
mailing list