[pve-devel] Proposal For Podman Container Support

[Filip Schauer]

Hello,

I have skimmed through the OCI Runtime+Image specs.
Here is a rough outline of OCI Runtime Bundles and OCI Images:

An **OCI Runtime Bundle** is:

 > A directory structure that is written ahead of time, distributed, and
 > used to seed the runtime for creating a container and launching a
 > process within it.

 > A Standard Container bundle contains all the information needed to
 > load and run a container. This includes the following artifacts:
 > 1. config.json: contains configuration data. This REQUIRED file MUST
 >    reside in the root of the bundle directory and MUST be named
 >    config.json. See config.json for more details.
 > 2. container’s root filesystem: the directory referenced by root.path,
 >    if that property is set in config.json.

**OCI Images** are the format used in distribution. They need to be
converted in order for the OCI Runtime to be able to use them.

 > When extracting an OCI Image into an OCI Runtime bundle, two
 > orthogonal components of the extraction are relevant:
 > 1. Extraction of the root filesystem from the set of filesystem
 >    layers.
 > 2. Conversion of the image configuration blob to an OCI Runtime
 >    configuration blob.


Once I have properly examined the OCI Runtime spec, I can start
prototyping a basic Rust implementation that leverages LXC for spinning
up containers from OCI Runtime bundles.

Maybe we can make use of this Rust crate [0] for parsing the OCI Runtime
config.
[0] https://github.com/youki-dev/oci-spec-rs

Filip

On 18/03/2025 10:45, Thomas Lamprecht wrote:
> Hi Raven King,
>
> I want to say thanks up-front for trying to improve on of our open
> source projects and reaching out upfront for doing so, highly
> appreciated.
>
> Am 13.03.25 um 19:03 schrieb Raven King:
>> This is my first time writing to this mailing list. I have never
>> contributed to proxmox but I would like to try and write a feature that
>> allows native container support (not inside an LXC or VM).
> FWIW, LXC definitively are "native containers", it's less confusing
> to use application containers (like OCI conform ones) and system
> containers (running a full distro), as LXC can be used for both,
> i.e. docker used LXC for a while for isolating application CTs.
>
>> My goal would be that you could manage those containers much like
>> LXC/VMs with similar UI behavior (resource usage views, easy access to
>> container console, and resource sharing).
>> Its a large undertaking, and I would probably want to get a little
>> experience with the proxmox codebase first.
> I think we should take a step back and not focus on integrating podman
> too much, but rather about adding support directly in our existing
> container toolkit. Actually I'm pondering about adding support for the
> OCI runtime spec [0] and maybe also the OCI image spec [1] over the
> last years and had some light talks with developers about that, so I
> think you and I agree on the end goal already, but not on the road or
> method to get there.
>
> [0]: https://github.com/opencontainers/runtime-spec
> [1]: https://github.com/opencontainers/image-spec
>
>> *Why do this?*
>>       1. It is parroted by users frequently. Just look up "run docker in
>> proxmox" and you will see dozens.
>>       2. It would add a major use case to proxmox.
>>       3. For me personally, it removes a major pain point of using
>> proxmox, which is setting up an LXC to then share resources with to then
>> setup a docker image to then share resources with.
>>           Or using docker directly and tearing my hair out as it
>> magically breaks all my proxmox network config.
> The why's are all fine, the reason it does not exist yet is not because
> we saw no reason, but rather because there are good workarounds that
> can be used and correctly implementing this into our container runtime
> while ensuring as much as possible is shared with our existing
> implementation.
>
>> *Why Podman?*
>>       1. Easy enough to use.
>>       2. Packaging. The support in debian is straightforward and won't
>> confuse anyone. This means the project won't have to maintain podman
>> itself in any way.
>>       3. Security. Podman needs limited privileges to operate compared to
>> docker. This makes it easier to mesh with things such as user accounts.
>>       4. Interop. It easily goes to/from kubernetes, which can help in
>> some enterprise use cases. Also doesn't interact in ways that break
>> existing pve config mechanisms.
>>
>> *What does podman offer an LXC doesn't?*
>>       1. Easy deployment, you can just pull images that someone prebuilt
>> for a purpose, including most docker images.
>>       2. Directly sharing a host directory (not a whole drive) such as
>> single zfs datastore. While achievable in LXC, you have to do a bunch of
>> user mapping and the setup is rather involved.
> You're mixing things here, LXC is a lower-level technology, it simply
> does not care about image management and certainly does not limit PVE
> on sharing CT and/or host directories at all. LXC rather provide
> building blocks that can definitively be used to support these things.
>
>> *What drawbacks have I considered?
>> *1. Using privileged ports in a podman container is a little tricky
>> without root. Proxmox mostly runs as root though, so this is really only
>> a problem for secondary users.
>> 2. I will take a lot of work to ensure the networking works in a way
>> consistent with other networking in proxmox.
>> 3. Increase support burden as users who aren't entirely familiar with
>> docker/podman containers ask questions that could be answered through
>> research.
>> 4. Some services people might want to run, such as nginx proxy manager,
>> are gonna be very hard to use in this way due to number 1.
> IMO above are not that significant but rather minor/mid-level technical
> hurdles that can be overcome, the biggest drawbacks of using podman
> directly are IMO rather:
>
> - two CT stack we need to support, ours and a third-party one
> - dependency on third party developers and a programming language (Go)
>    we do not use at all in any of our projects.
> - while great software, it does not align _that_ well with Proxmox VE's
>    ways of things, thus a clean and good integration that feels native
>    to PVE, and not just tacked on, is IMO rather hard to do.
>
> That all means quite some high permanent maintenance cost, which is what
> would have to bear, so I'm rather opposed to it, at least without some
> concrete plan of someone having intermediate+ experience with not just
> working with PVE but also internal development, as otherwise it's IMO
> too hard to ensure above concerns are unfounded or not relevant.
> In general, I'd highly favor a native implementation that we have control
> over and can neatly integrate into Proxmox VE and all its features (SDN,
> backup, HA that gets extended by orchestration ...), and given that
> there are specifications for what we want to support here, that should
> be (hopefully) doable without extreme effort, and ideally with not much
> more effort than integrating complete solutions like podman, at least
> if one also minds the long term maintenance cost.
>
>> I am writing to the mailing list before even beginning on this endeavor
>> to get several questions answered:
>>
>> 1. Do y'all have any general tips and pointers about navigating and
>> working with the proxmox codebase?
> There is some basic info here:
> https://pve.proxmox.com/wiki/Developer_Documentation
>
> Definitively does not cover all the things though, but basic patch
> handling should be described.
>
>> 2. Where is a good list to grab bugs to get familiar with proxmox
>> structure and what functionality is where? I have some hardware, but I
>> am not capable of testing stuff like multi-gpu setups.
>>       I see
>> https://bugzilla.proxmox.com/describecomponents.cgi?product=pve but
>> there is a lot of components to proxmox. I have a hard time picking a
>> spot to start.
> We try to add a "start-contributing" tag to some issue requests, i.e.:
> https://bugzilla.proxmox.com/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=UNDECIDED&bug_status=REOPENED&bug_status=MORE%20INFO%20NEEDED&bug_status=POSTPONED&list_id=5791&longdesc=\btag%3A\s*start-contributing&longdesc_type=regexp&query_format=advanced&resolution=---
>
> This can be filtered for the "container" component, albeit I did not
> ensured the three issues that come up are valid:
>
> https://bugzilla.proxmox.com/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=MORE%20INFO%20NEEDED&bug_status=UNDECIDED&bug_status=POSTPONED&component=Container%2FLXC&list_id=50251&longdesc=%5Cbtag%3A%5Cs%2Astart-contributing&longdesc_type=regexp&order=Importance&product=pve&query_format=advanced&resolution=---
>
>> 3. Are there any major drawbacks to container support that need
>> consideration?
> Just to ensure we speak of the same: Containers are already supported
> in general, albeit our runtime that wraps LXC and co is targeting
> system containers, not application ones.
>
>> 4. Are there specific drawbacks to podman that need consideration?
> See above.
>
>> 5. Anything else I am overlooking with this idea?
> The way I'd get started is evaluating the OCI specs, pve-container and
> potentially also existing runtimes that implement the OCI specs.
> For then implementing the spec and integrating that into pve-container
> we would definitively open, even lightly encouraging, doing so in rust,
> maybe at least the lower level building blocks for understanding/parsing
> formats as defined in the OCI specs. Then use perlmod [2] to integrate
> that rust modules into the existing pve-container Perl based code.
> You could also just stay in perl, that would be fine for us, but in
> general we try to use rust if possible for new (bigger) features and
> also find that it provides us with a lot of guarantees and modern
> language features to make one lives easier in the long term.
>
> [2]: https://git.proxmox.com/?p=perlmod.git;a=blob;f=README.md;hb=HEAD
>
> That's naturally a lot to ask for a new contributor, but if it was
> easy it would have been already done, and we simply want to favor
> native and well integrated solutions to avoid external dependencies,
> of which drawbacks we had to deal with too much in the past, so we're
> pretty much set on that part.
>
> FWIW, I directly CC'd one developer I talked lightly over adding OCI
> support to PVE, maybe he got some time to think over this and
> spearhead the initial effort.
> If you're still interested into tackling this we naturally will try
> to help you on any specific question, I still appreciate you wanting
> to move this forward, but I also wanted to manage expectations, as
> this might be quite the task, especially for one not accustomed to
> our project and its code basis.
>
> best regards
> Thomas