[PVE-User] Different IPs for iSCSI (+zfs) within cluster

Alwin Antreich sysadmin-pve at cognitec.com
Tue Mar 8 08:32:55 CET 2016

Hi list,

why do you use /30 networks? The bridge acts as a switch, you should be able to reach all your connected hosts on the same network.


On 03/07/2016 11:49 PM, Mikhail wrote:
> Hello Proxmox users!
> I'm in process of setting up 3-node HA cluster with shared
> ZFS-over-iSCSI storage. I ran into problem when I cannot setup properly
> my iSCSI storage. The setup follows:
> 1) Storage is a dedicated Linux server with ZFS filesystem. The server
> has 4 network cards. One network card is used for administration/general
> networking. Other three cards are cross-over connected to each of 3
> Proxmox nodes.
> 2) Three identical servers running same latest PVE 4.1 Proxmox community
> version. Each of these Proxmox nodes is cross-over connected using UTP-6
> cable to storage server's network cards. In other words node1 is
> connected to eth1 on storage, node2 is connected to the eth2 on storage,
> node3 is connected to eth3 on storage.
> My cluster is already configured - all nodes added, and all set. Storage
> server runs ZFS filesystem.
> Now I'm trying to setup shared storage on my cluster. First thing I
> thought is that I have to configure proper networking between nodes and
> storage server because all nodes have dedicated connection to storage
> server's network cards as described above. On storage system I created
> bridge interface "br0", added eth1,2,3 network cards into the bridge,
> assigned br0 IP address Next thing, on the nodes I
> configured network card that is connected into storage server: eth1
> (.11, and 12 on next nodes). I expected this to work - so
> that all nodes can access storage server using, but that did
> not work - nodes cannot see (storage) in this setup.
> Then I decided I could use a hostname instead of IP address as a storage
> server target. So I re-configured networking on storage - each NIC was
> assigned /30 address, and each node was also assigned /30 address, so
> that storage and node could connect to each other using individual IPs
> using /30 subnet. Obviously /30 networking setup did the trick and now I
> could reach storage from each node. Next thing I did was to setup
> hostname mapping on nodes in /etc/hosts of every node:
> on node1 - /etc/hosts: storage
> on node2 - /etc/hosts: storage
> and so on.
> After that I went to Proxmox GUI to add shared ZFS-over-iSCSI storage.
> Before entering target name I configured SSH access using key name
> "storage" (/etc/pve/priv/zfs/storage_id_rsa) from every node. As a
> target name, I entered "storage" assuming that each node will use the
> "storage" name instead of IP and this way will allow each node to see
> storage from it's /30 subnet. Obviously this failed because Proxmox
> executed "zfs" commands to storage server using IP address and not
> hostnames as expected. The whole idea failed because node1 can access
>, but node2 cannot access this IP based on above setup.
> Sorry for making this story long, but hopefully I made it clear. How can
> I overcome this problem now? Perhaps I'm doing something wrong with
> bridge setup - I thought that bridging interfaces on storage will solve
> this issue, but it is not working.
> Thanks!
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

with best regards,
Alwin Antreich
IT Analyst
antreich at cognitec.com
Cognitec Systems GmbH
Grossenhainer Strasse 101
01127, Dresden

Geschäftsführer: Alfredo Herrera
Amtsgericht Dresden, HRB 20776
Tel.: +49-351-862-92 0
Fax: +49-351-862-92 10
info at cognitec.com
VAT ID: DE 222661897

More information about the pve-user mailing list