CentOS8.2でteam+bridge+vlanを設定してvlanがうまく起動しない時の対応をしました

下記URLを参考にCentOS8.2でteam+bridge+vlanを設定してみました。
vlanデバイスがうまく起動しない時の対応手順も記載しました。

(参考URL)
CentOS 8 でteam + bridge + vlan を構築
https://www.eastforest.jp/centos8/6648

Linux nmcliコマンドによるBonding VLAN Bridge Interface設定
https://metonymical.hatenablog.com/entry/2018/10/27/064820

1.環境
1-1.仮想NIC
ens192:コントロール用NIC(192.168.0.35)
ens224:team01
ens256:team01

1-2.teamの接続名

・team01 (チーミング)
  ・team.vlan01 (タグVLAN)(VLAN ID:10)
    ・team.br01(ブリッジインターフェース)(192.168.28.51)
  ・team.vlan02 (タグVLAN)(VLAN ID:20)
    ・team.br02(ブリッジインターフェース)(192.168.27.51)

team + bridge + vlanの構成にします。

2.設定手順

2-1.ステータス確認

[root@centos82-02 ~]# nmcli con show
NAME           UUID                                  TYPE      DEVICE
System ens192  c9a13c4e-f36f-451b-882a-52030db1d545  ethernet  ens192
System ens224  4a2c06fe-a560-4184-b62b-5d1f1f9f9b43  ethernet  --
System ens256  7accb972-ce71-4483-aa58-2973f125be29  ethernet  --


2-2.teamdのインストール
dnf -y install teamd

2-3.teamインターフェースteam01の作成(LACP)
nmcli connection add type team con-name team01 ifname team01 team.runner lacp
nmcli connection mod team01 ipv4.method disabled ipv6.method ignore
nmcli connection modify team01 team.link-watchers "name=ethtool delay-up=2500 delay-down=1000"


2-4.ens224をteam01にアサイン
nmcli connection add type team-slave autoconnect yes ifname ens224 master team01

2-5.ens256をteam01にアサイン
nmcli connection add type team-slave autoconnect yes ifname ens256 master team01

2-6.team01のUP
nmcli connection up team01

2-7.Bridgeインターフェースteam.br01の作成

nmcli connection add type bridge autoconnect yes con-name team.br01 ifname team.br01
nmcli connection modify team.br01 bridge.stp no
nmcli connection modify team.br01 ipv4.method manual ipv4.address '192.168.28.51/24' ipv4.gateway '192.168.28.1' ipv4.dns '192.168.0.1 192.168.0.2' ipv6.method ignore
nmcli connection up team.br01

2-8.VLANインターフェースteam.vlan01の作成+team.vlan01をteam01にアサイン

nmcli connection add type vlan con-name team.vlan01 ifname team.vlan01 dev team01 id 10

2-9.team.vlan01をteam.br01にアサイン

nmcli connection modify team.vlan01 connection.master team.br01 connection.slave-type bridge
nmcli connection up team.vlan01


2-10.Bridgeインターフェースteam.br02の作成

nmcli connection add type bridge autoconnect yes con-name team.br02 ifname team.br02
nmcli connection modify team.br02 bridge.stp no
nmcli connection modify team.br02 ipv4.method manual ipv4.address '192.168.27.51/24' ipv6.method ignore
nmcli connection up team.br02

2-11.VLANインターフェースteam.vlan02の作成+team.vlan02をteam01にアサイン

nmcli connection add type vlan con-name team.vlan02 ifname team.vlan02 dev team01 id 20

2-12.team.vlan02をteam.br02にアサイン

nmcli connection modify team.vlan02 connection.master team.br02 connection.slave-type bridge
nmcli connection up team.vlan02

3.設定後のステータス確認

[root@centos82-02 ~]# nmcli con show
NAME               UUID                                  TYPE      DEVICE
System ens192      c9a13c4e-f36f-451b-882a-52030db1d545  ethernet  ens192
team.br01         bc476a56-2af7-4ee9-a8ac-412237cb874c  bridge    team.br01
team.br02         962ba66b-bca0-4810-96b8-1088cb792f92  bridge    team.br02
team.vlan01       67452bcd-a426-42e8-ae3e-9470d45446ca  vlan      team.vlan01
team.vlan02       cf06c8ae-6135-4f87-904d-26df50043b45  vlan      team.vlan02
team01            d0535e98-1417-42d1-8d46-bc69ffb3486b  team      team01
team-slave-ens224  f0390091-3b32-4683-822c-d276c114956d  ethernet  ens224
team-slave-ens256  2ae14a03-2b74-4f31-85ac-50d265e077ca  ethernet  ens256
System ens224      4a2c06fe-a560-4184-b62b-5d1f1f9f9b43  ethernet  --
System ens256      7accb972-ce71-4483-aa58-2973f125be29  ethernet  --

[root@centos82-02 ~]# bridge link show
13: team.vlan01@team01: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master team.br01 state disabled priority 32 cost 100
15: team.vlan02@team01: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master team.br02 state disabled priority 32 cost 100
[root@centos82-02 ~]#

[root@centos82-02 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:ae:e3:0a brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.35/24 brd 192.168.0.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master team01 state UP group default qlen 1000
    link/ether 00:0c:29:ae:e3:14 brd ff:ff:ff:ff:ff:ff
4: ens256: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master team01 state UP group default qlen 1000
    link/ether 00:0c:29:ae:e3:14 brd ff:ff:ff:ff:ff:ff
11: team01: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:0c:29:ae:e3:14 brd ff:ff:ff:ff:ff:ff
12: team.br01: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:0c:29:ae:e3:14 brd ff:ff:ff:ff:ff:ff
    inet 192.168.28.51/24 brd 192.168.28.255 scope global noprefixroute team.br01
       valid_lft forever preferred_lft forever
13: team.vlan01@team01: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master team.br01 state LOWERLAYERDOWN group default qlen 1000
    link/ether 00:0c:29:ae:e3:14 brd ff:ff:ff:ff:ff:ff
14: team.br02: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:0c:29:ae:e3:14 brd ff:ff:ff:ff:ff:ff
    inet 192.168.27.51/24 brd 192.168.27.255 scope global noprefixroute team.br02
       valid_lft forever preferred_lft forever
15: team.vlan02@team01: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master team.br02 state LOWERLAYERDOWN group default qlen 1000
    link/ether 00:0c:29:ae:e3:14 brd ff:ff:ff:ff:ff:ff
[root@centos82-02 ~]#


[root@centos82-02 ~]# teamdctl team01 state -v
setup:
  runner: lacp
  kernel team mode: loadbalance
  D-BUS enabled: yes
  ZeroMQ enabled: no
  debug level: 0
  daemonized: no
  PID: 10130
  PID file: /var/run/teamd/team01.pid
ports:
  ens224
    ifindex: 3
    addr: 00:0c:29:ae:e3:14
    ethtool link: 10000mbit/fullduplex/up
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 0
        link up delay: 0
        link down delay: 0
    runner:
      aggregator ID: 0
      selected: no
      state: current
      key: 0
      priority: 255
      actor LACPDU info:
        system priority: 65535
        system: 00:0c:29:ae:e3:14
        key: 0
        port_priority: 255
        port: 3
        state: 0x5
      partner LACPDU info:
        system priority: 65535
        system: 00:0c:29:ae:e3:14
        key: 0
        port_priority: 255
        port: 4
        state: 0x5
  ens256
    ifindex: 4
    addr: 00:0c:29:ae:e3:14
    ethtool link: 10000mbit/fullduplex/up
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 0
        link up delay: 0
        link down delay: 0
    runner:
      aggregator ID: 0
      selected: no
      state: current
      key: 0
      priority: 255
      actor LACPDU info:
        system priority: 65535
        system: 00:0c:29:ae:e3:14
        key: 0
        port_priority: 255
        port: 4
        state: 0x5
      partner LACPDU info:
        system priority: 65535
        system: 00:0c:29:ae:e3:14
        key: 0
        port_priority: 255
        port: 4
        state: 0x5
runner:
  active: yes
  fast rate: no
  system priority: 65535
[root@centos82-02 ~]#

4.なぜかVLANが上がらない時の対応
項番2の設定手順を実施してもリブートするとvlanデバイスがUPしない事象に遭遇。
手動で下記の順番でデバイスを起動すると起動成功することがわかりました。

nmcli con up team.br01
nmcli con up team.br02

nmcli connection up team01

nmcli con up team.vlan01
nmcli con up team.vlan02

nmcli connection up team-slave-ens224
nmcli connection up team-slave-ens256

仮想OSでは、
bridge>team>vlan>team-slaveの順番です。

対処方法としては、
A.nmcliのpriorityで制御する
B.デバイス起動スクリプトを作成し、systemdにサービスとして登録する
が挙げられます。

4-A.nmcliのpriorityで制御する
4-A-1.connection.autoconnect-priorityを設定する。
値は小さなものが優先度が高いです。(デフォルト値は0)

nmcli con modify team.br01 connection.autoconnect-priority 10
nmcli con modify team.br02 connection.autoconnect-priority 10
nmcli con modify team01 connection.autoconnect-priority 20
nmcli con modify team.vlan01 connection.autoconnect-priority 30
nmcli con modify team.vlan02 connection.autoconnect-priority 30
nmcli con modify team-slave-ens224 connection.autoconnect-priority 40
nmcli con modify team-slave-ens256 connection.autoconnect-priority 40

4-A-2.優先度の確認

nmcli -f autoconnect-priority,name c

AUTOCONNECT-PRIORITY  NAME
0                     System ens192
10                    team.br01
10                    team.br02
30                    team.vlan01
30                    team.vlan02
20                    team01
40                    team-slave-ens224
40                    team-slave-ens256
0                     System ens224
0                     System ens256

4-B.デバイス起動スクリプトを作成し、systemdにサービスとして登録する

4-B-1.NetworkManagerによる自動起動の停止
nmcli connection modify team.vlan01 autoconnect no
nmcli connection modify team.vlan02 autoconnect no

nmcli connection modify team.br01 autoconnect no
nmcli connection modify team.br02 autoconnect no

nmcli connection modify team-slave-ens224 autoconnect no
nmcli connection modify team-slave-ens256 autoconnect no

nmcli connection modify team01 autoconnect no

4-B-2.スクリプトの作成(3つ)
vi /root/if_up.sh
#!/bin/bash

nmcli con up team.br01
nmcli con up team.br02

nmcli connection up team01

nmcli con up team.vlan01
nmcli con up team.vlan02

nmcli connection up team-slave-ens224
nmcli connection up team-slave-ens256


vi /root/if_down.sh
#!/bin/bash

nmcli connection down team-slave-ens224
nmcli connection down team-slave-ens256

nmcli con down team.vlan01
nmcli con down team.vlan02

nmcli connection down team01

nmcli con down team.br01
nmcli con down team.br02


vi /root/if_restart.sh
#!/bin/bash

nmcli connection down team-slave-ens224
nmcli connection down team-slave-ens256

nmcli con down team.vlan01
nmcli con down team.vlan02

nmcli connection down team01

nmcli con down team.br01
nmcli con down team.br02

nmcli con up team.br01
nmcli con up team.br02

nmcli connection up team01

nmcli con up team.vlan01
nmcli con up team.vlan02

nmcli connection up team-slave-ens224
nmcli connection up team-slave-ens256


4-B-3.ファイルアクセス権の付与
chmod 755 /root/if_up.sh
chmod 755 /root/if_down.sh
chmod 755 /root/if_restart.sh


4-B-4.サービス作成
vi /etc/systemd/system/network_if.service


[Unit]
# サービスの説明文です。内容は自由です。
Description = network_if service
 
After = network.target

[Service]
# スクリプト実行時のユーザを指定します。
User = root

# systemctl start [サービス名] を実行すると、ここに設定されたスクリプトをsystemdが実行します。
ExecStart = /root/if_up.sh
ExecReload = /root/if_restart.sh
ExecStop = /root/if_down.sh

# サービスが停止した際の動作を指定します。alwaysで常に再起動を実施します。
Restart = no

# 起動完了の判定方法です。simpleでコマンド実行時に起動完了と判断します。
Type = oneshot

# systemd はプロセスが終了した後もサービスがアクティブだとみなします。
RemainAfterExit=yes
 
[Install]
WantedBy=multi-user.target

4-B-5.サービスを認識させる
systemctl daemon-reload

4-B-6.サービス自動起動設定
systemctl enable network_if

5.まとめ
CentOS8.2でteam+bridge+vlanを設定してvlanがうまく起動しない時の対応もしてみました。
仮想でlacpを設定すると対向のスイッチが用意できないのでステータスはLinkdownですが、
LinkdownしていてもデバイスはUPできます。
物理サーバでうまくいくかも試してみたいです。

Comments are closed.