General

Enterprise 7 Note

Networking drastically changed under Enterprise 7, which now has Network Manager fully integrated (which was recommended to be uninstalled in previous releases when used on a server/static configuration). On 7, the convention is now not to modify any config files (which may be auto-generated, and/or will get overwritten by updates), but to use command-line utilities to modify any of the configuration settings/parameters (which normally does not modify the primary config file, but creates an override config file usually in a separate location).

Network Setup (ent 7)

1. determine interface and connection name setup, for the subsequent nmcli commands, use what it shows for the Connection name (Device and Connection name are normally the same, but not always the same, this needs to be checked to verify what to use).

# nmcli dev

2. configure the interface, IP, and gateway (defaults to "automatic" (DHCP), change to "manual" to be able to configure static parameters - settings saved in an ifcfg-interface file in /etc/sysconfig/network-scripts/, in this example; ifcfg-eno16777728) Note: the network device can be determined by doing a "nmcli dev" (show/list devices, "show" is a default argument in most cases, and does not need to be specified, unless other "show" arguments/details are needed)

# nmcli con mod eno16777728 ipv4.method manual ipv4.addresses 172.100.200.140/24 ipv4.gateway 172.100.200.1

3. configure hostname (saved in /etc/hostname) and domain (saved in an ifcfg-interface file in /etc/sysconfig/network-scripts/)

# nmcli gen hostname ks-c7a.lab.example.com
# nmcli con mod eno16777728 ipv4.dns-search lab.example.com

4. configure DNS servers (see Data Center Services page for the DNS servers to use for each data center - saved in an ifcfg-interface file in /etc/sysconfig/network-scripts/)

# nmcli con mod eno16777728 ipv4.dns 172.100.170.90,172.100.130.90

5. optional, list networking/connection info

# nmcli con show eno16777728

Network Setup (before ent 7)

most of this is common knowledge, but some of these additional steps have been added (moreso) as a comparison to the new ent 7 listing/steps (above)

1. configure the IP and gateway 1a. set the following in an ifcfg-interface file (usually ifcfg-eth0) under /etc/sysconfig/network-scripts/ Note: ent 6 defaults to/prefers quotes around the params, i.e. BOOTPROTO="none", ent 6 also introduced CIDR notation, ex; PREFIX="24" to replace the old/longer netmask convention (NETMASK="255.255.255.0")

BOOTPROTO=none
IPADDR=172.200.110.140
NETMASK=255.255.255.0

1b. configure the default gateway and disable the dynamic link-local (DHCP network) address in /etc/sysconfig/network Note: ent 6 defaults to/prefers quotes around the params, i.e. GATEWAY="172.100.130.1"

GATEWAY=172.100.130.1
NOZEROCONF=yes

2. configure hostname 2a. set the FQDN hostname in /etc/sysconfig/network

HOSTNAME=ks-c7a.lab.example.com

2b. set the domain in /etc/resolv.conf

domain lab.example.com

2c. set the IP and hostname info in /etc/hosts (required for 'hostname -s' and other types of resolution)

172.100.200.140   ks-c7a ks-c7a.lab.example.com

3. configure the DNS servers in /etc/resolv.conf (see Data Center Services page for the DNS servers to use for each data center)

nameserver 172.100.170.90
nameserver 172.100.130.90

4. optional, list interface info

Ent 5 or older
# ifconfig
Ent 6
# ip addr

Changing the IP (ent 7)

1. set the new IP and netmask Note: if the IP is not getting set, please check or go through the Network Setup steps (above), specifically see the note on the default setting with DHCP/manual mode

# nmcli con mod eno16777728 ipv4.addresses 172.100.200.140/24 ipv4.gateway 172.100.200.1
OR
# nmcli con mod eno16777728 ipv4.addresses 172.100.200.140/24
# nmcli con mod eno16777728 ipv4.gateway 172.100.200.1

2. restart networking Note: this has worked remotely over ssh, as long as the two commands are entered together as per this example, otherwise doing the single down command will cause you to lose your connection and require console access to resolve/fix

# nmcli con down eno16777728 ; nmcli con up eno16777728

Changing the IP (before ent 7)

1. set the new IP and netmask in the ifcfg-interface file (usually ifcfg-eth0) under /etc/sysconfig/network-scripts/ Note: ent 6 defaults to/prefers quotes around the params, i.e. NETMASK="255.255.255.0", ent 6 also introduced CIDR notation, ex; PREFIX="24" to replace the old/longer netmask convention (NETMASK="255.255.255.0")

IPADDR=172.200.110.140
NETMASK=255.255.255.0

2. set the new default gateway in /etc/sysconfig/network Note: ent 6 defaults to/prefers quotes around the params, i.e. GATEWAY="172.100.130.1"

GATEWAY=172.100.130.1

3. set the IP info in /etc/hosts (required for 'hostname -s' and other types of resolution)

172.100.200.140   ks-c7a.lab.example.com ks-c7a

4. restart networking Note: recommended to use "&" when connected remotely so the command will continue after the network gets disconnected (your session is normally not lost in this case, and you would normally stay connected)

# service network restart &

Adding Additional IP's/Aliases (ent 7)

1. add the additional IP

# nmcli con mod eno16777728 +ipv4.addresses 172.100.200.140/24

2. restart networking Note: this has worked remotely over ssh, and long as the two commands are entered together as per this example, otherwise doing the single down command will cause you to lose your connection and require console access to resolve/fix

# nmcli con down eno16777728 ; nmcli con up eno16777728

Adding Additional IP's/Aliases (before ent 7)

1. create an ifcfg-interface:aliasnumber file in /etc/sysconfig/network-scripts/ (ifcfg-eth0:0 for this example), with the following contents

DEVICE="eth0:0"
IPADDR="172.100.200.140"
NETMASK="255.255.255.0"
ONPARENT="yes"

2. restart networking Note: recommended to use & when connected remotely so the command will continue after the network gets disconnected (your session is normally not lost in this case, and you would normally stay connected)

# service network restart &

Adding Static Routes (ent 7)

Set the address range and gateway address (saved in an route-interface file in /etc/sysconfig/network-scripts/, in this example; route-eno16777728)

# nmcli con mod eno16777728 ipv4.routes "172.200.210.0/24 172.200.210.120"

Adding Static Routes (before ent 7)

Create a route-interface file (for IPv4), example /etc/sysconfig/network-scripts/route-eth0 (or route-bond0 for a bonding interface);

ADDRESS0=172.200.210.0
NETMASK0=255.255.255.0
GATEWAY0=172.200.210.120

Multi-NIC Routing (ent 7)

The multi-NIC routing scenario has not yet been tried/tested on Enterprise 7. Things may work correctly based on (possibly) proper gateway settings per NIC (if this works correctly under ent 7). If not, we know how to add static routes on ent 7, and can replicate the configuration for pre-ent 7 envs via Network Manager (nmcli).

There was some testing done here, we ended up doing the Source-based Routing (below).

Multi-NIC Routing (before ent 7)

Before Enterprise 7, since we could not have a gateway (that works) per interface (even though it lets you set a gateway in every interface config file: which it uses to overwrite the default gateway), we have to set the default gateway to the outside or customer facing network (since we cannot possibly know all IP's/networks these connections would be coming from), and then set static routes to every possible network and host it needs access to for our inside network. These are examples for /etc/sysconfig/network-scripts/route-eth1 (where the eth0/default is the primary/outside/customer network, and eth1 in the secondary/internal/private network).

Static list for NOTEL (example, the NOTEL data center no longer exists)

# default network (set this for your specific env/stack)
ADDRESS0=172.200.200.0
NETMASK0=255.255.255.0
GATEWAY0=172.200.200.1
# VPN network
ADDRESS1=10.100.100.0
NETMASK1=255.255.255.0
GATEWAY1=172.200.200.1
# DNS host 1
ADDRESS2=210.210.90.80
NETMASK2=255.255.255.255
GATEWAY2=172.200.200.1
# DNS host 2
ADDRESS3=210.210.120.140
NETMASK3=255.255.255.255
GATEWAY3=172.200.200.1
# spacewalk host
ADDRESS4=172.200.90.60
NETMASK4=255.255.255.255
GATEWAY4=172.200.200.1
# trusted host
ADDRESS5=172.200.90.50
NETMASK5=255.255.255.255
GATEWAY5=172.200.200.1

Teaming (ent 7)

1. add the teaming inferface

# nmcli con add type team con-name team0 ifname team0 config '{"runner": {"name": "loadbalance"}}'

2. set IP address info

# nmcli con mod team0 ipv4.method manual ipv4.addresses 172.100.200.140/24

3. add the first NIC

# nmcli con add type team-slave con-name team0-slave1 ifname em1 master team0

4. add the second NIC

# nmcli con add type team-slave con-name team0-slave2 ifname em2 master team0

Bonding (before ent 7)

Before Enterprise 7, interface Bonding was configured via various config files in /etc/sysconfig/network-scripts/ (this has been rewritten in ent 7 and is now called Teaming), example setup;

eth0 config (ifcfg-eth0)

# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
HWADDR=D4:BE:D9:AA:D7:16
MASTER=bond0
SLAVE=yes

eth1 config (ifcfg-eth1)

# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes                                                                                 
HWADDR=D4:BE:D9:AA:D7:18
MASTER=bond0
SLAVE=yes

bond0 config (ifcfg-bond0)

DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
BONDING_OPTS="miimon=100 mode=1"
IPADDR=172.200.110.140
NETMASK=255.255.255.0

Additional bond IP's bond0:0 config (ifcfg-bond0:0)

DEVICE=bond0:0
BOOTPROTO=none
ONBOOT=yes
IPADDR=172.200.110.200
NETMASK=255.255.255.0

ifconfig output

bond0    Link encap:Ethernet  HWaddr D4:BE:D9:AA:D7:16
         inet addr:172.200.110.140  Bcast:172.200.110.255  Mask:255.255.255.0
         inet6 addr: fe80::d6be:d9ff:feaa:d716/64 Scope:Link
         UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
         RX packets:951518061 errors:0 dropped:244110 overruns:0 frame:0
         TX packets:377721364 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:868579848472 (808.9 GiB)  TX bytes:88332253777 (82.2 GiB)

bond0:0  Link encap:Ethernet  HWaddr D4:BE:D9:AA:D7:16
         inet addr:172.200.110.200  Bcast:172.200.110.255  Mask:255.255.255.0
         UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

eth0     Link encap:Ethernet  HWaddr D4:BE:D9:AA:D7:16
         UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
         RX packets:244110 errors:0 dropped:244110 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:15623040 (14.8 MiB)  TX bytes:0 (0.0 b)

eth1     Link encap:Ethernet  HWaddr D4:BE:D9:AA:D7:18
         UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
         RX packets:3095102322 errors:0 dropped:0 overruns:0 frame:0
         TX packets:2613440853 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:2651544232860 (2.4 TiB)  TX bytes:1948544659918 (1.7 TiB)

Renumbering Ports (ent 6)

Example is from a R630 system used as an appliance with 4 ports on the motherboard that had 2 coppper & 2 fiber. For this appliance they wanted the 2 copper ports to be eth0/1 and the fiber be eth2/3, but a recently built system had them designated in reverse. The renaming/mapping went as follows;

eth0 (fiber)  -> eth2
eth1 (fiber)  -> eth3
eth2 (copper) -> eth0
eth3 (copper) -> eth1

Relabel the ports by changing the udev net rules file, change eth0 to eth2, etc, change only the NAME= lines (as mentioned in the comment at the top of the file)

# vi /etc/udev/rules.d/70-persistent-net.rules

Rename all the network config files

# cd /etc/sysconfig/network-script/
# cp ifcfg-eth* /tmp/
# cp /tmp/ifcfg-eth0 ifcfg-eth2
etc

Fix the device names in each file, new ifcfg-eth0 has DEVICE=eth2, change this to say eth0, etc

# vi ifcfg-eth?

Reboot when done to properly pick up all the udev/network config changes/etc

Source-based Routing (ent 7)

Note: using NetworkManager

In this scenario, the system is using the gateway on the primary NIC. Any incoming packets on the 2nd interface end up going out the primary interface, and packets are not returning to devices on the 2nd network.

Note: table '2' was chosen since this is the 2nd NIC. Names can be used if the proper mapping is set in /etc/iproute2/rt_tables

1. Add policy routing to NetworkManager

# yum install NetworkManager-dispatcher-routing-rules
# systemctl enable NetworkManager-dispatcher.service
# systemctl start NetworkManager-dispatcher.service

2. Add policy rule Note: ens33 is the 2nd NIC, 10.60.130.250 is the NIC IP

# vi /etc/sysconfig/network-scripts/rule-ens33
iif ens33 table 2
from 10.60.130.250 table 2

3. Add static routes using policy rules (may be able to do this w/nmcli) Note: 10.60.130.0/24 is the subnet/cidr of the 2nd network, 10.60.130.1 is the gateway

# vi /etc/sysconfig/network-scripts/route-ens33
10.60.130.0/24 dev ens33 table 2
default via 10.60.130.1 dev ens33 table 2

4. Load the new/changed config files

# nmcli connection reload
# nmcli connection down ens33 ; nmcli connection up ens33

Quick Reference

past what is shown above, here are a few additional/helpful commands

Show IP Info (before ent 6)

# ifconfig

Show IP Info (ent 6 and newer)

# ip addr

Show IP Configuration (before ent 7)

# cat /etc/sysconfig/network /etc/sysconfig/network-scripts/ifcfg-e*

Show IP Configuration (ent 7)
Note: run 'nmcli dev' to find the device name

# nmcli dev
DEVICE  TYPE      STATE      CONNECTION
ens32   ethernet  connected  ens32
lo      loopback  unmanaged  --
# nmcli con show ens32

Show Listening Ports (only)
Note: the : in the output normally denotes a listening port

# lsof -nP | grep ":"

Show Network Connections (before ent 7)

# netstat -an

Show Network Connections (ent 7)

# ss -an