I have been doing research for my next big series “So You Want to Learn ZFS.” This series is basically going to be a multi-part series of How-To’s which hopefully will give you the ability to build a file server (or even a SAN) based on ZFS if you so choose. However there are a few things that I failed to take into account.
I figured that I’d be able to kind of gloss over the high points of how to get your system up and running and just dive right into the fun ZFS stuff. So before we get into the good stuff there are some basics that we will need to go over first. Today we will cover basic networking.
First off with the acquisition of Sun by Oracle last year the documentation is kind of scattered. The most important place to know of is here. I am sure Oracle will get this under control eventually.
Alright so what makes network configuration so difficult with Solaris 11? Some things are much easier than they should be while others are just ridiculously difficult. I personally attribute this to a tendency towards over-engineering on the part of Sun Engineers, everything is done in the most correct way. Now this is not to say that Solaris is better than everything or that Sun hardware was better than anything else. My basic point is that the most correct way is not always the best way. I think that Sun’s over-engineering hurt them in the long run (which ultimately is why Oracle bought them and not the other way around). However there is one area where I think the over-engineering paid off and the most correct way was actually the best way, this would be ZFS. But I digress that will be for a later article.
If you install Solaris 11 Express then by default a service called Network Auto Magic (NWAM), which simplifies the process significantly, however if you look to do more advanced tasks such as aggregation then this won’t work for you. NWAM is really very much the same as Network Manger, it can provide location based networking profiles and manage multiple types of interfaces (wireless and wired) seamlessly, although it may not be the best for a server configuration.
Disable Network Auto Magic
# svcadm disable network/physical:nwam<br /> # svcadm enable network/physical:default
Once we have disabled NWAM we will lose all network connectivity and configurations.
View the Datalink Devices
Solaris 11 devices have many layers to their configuration, which makes advanced configurations much simpler however does complicate basic configurations. Basically the kernel is aware of the physical hardware and we can see this visibility with the first command.
# dladm show-phys<br /> LINK MEDIA STATE SPEED DUPLEX DEVICE<br /> bge0 Ethernet unknown 1000 full bge0
The second command gives us the ability to see the physical interface linked to a logical interface. After disabling NWAM you will NOT have a logical interface linked to your physical device (in my case bge0) because of this you will see that the state of the data-link device is “unknown”. Also it is important to note that the device names are based off of vendor bge = broadcom and they are incremented based on the number of devices in the machine.
# dladm show-link<br /> LINK CLASS MTU STATE BRIDGE OVER<br /> bge0 phys 1500 unknown -- --
Also before we move on we will just take a look at our existing logical interfaces, the only one you should have after disabling NWAM is lo0 which is your loopback interface.
# ipadm show-if<br /> IFNAME STATE CURRENT PERSISTENT<br /> lo0 ok -m-v------46 ---
Create and Configure a Logical Interface
So the first step is creating a logical interface, then we can apply an IP configuration against it. This will create a link from the logical interface to the physical interface, and will change the state to “up” from “unknown” that we saw before.
# ipadm create-if bge0<br /> # dladm show-link<br /> LINK CLASS MTU STATE BRIDGE OVER<br /> bge0 phys 1500 up -- --<br /> # ipadm show-if<br /> IFNAME STATE CURRENT PERSISTENT<br /> lo0 ok -m-v------46 ---<br /> bge0 down bm--------46 -46<br />
Now above we have successfully created the logical interface and we can now apply an IP configuration to it. This is where it gets a bit tricky. Notice below we are going to apply DHCP as the configuration, we will end up deleting this configuration and making it static, this way you also get the opportunity to learn how to change the configuration (which is really a delete and add). We will go through the specifics of the ipadm create-addr command after we also go over the static command as well since they are very similar.
# ipadm create-addr -T dhcp bge0/v4<br /> # ipadm show-addr<br /> ADDROBJ TYPE STATE ADDR<br /> lo0/v4 static ok 127.0.0.1/8<br /> bge0/v4 dhcp ok 192.168.100.225/24<br /> lo0/v6 static ok ::1/128
Now to delete the DHCP configuration from the logical interface so that we can make it static.
# ipadm delete-addr bge0/v4
And to create a static IP configuration on the logical interface.
ipadm create-addr -T static -a 192.168.100.200/24 bge0/v4<br /> # ipadm show-addr<br /> ADDROBJ TYPE STATE ADDR<br /> lo0/v4 static ok 127.0.0.1/8<br /> bge0/v4 static ok 192.168.100.200/24<br /> lo0/v6 static ok ::1/128
Alright so as we can see these are the two commands to create the configurations.
# ipadm create-addr -T dhcp bge0/v4
# ipadm create-addr -T static -a 192.168.100.200/24 bge0/v4
Now the -T option defines the type of configuration static and dhcp are the most common options, -a is for the address on a static configuration and you will notice that we are not using the logical interface name (bge0), but instead a variation (bge0/v4). This represents the version of the IP protocol the configuration is using. So you can have a bge0/v6 and a bge0/v4.
Alright so you have successfully configured your network interfaces, however NWAM was doing more than just this, so you might not have full network connectivity yet.
Verify Full Network Configuration and Connectivity
Using some of the above commands we can review our configurations.
# ipadm show-addr<br /> ADDROBJ TYPE STATE ADDR<br /> lo0/v4 static ok 127.0.0.1/8<br /> bge0/v4 static ok 192.168.100.200/24<br /> lo0/v6 static ok ::1/128
Additionally we need to verify name resolution and routing in order to be confident in our configuration.
# netstat -r</p> <p>Routing Table: IPv4<br /> Destination Gateway Flags Ref Use Interface<br /> -------------------- -------------------- ----- ----- ---------- ---------<br /> solaris solaris UH 2 0 lo0<br /> 192.168.100.0 192.168.100.200 U 3 1 bge0</p> <p>Routing Table: IPv6<br /> Destination/Mask Gateway Flags Ref Use If<br /> --------------------------- --------------------------- ----- --- ------- -----<br /> solaris solaris UH 2 4 lo0
Above will display the current routing table (which does not have a default route), ensure your default route is defined and correct. If you need to create it use the below command.
# route -p add default 192.168.100.1<br /> add net default: gateway 192.168.100.1<br /> add persistent net default: gateway 192.168.100.1
Once it has been corrected it should look something like this, and you should be able to ping off-net.
# netstat -r</p> <p>Routing Table: IPv4<br /> Destination Gateway Flags Ref Use Interface<br /> -------------------- -------------------- ----- ----- ---------- ---------<br /> default fw01.allanglesit.net UG 2 10466<br /> solaris solaris UH 2 12 lo0<br /> 192.168.100.0 192.168.100.200 U 6 1810 bge0</p> <p>Routing Table: IPv6<br /> Destination/Mask Gateway Flags Ref Use If<br /> --------------------------- --------------------------- ----- --- ------- -----<br /> solaris solaris UH 2 156 lo0
To verify DNS configuration check the /etc/resolv.conf and then verify the functionality with nslookup or dig.
# cat /etc/resolv.conf<br /> domain allanglesit.net<br /> nameserver 192.168.100.22<br /> nameserver 192.168.100.25
# dig www.google.com
Solaris additionally uses /etc/nsswitch.conf to tell the system what types of name resolution to use for different types of lookups. When disabling NWAM (which was configuring /etc/nsswitch.conf for us) then we will have a hosts file only configuration, which means our system won’t attempt to use DNS on its own (nslookup and dig will work since they know to use DNS themselves, but things like Firefox, wget, samba, etc only look to the system for name resolution).
# cat /etc/nsswitch.conf<br /> .<br /> .<br /> hosts: files dns<br /> ipnodes: files dns<br /> .<br /> .
I trimmed the above file for brevity.
At this point you should have full network connectivity without using NWAM. So now just reboot to ensure that your settings persist after a reboot.
For WAY more information…
September 16, 2011
In the comments below you will notice “Kristen” mentioned that the ipadm command has changed in newer builds of Solaris 11. At the time she was using a newer build than I had available to me, so I could not verify her claim, however now I have verified this change against the Solaris 11 Early Adopter release snv_173. So be prepared to make the following changes.
<br /> # ipadm create-if bge0<br /> # ipadm delete-if bge0<br />
Will now be
<br /> # ipadm create-ip bge0<br /> # ipadm delete-ip bge0<br />
The following were not changed:
February 28, 2012
Another astute user “j.marcos” (comment below) pointed out another change in the GA version of Solaris 11
For Solaris 11, instead of disabling network/physical:nwam and enabling network/physical:default we control NWAM by setting the ncp mode to DefaultFixed
# netadm enable -p ncp DefaultFixed
If you wanted to re-enable NWAM then we can set the ncp mode back to Automatic
# netadm enable -p ncp Automatic