If you allow any of the guest domain or control domain to access the any I/O (PCI) cards directly from the hardware ,usually that domain is called I/O domain or root domain.By default, two virtual I/O services that you need to configure on primary domain to provide the I/O to the guest domains.One is virtual disk (vds) for storage and another one is virtual switch(vsw) for network communication.But in I/O domain,you can directly access the real physical LUN or physical network port.But the typical guest domain will be using the virtual I/O services.For an example, SAN LUN’s will be mapped to control domain and from there , you can map the physical LUNS to guest domains as virtual services.
Oracle VM for SPARC Tutorial:
[checklist]
- Installation and Configuring the Primary domain
- Creating Guest Domains
- Configuring I/O Domain (You Are Here)
- Dynamically adding/removing Resources
- Live Migration for Guest Domains
- Adding ISO image to Multiple Guests
- Configuration Backup & Restore
- Removing the LDOM and Services
[/checklist]
In the above image, control domain aka primary domain and SOL10 Guest domain also called I/O domains since its directly accessing the PCI card(HBA). Here we will see how we can configure the guest domain as I/O domain.
1. Login to control domain and list the complete hardware details of the server .
-bash-3.2# ldm list-devices -a
CORE
ID %FREE CPUSET
0 0 (0, 1, 2, 3)
1 0 (4, 5, 6, 7)
2 0 (8, 9, 10, 11)
3 0 (12, 13, 14, 15)
4 0 (16, 17, 18, 19)
5 0 (20, 21, 22, 23)
6 0 (24, 25, 26, 27)
7 0 (28, 29, 30, 31)
VCPU
PID %FREE PM
0 0 no
1 0 no
2 0 no
3 0 no
4 0 no
5 0 no
6 0 no
7 0 no
8 0 no
9 0 no
10 0 no
11 0 no
12 0 no
13 0 no
14 0 no
15 0 no
16 0 no
17 0 no
18 0 no
19 0 no
20 0 no
21 0 no
22 0 no
23 0 no
24 0 no
25 0 no
26 0 no
27 0 no
28 0 no
29 0 no
30 0 no
31 0 no
MAU
ID CPUSET BOUND
0 (0, 1, 2, 3) primary
1 (4, 5, 6, 7)
2 (8, 9, 10, 11)
3 (12, 13, 14, 15)
4 (16, 17, 18, 19)
5 (20, 21, 22, 23)
6 (24, 25, 26, 27)
7 (28, 29, 30, 31)
MEMORY
PA SIZE BOUND
0x0 512K _sys_
0x80000 1536K _sys_
0x200000 62M _sys_
0x4000000 64M _sys_
0x8000000 2G primary
0x88000000 512M ldom1
0xa8000000 512M ldom3
0xc8000000 896M ldom2
IO
DEVICE PSEUDONYM BOUND OPTIONS
pci@780 bus_a yes
pci@7c0 bus_b yes
-bash-3.2#
In the bottom of the above command output shows that IO (pci@780 and pci7c0) have already bind to some of the domain.
2.To find where the PCI IO card has been bound , you need to use “ldm list-bindings” command.
-bash-3.2# ldm list-bindings
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -ndcv- SP 8 2G 0.6% 4h 58m
UUID
0066bb91-df76-c6ca-a92b-f18faf256978
MAC
00:14:4f:9d:ca:6a
HOSTID
0x849dca6a
CONTROL
failure-policy=ignore
DEPENDENCY
master=
CORE
CID CPUSET
0 (0, 1, 2, 3)
1 (4, 5, 6, 7)
VCPU
VID PID CID UTIL STRAND
0 0 0 0.2% 100%
1 1 0 0.2% 100%
2 2 0 0.1% 100%
3 3 0 0.1% 100%
4 4 1 0.1% 100%
5 5 1 0.4% 100%
6 6 1 0.1% 100%
7 7 1 0.1% 100%
MAU
ID CPUSET
0 (0, 1, 2, 3)
MEMORY
RA PA SIZE
0x408000000 0x8000000 2G
VARIABLES
keyboard-layout=US-English
IO
DEVICE PSEUDONYM OPTIONS
pci@780 bus_a
pci@7c0 bus_b
The above outputs confirms both the PCI card has been assigned to primary domain.So our primary domain is also called I/O domain or root domain.
3.We will remove one the PCI card from primary and we will assign to one of the guest domain. Before removing,please make sure that card is not used in primary .If its in use, please un-configure it .Let me confirm which card can be removed .(pci@780 and pci@7c0)
pci@780 is used for internal disks. We are good to move the pci@7c0 to one of the guest domains since there is no device is currently using it from primary domain.
AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@780/pci@0/pci@9/scsi@0/sd@0,0
1. c0t1d0
/pci@780/pci@0/pci@9/scsi@0/sd@1,0
-bash-3.2# dladm show-dev
vsw0 link: up speed: 1000 Mbps duplex: full
e1000g0 link: up speed: 1000 Mbps duplex: full
e1000g1 link: unknown speed: 0 Mbps duplex: half
e1000g2 link: unknown speed: 0 Mbps duplex: half
e1000g3 link: unknown speed: 0 Mbps duplex: half
-bash-3.2# cat /etc/path_to_inst |grep e1000g
"/pci@780/pci@0/pci@1/network@0" 0 "e1000g"
"/pci@780/pci@0/pci@1/network@0,1" 1 "e1000g"
"/pci@7c0/pci@0/pci@2/network@0" 2 "e1000g"
"/pci@7c0/pci@0/pci@2/network@0,1" 3 "e1000g"
-bash-3.2#
4.List the configured guest domain.
-bash-3.2# ldm list-domain NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -n-cv- SP 8 2G 0.5% 4h 51m ldom1 bound ------ 5000 8 512M ldom2 bound ------ 5001 8 896M ldom3 active -n---- 5002 8 512M 0.0% 3h 20m -bash-3.2#
5. Remove or unbind the pci@7c0 to from primary domain.You can see couple of physical network adapters has been removed from primary.You may need to reboot the server to release the pci cards sometimes.
-bash-3.2# ldm remove-io pci@7c0
-bash-3.2# dladm show-dev
vsw0 link: up speed: 1000 Mbps duplex: full
e1000g0 link: up speed: 1000 Mbps duplex: full
e1000g1 link: unknown speed: 0 Mbps duplex: half
-bash-3.2#
-bash-3.2# ldm list-bindings
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- SP 8 2G 0.2% 3h 57m
IO
DEVICE PSEUDONYM OPTIONS
pci@780 bus_a
6.Now add the pci card pci7c0 to ldom1 .sometimes you may need to reboot the system to see the new pci cards.
# ldm add-io pci@7c0 ldom1 Initiating a delayed reconfiguration operation on the primary domain. All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. -bash-3.2#
7.You can list the binding using “ldm list-bindings” command to confirm the changes.sometimes you need to reboot the system to see the devices in operating system level.
ldm list-bindings ldom1
NAME SERVICE PORT
SP
------------------------------------------------------------------------------
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
ldom1 bound ------ 5000 8 512M
UUID
613e0c0c-3554-610d-e36f-9441124ce387
MAC
00:14:4f:fb:11:83
HOSTID
0x84fb1183
CONTROL
failure-policy=ignore
DEPENDENCY
master=
CORE
CID CPUSET
2 (8, 9, 10, 11)
3 (12, 13, 14, 15)
VCPU
VID PID CID UTIL STRAND
0 8 2 100%
1 9 2 100%
2 10 2 100%
3 11 2 100%
4 12 3 100%
5 13 3 100%
6 14 3 100%
7 15 3 100%
MEMORY
RA PA SIZE
0x8000000 0x88000000 512M
VARIABLES
auto-boot?=false
boot-device=vdisk1:a disk net
keyboard-layout=US-English
IO
DEVICE PSEUDONYM OPTIONS
pci@7c0 bus_b
8. Login to ldom1 and see the pci card devices .
-bash-3.2# dladm show-dev vsw0 link: up speed: 1000 Mbps duplex: full e1000g2 link: up speed: 1000 Mbps duplex: full e1000g3 link: up speed: 1000 Mbps duplex: full -bash-3.2#
Now ldom1 is also called as I/O domain since its accessing the network port directly without hypervisor. The same way if your system have multiple HBA-cards , you can assign one of the cards to guest domain and SAN team can directly allocate tothe luns to guest domain.(ohhh sorry now its called I/O domain)
You have successfully removed the pci card from primary domain and assigned to the guest domain. So the guest domain also become I/O domain now. These I/O domain will be used on special application requirements. Otherwise we will configure only the guest domains.
Suhail says
Can we unbind only the HBA which is completely free and not being used for any other purpose such as for the SAN storage. Is there a way we can still use an HBA or any card both in the primary domain and at the same time in other IO domains ? The same HBA card cannot be shared among different IO doamins and the primary domain?
Thanks.