Virtual Andy’s new direction can be summarized with a paraphrased interview with Paul Maritz:

Asked guys how many system admins per server Google used, came back to 1 to 1000…VMware ratio was 1:20 or 1:50…have to get to point where enterprises, regardless of where they are consuming IT, have to do it at similar level of efficiency to free up operations spend… If they can’t free up resources they won’t be able to address the legacy code bases

So what’s all of this mean for the site? 

It means exploring new technologies and new tools which will enable sysadmins to do more with less. It means helping sysadmins pitch these tools internally to benefit the business. It means arming sysadmins with the tools to move their organizations to the next generation of technology.

This is a very big update to the post I made in December 2009 entitled VMware Expert System with PowerCLI and Excel. Problem 1: Solved. Problem 2: In progress…

The VMware Expert System has changed significantly. Here’s a summary of the changes:

• Added deployment via PowerCLI and the OSCustomizationSpec/OSCustomizationNicMapping in PowerCLI 4.0 Update 1
• Completely rewritten its capacity gathering functionality using SQL queries instead of PowerCLI
• Added chargeback functionality
• Made a pretty invoice system

It’s not quite ready for posting… but here’s a sneak preview:

1 – Entering in the systems basic requirements (similar to the expert system screenshots) Users can only select guest OSes with a corresponding template that has been created and maintained by our team.

2 – Viewing the quote/invoice. After this step, the document printed as a PDF and sent to the requesting department.

3 – After departmental approval, the deployment details are opened and more specifics are added

It is important to note that this sheet is pretty flexible. It supports up to 4 vNICs, and 60 disks across any datastore that is available to vCenter. Capacity for each disk is checked before allowing a deployment. The templates we use only have one vNIC and virtual hard disk, so the additional required ones are added after deployment. DEPLOY VIRTUAL MACHINE fires off PowerCLI script with the appropriate parameters to make the deployment happen, validates them, and deploys the VM.

4- Cost data. We had to decide how many vCPUs we would be able to get per CPU.  After that, it became a cost accounting exercise. Another important note: we built this to be the full capacity of the two UCS chassis purchased, even though our initial outlay was 8 B200s spread out over two chassis.   We broke our costs down per GB RAM, GB SAN storage, vCPU, and applied fixed VMware Licensing/support and UCS support costs.

I plan to release the spreadsheet, its underlying VBA source code and the PowerCLI in the coming weeks. I’m particularly interested in other chargeback models for similarly sized and positioned organizations.

Luckily, with vCenter Converter, not VMware Converter Standalone, VMware delivers a nice fix for this issue.

Follow normal steps in vCenter Converter – import a machine, setup your source, and as the Converter Agent is installed you’ll be prompted for a location to the VMware SCSI Disk Driver (scsiport.sys, not scsiport.flp) which is accessible by vCenter Server. After that, the P2v works and your Windows 2000 machine will not blue screen.

If you are wondering if Banner UDC can be virtualized the answer is YES. VMware’s website has very little information about virtualizing Banner UDC… there’ just info about virtualizing Banner Xtender, the integrated document imaging solution. Many sysadmins will have difficulty finding a definitive answer to the “Are we supported on VMware?”  question.

Get an account with the Sungard Commons and Sungard Connect and see for yourself. You’ll find Sungard has this to say in FAQ 1-44BOB8:

Banner UDC clients can realize significant cost benefits through the use of virtualization technologies.  Virtualization allows for server consolidation, with potential savings in hardware acquisition and maintenance, operating costs, and administrative time. Virtualization can also play an important role in High Availability and Disaster Recovery strategies. Banner UDC applications are supported with two software solutions, Sun Solaris Containers and VMware ESX Server. Also supported are hardware partitions offered by Solaris Logical Domains (LDoms), HP-UX Virtual Partitions (vPars),  and AIX Logical Partitions (LPARs).  Virtualization is a very dynamic field and additional solutions will be supported in accordance with market demand.

The SunGard Higher Education Support Center will not require clients to replicate every Banner UDC issue in a native supported environment. If we have reason to believe that virtualization is part of the problem, we may ask clients to independently verify that the problem exists in a native environment.  We will also work with virtualization vendors to jointly diagnose and resolve issues. The Support Center is not able to accept virtual images from clients to use as debugging aids. Such images may contain software for which SunGard Higher Education is not licensed.

The note specifically says the Banner Database server is not recommended on vSphere, citing Metalink 249212.1. Google it for yourself: we have found that the Oracle support is just perpetuating fear, uncertainty, and doubt.  Aside from Google, another good place to research is  Sungard’s BORACLE mailing list (customers only) – you’ll find there are several Banner UDC customers large and small running their full suite (including the database) VMware vSphere with no issues.

The support note also mentions ‘Other Third-Party Products’. Many Banner UDC clients use Cognos for reporting/BI. IBM offers Cognos support on VMware. There’s also Luminis, the most popular web portal with Banner UDC, which is also supported as a virtual machine. For more information, see documents titled “Luminis Platform 5″ and “Luminis Platform Virtualization”.

As for our organization, we have been running a couple of parts of the Banner UDC install in virtual machines from the beginning: eVisions Intellecheck and Banner ePrint have given zero problems with regard to virtualization.

Our journey to full suite virtualization started out by performing a large Banner UDC upgrade (Banner 7 to Banner 8) with vSphere. The testing of the upgrade was successful (and so the real upgrade) AND we obtained Oracle DBA buy-in to running Banner UDC on virtual machines.

Since then, the staging environment’s application and database servers are on virtual machines and no one’s noticed.  Once the staging environment pilot is done, the next step is to work toward moving our production servers to vSphere.

“The N Series VMware ESX Host Utilities” have a couple of ISOs that have scripts for tweaking the guest OS disk timeouts.

From the IBM Redbook IBM System Storage N series and VMware vSphere Storage Best Practices:

This is a job for PowerCLI. Inspired from Jase’s script.

Considerations for Linux: Use an account that can do passwordless sudo (or root- you don’t have to have root SSH enabled for Invoke-VMscript).

# Requires VMware Tools to be installed on each guest.
$winisoName = "[datastore] path/to/windows_gos_timeout.iso"
$linisoName = "[datastore] path/to/linux_gos_timeout.iso"
# Get Windows Guest Credentials
Write-Host "Enter Windows Credentials"
$wincred = Get-Credential
# Get Linux Guest Credentials
Write-Host "Enter Linux Credentials"
$lincred = Get-Credential
# Get ESX Host Creds
Write-Host "Enter ESX  Credentials"
$esxcred = Get-Credential
# Get ALL VMs
$vms = Get-VM
foreach($vms as $vm){
	$driveName = "CD/DVD Drive 1"
	$vm = $vm | Get-View
	$dev = $vm.Config.Hardware.Device | where {$_.DeviceInfo.Label -eq $driveName}

	$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
	$spec.deviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec[] (1)
	$spec.deviceChange[0] = New-Object VMware.Vim.VirtualDeviceConfigSpec
	$spec.deviceChange[0].operation = "edit"
	$spec.deviceChange[0].device = $dev
	$spec.deviceChange[0].device.backing = New-Object VMware.Vim.VirtualCdromIsoBackingInfo
	$spec.deviceChange[0].device.backing.fileName = $isoName

	$vm.ReconfigVM_Task($spec)

	$winscript = "regedit /s windows_gos_timeout.reg"
	$linscript = "sudo /media/cdrom/linux_gos_timeout-install.sh"

	if($vm.Guest.OSFullName -match "Microsoft") {
			Invoke-VMScript -HostCredentials $esxcred -GuestCredentials -ScriptText $winscript
	}
	else if($vm.Guest.OSFullName -match "Linux") {
			Invoke-VMScript -HostCredentials $esxcred -GuestCredentials -ScriptText $linscript
	}
}

Back to work.

From the VMware vCenter Converter 4.2 for vCenter Server 4.1 Release Notes:

What’s New

The VMware vCenter Converter 4.2 is a substantial upgrade from vCenter Converter 4.1 and includes the following new functionality (previously found only in vCenter Converter Standalone 4.0.x):

Physical to virtual machine conversion support for Linux sources including:

Red Hat Enterprise Linux 2.1, 3.0, 4.0, and 5.0

SUSE Linux Enterprise Server 8.0, 9.0, 10.0, and 11.0

Ubuntu 5.x, 6.x, 7.x, and 8.x

This sounds too good to be true. I upgraded my lab environment and confirmed:  you can now do scheduled Linux P2V.

The lab setup was ESX 4.1, vCenter 4.1, and vCenter Converter 4.2 for vCenter Server with a RHEL4.7 AS server imported.

Our Problems

Problem 1: Servers are not being virtualized due to a decentralized procurement process

A decentralized server procurement process presents many problems to an organization. There are many gains with standardizing OS/hardware platforms.

Problem 2: Servers are not being virtualized because knowledge is required to make “Virtualize/Don’t Virtualze” decision

The benefits of server virtualization are easy to explain and are a part of our culture. However, the organization has not adopted a “virtualize first” mentality. There is still a lack of stakeholder understanding with regards to virtualization.

Due to lack of knowledge, ROI is not maximized. This knowledge exists in two places – the virtual infrastructure itself and as tacit knowledge with the VMware administrator.

PROBLEM ANALYSIS

Problem 1: Servers are not being virtualized due to a decentralized procurement process
This problem is outside of the scope of the CIS645 class. We’re working on it.

Problem 2: Servers are not being virtualized because knowledge is required to make “Virtualize/Don’t Virtualze” decision

Problem 2 has two major parts.

CAPACITY – CAN OUR VIRTUAL INFRASTRUCTURE SUPPORT THIS APPLICATION?

This question has historically been answered heuristically with ball park figures. Manually gathering current storage and RAM capacity data too time consuming.

CANDIDACY – BASED ON SYSTEM REQUIREMENTS AND INDUSTRY KNOWLEDGE, IS VIRTUALIZATION SUITABLE FOR THIS APPLICATION?

This is the harder question. Typically you’ll hear consultants say “it depends”. Answering this question usually involves a phone call with the VMware administrator. The conversation is series of questions from the administrator to the stakeholder.

RECOMMENDATION

When the two questions have been answered, a recommendation of Virtualize/Don’t Virtualize is made. If a Virtualize decision is made, the VMware administrator must find the optimal storage unit to deploy to and coordinate the deployment with the stakeholder.

SOLUTION DESIGN

USER INTERFACE

The users of this system are already familiar with Excel and would prefer to utilize Excel’s familiarity and What-If scenario planning.

What if we added another 2TB of storage?
What if we upgraded our RAM?
What if we didn’t have to have the license dongle?

Excel quickly enables these questions to be answered. A normal ‘GUI’ application would take more time to develop and would not invite queries of an ad-hoc nature.

CAPACITY

Capacity data resides at several levels: the virtual machine itself, the host, and the data store. The data is put into Excel using VMware’s PowerCLI. PowerCLI is a Windows PowerShell snap-in that integrates with any VMware Virtual Infrastructure. Windows PowerShell also integrates nicely with Excel.
Here are the steps to capacity gathering with the VMware Expert System:

• Open the Excel Spreadsheet
• Clear previously gathered data
• Connect to a vCenter Server
• Gather datastore information
• Gather host information
• Gather virtual machine information
• Write values to ‘Capacity’ Worksheet
• Write values to ‘New Virtual Machine’ Worksheet
• Save Excel Spreadsheet
• Clean up and quit Excel

CANDIDACY

The user of the VMware Expert System will answer a series of questions to determine system candidacy. Through knowledge capture, the conversation with the VMware Administrator does not need to take place. The knowledge is generally accepted by a community of VMware experts.

RECOMMENDATION

After answering the capacity and candidacy questions, the user receives a final recommendation. The recommendation is only “Virtualize” if capacity is available and candidacy is met.

The interface also displays reasons why a machine is not suitable for virtualization to enable What-If analysis.

DECISION TREE

Modified from VI:OPS P2V Decision Tree

RUNNING THE VMWARE EXPERT SYSTEM

PREREQUISITES

• VMware PowerCLI 4.0 or better
  • Installation guide: http://www.vmware.com/support/developer/windowstoolkit/wintk40/doc/viwin_install.pdf

STEPS TO RUN THE VMWARE EXPERT SYSTEM

• Download and extract vmware-expert-system.zip
• Rename launch.tab to launch.bat
• Edit launch.bat, line 2
  • Substitute your path to updatespreadsheet.ps1 where you see “C:\users\%username%\Documents\cis645\Project\vmware_expert_system\updatespreadsheet.ps1″, make sure the path is in quotation marks
• Edit updatespreadsheet.ps1, line 11
  • Substitute your path to vmware_expert_system.xlsm where you see “C:\users\%username%\Documents\cis645\Project\vmware_expert_system\vwmare_expert_system.xlsm”, make sure the path is in quotation marks
• Run click ‘launch.bat’
• A screen similar to this will appear:
  
• Launch the spreadsheet “vmware_expert_system.xlsm” and enable macros
• Enter system requirements
• Press “Send Work Order”

EXAMPLE SYSTEM: NEW WEB SERVER

• Enter the hostname: newwebserver
  • The hostname must not be already existing and must be a valid hostname (“The Internet Engineering Task Force (IETF)”)
• Enter a functional contact: Andy Hill
• Enter a staff contact: Andy Hill
• Select an Operating System: Windows Server 2003
• Enter a storage requirement: 20 GB
  • The minimum storage requirement must be >8 GB and less than the size of a maximum single disk
• Enter a RAM requirement: 1024 MB
  • The minimum RAM requirement is 256MB and must be less than one host and still tolerant of a host failure
• Number of Processors: 1
  • Must be numeric, greater than or equal to 1, less than or equal to 4
• Number of NICs: 1
  • Must be numeric, greater than or equal to 1, less than or equal to 4
• Average CPU utilization: 5%
  • Must be numeric, between 0 and 1, if 4 processors are used average utilization cannot exceed 50%
• Average RAM utilization: 256 MB
  • Must not exceed 8GB
• Average NIC utilization: 1 MBps
  • Must not exceed 100MBps
• Maximum Disk IO: 10 MBps
  • Must not exceed 100MBps
• Answer TRUE/FALSE to the following hardware components
  • Modems: FALSE
  • Fax Cards: FALSE
  • License Dongles: FALSE
  • Security Dongles: FALSE
  • Hardware Encryption: FALSE
• Answer TRUE/FALSE to Vendor Support: TRUE
• Recommendation: Virtualize!

ADD NEW SUPPORTED GUEST OS

VMware’s Guest OS Compatibility Guide (“VMware, Inc.”) is exhaustive and does not line up with Murray State University’s environment. The drop-down list is populated from a hidden worksheet within Excel. For our environment, we limited this drop down to Guests OSes which have regularly maintained templates.

To add, delete, or change an entry in the operating system list follow these steps:

1. Toward the bottom of Excel, right click the current worksheet
2. From the context menu, select “Unhide…”
   
3. From the Unhide Window, Select ‘Supported Guest Operating Systems’ and press OK
   
4. Navigate to the ‘Supported Guest Operating Systems’ Worksheet. Make changes Column A. Only changes in Column A will be reflected in the spreadsheet. Save your changes.

Future Considerations

# VMware Expert System Capacity Gathering
# v0.2
# by Andy Hill
# http://virtualandy.wordpress.com

# gathering data for VMware capacity
$viserver = Read-Host "Enter a vCenter server";
Write-Host "Gathering Excel data...1/8"

$excel = new-object -comobject Excel.Application
# Edit this value to the location of your vmware_expert_system.xlsm
$excelfile = $excel.workbooks.open("C:\Users\andy.hill\Documents\cis645\Project\vmware_expert_system\vmware_expert_system.xlsm")
$worksheet = $excelfile.worksheets.item(3) # Select Capacity Worksheet

Write-Host "Clearing existing capacity data...2/8"
# Clear existing data
$worksheet.Range("A5:N65000").Clear() | out-null
$worksheet.cells.item(1,2) = $viserver

Write-Host "Connecting to $viserver, this may take a moment...3/8"
connect-viserver $viserver -erroraction stop -WarningAction SilentlyContinue | out-null

# datastore information
Write-Host "Gathering disk information...4/8"
$i = 5
$disks = get-datastore
foreach($disk in $disks) {
	$worksheet.cells.item($i, 1) = $disk.name;
	$worksheet.cells.item($i, 2) = $disk.freespaceMB;
	$worksheet.cells.item($i, 3) = $disk.capacityMB;
	$i++;
}
$disk_count = $i;

$i = 5
Write-Host "Gathering host information...5/8"
# host information
Get-VMHost | %{Get-View $_.ID} | %{
	$esx = "" | select Name, NumCpuPackages, NumCpuCores, Hz, Memory
	$esx.NumCpuPackages = $_.Hardware.CpuInfo.NumCpuPackages
	$esx.NumCpuCores = $_.Hardware.CpuInfo.NumCpuCores
	$esx.Hz = $_.Hardware.CpuInfo.Hz
	$esx.Memory = $_.Hardware.MemorySize
	$esx.Name = $_.Name

	$worksheet.cells.item($i, 6) = $esx.Name
	$worksheet.cells.item($i, 7) = $esx.NumCpuPackages
	$worksheet.cells.item($i, 8) = $esx.NumCpuCores
	$worksheet.cells.item($i, 9) = $esx.hz / 1000 / 1000
	$worksheet.cells.item($i, 10) = $esx.memory / 1024 / 1024;
	$i++;
}
$host_count = $i;

# vm information
$i = 5
Write-Host "Gathering virtual machine information...6/8"

get-vm | % {
	$vm = "" | select name, MemoryMB
	$worksheet.cells.item($i, 13) = $_.Name
	$worksheet.cells.item($i, 14) = $_.MemoryMB
	$i++;
}

# Create the totals and amount utilized
$worksheet.cells.item(($i+1),13) = "Total"
$worksheet.cells.item(($i+1),14) = "=sum(N6:N" + $i + ")"
$vm_count = $i;

Write-Host "Writing values to Excel Spreadsheet...7/8"
#add some formatting
$worksheet.cells.item(($disk_count + 2), 1) = "Datastore with most free space";
$worksheet.cells.item(($disk_count + 3), 1) = "Memory (MB) Available";
$worksheet.cells.item(($disk_count + 4), 1) = "Memory Utilization %";
$worksheet.cells.item(($disk_count + 5), 1) = "Storage Available (GB)";
$worksheet.cells.item(($disk_count + 6), 1) = "Storage Utilization %";
$worksheet.cells.item(($disk_count + 7), 1) = "Most Storage Available on a datastore (GB)";

# add the formulas
$worksheet.cells.item(($disk_count + 2), 2) = "=INDEX((A5:A" + $disk_count + "),MATCH(MAX(B5:B" + $disk_count + "),B5:B" + $disk_count + ",0))";
$worksheet.cells.item(($disk_count + 3), 2) = "=SUM(J5:J" + $host_count + ") - N" + ($vm_count+1);
$worksheet.cells.item(($disk_count + 4), 2) = "=N" + ($vm_count+1) + "/SUM(J5:J" + ($host_count-1) + ")"; # n-1 hosts for HA failover
$worksheet.cells.item(($disk_count + 5), 2) = "=SUM(B5:B" + $disk_count + ")/1024";
$worksheet.cells.item(($disk_count + 6), 2) = "=1-SUM(B5:B" + $disk_count + ")/SUM(C5:C" + $disk_count + ")";
$worksheet.cells.item(($disk_count + 7), 2) = "=INDEX((B5:B" + $disk_count + "),MATCH(MAX(B5:B" + $disk_count + "),B5:B" + $disk_count + ",0))/1024";

Write-Host "Saving Excel Spreadsheet...8/8";
# Select main worksheet
$worksheet = $excelfile.worksheets.item(1);
# Update the 'new virtual machine' worksheet with capacity data
$worksheet.cells.item(8,4) = "=Capacity!B" + ($disk_count + 5) + "-'New Virtual Machine'!B8";
$worksheet.cells.item(8,7) = "=MAX(Capacity!B5:" + "B" + ($disk_count - 1) + ")/1024";
$worksheet.cells.item(9,4) = "=(Capacity!B" + ($disk_count +3) + ")/1024";
$worksheet.cells.item(29,2) = "=Capacity!B" + ($disk_count + 2);
$excel.activeworkbook.save();
$excel.quit();

It’s been a busy summer but I’m delivering some new posts:

• iSCSI SAN performance woes in ESX 3.5
• ESX 3.5 -> ESX 3.5 U4 and Virtual Center 2.5 -> vCenter 2.5 U4
• ESX 3.5 U4 Kickstart for IBM xSeries and QLA4050

While the big boys are at VMworld the small shops are back to getting things done.

I’ve edited a few parts out of this, but I spent a while finding several kickstart scripts that accomplished parts of what we needed. I highly customized one for our environment.

What it does:

• Configures licensing for the host using a license server
• Configures NTP
• Adds users, expires their accounts and configures a sudo group
• MOTD
• Configures NICs and VMware ESX Networking
• Creates a script to download and install IBM iSCSI Host Utilities Kit
• Creates a script to download and install QLA4050C BIOS and firmware updates

Thanks to Leo’s ESX 3.5 Kickstart script – part 3.

You will need to download IBM iSCSI Host Utilities Kit from IBM and the QLA4050C BIOS and Firmware from QLogic to a server with scp capabilities.

# make sure this file is UNIX formatted so the line breaks can be handled.
install
lang en_US.UTF-8
langsupport --default en_US.UTF-8
keyboard us
mouse genericwheelps/2 --device psaux
skipx
network --device eth0 --bootproto static --ip <ip> --netmask <netmask> --gateway <gw> --nameserver <dns1>,<dns2> --hostname <hostname> --addvmportgroup=0 --vlanid=0
# Encrypted root password
rootpw --iscrypted
<password>
firewall --enabled
authconfig --enableshadow --enablemd5
timezone America/Chicago
bootloader --location=mbr
# The following is the partition information you requested
# Note that any partitions you deleted are not expressed
# here so unless you clear all partitions first, this is
# not guaranteed to work
vmaccepteula
# test license server
vmlicense --mode=server --server=27000@<vc> --edition=esxFull --features=vsmp,backup
reboot
firewall --enable
clearpart --exceptvmfs --drives=sda
part /boot --fstype ext3 --size=100 --ondisk=sda
part / --fstype ext3 --size=1800 --grow --maxsize=5000 --ondisk=sda
part swap --size=544 --grow --maxsize=544 --ondisk=sda
part /var/log --fstype ext3 --size=100 --grow --ondisk=sda

%packages
grub
@base

%post
cat > /etc/rc.d/rc3.d/S11servercfg << EOF

#Configure NTP
echo "Configuring NTP"
chkconfig --level 345 ntpd on
echo "restrict kod nomodify notrap noquery nopeer" > /etc/ntp.conf
echo "restrict 127.0.0.1" >> /etc/ntp.conf
echo "server <ntp> >> /etc/ntp.conf
echo "driftfile /var/lib/ntp/drift" >> /etc/ntp.conf
echo <ntp>" > /etc/ntp/step-tickers
service ntpd start

#Adding users with default password "changeme" generated with `openssl passwd changeme`

echo "Adding users"
adduser <user1> -p MKgX23V6snwoc
chage -d 0 -M 99999 <user1>
adduser <user2> -p MKgX23V6snwoc
chage -d 0 -M 99999 <user2>
adduser <user3>  -p MKgX23V6snwoc
chage -d 0 -M 99999 <user3>
usermod -G wheel user
usermod -G wheel user2
usermod -G wheel user3
echo "Done adding users"

echo "Configuring sudoers"
cat > /etc/sudoers << SUDO
# sudoers file.
#
# This file MUST be edited with the 'visudo' command as root.
#
# See the sudoers man page for the details on how to write a sudoers file.
#
# Host alias specification
# User alias specification
# Cmnd alias specification
# Defaults specification
Defaults syslog=local2
# User privilege specification
root ALL=(ALL) ALL
# Uncomment to allow people in group wheel to run all commands
%wheel ALL=(ALL) ALL
# Same thing without a password
# %wheel ALL=(ALL) NOPASSWD: ALL
# Samples
# %users ALL=/sbin/mount /cdrom,/sbin/umount /cdrom
# %users localhost=/sbin/shutdown -h now
SUDO
echo "Done configuring sudoers"

echo "Configuring MOTD"
echo "MOTD HERE" > /etc/motd
echo "Done configuring MOTD"

echo "Configuring hosts file"
echo "ip hostname.fqdn hostname" >> /etc/hosts
echo "Done configuring hosts file"

# we have 6 nics
echo "Configuring NIC duplex/speeds"
/usr/sbin/esxcfg-nics -s 1000 -d full vmnic0
/usr/sbin/esxcfg-nics -s 1000 -d full vmnic1
/usr/sbin/esxcfg-nics -s 1000 -d full vmnic2
/usr/sbin/esxcfg-nics -s 1000 -d full vmnic3
/usr/sbin/esxcfg-nics -s 1000 -d full vmnic4
/usr/sbin/esxcfg-nics -s 1000 -d full vmnic5
echo "Configuring NIC duplex/speeds"

echo "Configuring networking"
# VMNetwork
/usr/sbin/esxcfg-vswitch -a vSwitch1
# Blind Switch
/usr/sbin/esxcfg-vswitch -a vSwitch2
# VMkernel
/usr/sbin/esxcfg-vswitch -a vSwitch3
# Add NIC 1 and 3 to vSwitch1 (VMNetwork)
/usr/sbin/esxcfg-vswitch -L vmnic1 vSwitch1
/usr/sbin/esxcfg-vswitch -L vmnic3 vSwitch1
# Add NIC 2 to vSwitch0 (Service Console, already contains NIC 0)
/usr/sbin/esxcfg-vswitch -L vmnic2 vSwitch0
# Add NIC 4 and 5 to vSwitch3 (VMkernel)
/usr/sbin/esxcfg-vswitch -L vmnic4 vSwitch3
/usr/sbin/esxcfg-vswitch -L vmnic5 vSwitch3
# Give appropriate port group labels to vSwitches
/usr/sbin/esxcfg-vswitch -A "Blind Switch" vSwitch2
/usr/sbin/esxcfg-vswitch -A "VMkernel" vSwitch3
/usr/sbin/esxcfg-vswitch -A "VMNetwork" vSwitch1
# Configure IP addresses for service console and VMkernel
/usr/sbin/esxcfg-vswif -i <ip> -n 255.255.255.0 vswif0
/usr/sbin/esxcfg-vmknic -a -i <vmotion address> -n 255.255.255.0 VMotion
/usr/sbin/esxcfg-vswif -E
# Enable SSH Client through firewall
/usr/sbin/esxcfg-firewall -e sshClient
echo "Done configuring networking"

# generate script to download/install HUK, make it executable
echo "Generating host utilities download/install script"
cat > /root/huk-install.sh << HUK
cd /home/user/
scp user@host:/home/user/ibm_iscsi_esx_host_utilities_3_1.tar.gz .
tar -zxf ibm_iscsi_esx_host_utilities_3_1.tar.gz
cd ibm_iscsi_esx_host_utilities_3_1
./install
echo "Done generating host utilities download/install script"
HUK
chmod a+x /root/huk-install.sh

# generate script to download/install iscli and firmware/BIOS updates, make it executable
echo "Generating iscli and firmware update script"
cat > /root/iscli-script.sh << ISCLI
cd /home/user/
scp user@host:/home/user/iscli-1.2.00-15_linux_i386.install.tar.gz user@host:/home/user/ql4022rm.BIN user@host:/home/user/VER4032_03_00_01_53.zip .
tar -xvzf iscli-1.2.00-15_linux_i386.install.tar.gz
unzip VER4032_03_00_01_53.zip
chmod +x iscli.dkms.install.sh
./iscli.dkms.install.sh install
# HBA 0
/usr/local/bin/iscli -f 0 /home/user/qla4022.dl
sleep 5
/usr/local/bin/iscli -bootcode 0 /home/user/ql4022rm.BIN
sleep 5
# HBA 1
/usr/local/bin/iscli -f 1 /home/user/qla4022.dl
sleep 5
/usr/local/bin/iscli -bootcode 1 /home/user/ql4022rm.BIN
sleep 5
reboot
ISCLI
echo "Done generating iscli and firmware script"

# Moves this file so it will not be called on next host boot
mv /etc/rc.d/rc3.d/S11servercfg /root/unsw-setup.sh
rm -f /root/system-info
EOF
/bin/chmod a+x /etc/rc.d/rc3.d/S11servercfg

In general:

• Do lots of up front work with kickstarts and analysis

Each ESX Host

• Put host in maintenance mode
• Shut Down
• File request with storage administrator to make only boot LUN is visible to host as we are about to do some potentially damaging operations
• Put in new HBA (QLA4050)
• Boot to floppy diskette with QLA 4050 BIOS firmware updates
• Upgrade HBA BIOS
• iFlash
• If the system detects a QLx40xx controller, it displays the following message:
• QLx40xx Adapter found at I/O address: xxxxxxxx
• You will need to enter the adapter address
• Select “FB” to flash the BIOS. The iFlash program will write flash to the adapter using ql4022rm.BIN found in the same directory.
• Reboot. Press CTRL+Q on the second (new) HBA to manage boot settings
• Configure Host Adapter according to IP / initiator name
• Configure iSCSI Target
  • You will need:
  • iSCSI name
  • IP Address
  • Subnet Mask
  • Default Gateway
  • iSCSI Target
  • IP Address:port
  • Target Name
  • Host Boot Settings = MANUAL
  • Exit and Reboot
• Insert ESX 3.5 U4 CD (We don’t have PXE boot available yet)
• Reboot system to boot from ESX 3.5 U4 CD
• Install ESX 3.5 U4
• type ‘esx ks=<url to kickstart file> ksdevice=eth0 method=cdrom‘
• More on the kickstart file is here
• Press enter. This installs ESX with all appropriate settings. Ask someone for the root password.
• Log in as root
• sh iscli-script.sh (from the kickstart)
• sh huk-install.sh (from the kickstart)
• Launch VirtualCenter
• Disconnect the host from VirtualCenter (Right click, disconnect)
• Reconnect the host to VirtualCenter (Right click, connect)
• Enter maintenance mode (so no VMs are vMotioned on)
• VMotion doesn’t get set up correctly via kickstart because the host does not have shared storage. Contact the SAN Administrator to make the other ESX LUNs  visible and rescan.
• Delete the VMKernel Switch
• Add the VMkernel switch (nic4 and nic5), enabling vmotion. <IP address> subnet <subnet> – no default GW since not routed
• Configuration -> Memory -> Increase Service Console RAM to 800MB
• Configure Storage Paths in Active/Passive
• Reboot Host (to enact Service Console RAM changes)
• Exit Maintenance Mode

vCenter Database Server

• Manually backup VMware database

BACKUP DATABASE [VMWare] TO  DISK =  N'C:\Program Files\Microsoft SQL  Server\MSSQL.1\MSSQL\Backup\VMWare\VMWare_backup_preupgrade.bak' WITH NOFORMAT,  NOINIT,  NAME = N'VMWare-Full Database Backup', SKIP, NOREWIND, NOUNLOAD,  STATS  = 10
	GO

• Manually backup UpdateManager

BACKUP DATABASE [UpdateManager] TO  DISK =  N'C:\Program Files\Microsoft SQL  Server\MSSQL.1\MSSQL\Backup\UpdateManager\UpdateManager_backup_preupgrade.bak'  WITH NOFORMAT, NOINIT,  NAME = N'UpdateManager-Full Database Backup', SKIP,  NOREWIND, NOUNLOAD,  STATS = 10
GO

• Grant MSDB owner permissions for SQL user

USE [msdb]GO

EXEC sp_addrolemember  N'db_owner', N'USER'

GO

vCenter Server

• Log in as local administrator
• Back up the License File
  • copy "C:\Program Files\VMware\VMware License Server\Licenses\vmware.lic" \\server\share\vmware-license-backup.lic
• Mount vCenter DVD ISO
• Back up sysprep files for templates
  • copy C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\sysprep\.* \\server\share
• Run vCenter Install
• Reboot Server
• Notify users of upgrades
• Schedule times for VMware Tools Upgrades

vCenter Database Server

• Revoke MSDB owner permissions for SQL user

USE [msdb]GO
	EXEC  sp_droprolemember N'db_owner', N'USER'
        GO