Photo by John @ ThinkHole.com

On Tuesday night I gave a presentation on ZFS to the Central PA Linux User Group. Since the audience was a Linux user group, I wasn’t expecting too many in the crowd to be familiar with ZFS, but I was pleasantly surprised that about 40% of the ~ 20 people in attendance had used ZFS in some capacity. If you are already a seasoned ZFS user, I would highly recommend Richard Elling’s ZFS presentation which he uses in his day-long tutorials.

pkg set-publisher -O http://pkg.codenursery.com/ codenursery.com
pkg install puppet

groupadd puppet
useradd -g puppet puppet

mkdir /etc/puppet /var/puppet

/usr/ruby/1.8/sbin/puppetd  --genconfig > /etc/puppet/puppet.conf

svcadm enable puppet/master

On Monday I attended a tutorial by Richard Elling on ZFS: A Filesystem for Modern Hardware.

On Tuesday I attended two tutorials.  The first was Jacob Farmer’s Disk-to-Disk Backup and Eliminating Backup System Bottlenecks.  The second was Tom Limoncelli’s Design Patterns for System Administrators.

Unfortunately on both Monday and Tuesday I had to spend a significant amount of time on conference calls helping to troubleshoot some work related issues, but the time I spent in all 3 sessions and viewing their materials was helpful.  I would definitely recommend attending tutorials by any of the 3 people above if they are teaching a topic of interest to you.

On Tuesday night I attended some (Open)Solaris birds-of-a-feather sessions.  There were a few times that people in the crowd were being belligerent towards a speaker (mostly complaining about the difficulty of finding information of various types), even though the speaker certainly had no sway over what the person in the crowd was upset about.  I don’t care how much money your company spends with a vendor, there is never a reason to be rude.   David Miner gave a talk about whats coming in Solaris.next and Ben Rockwood gave an entertaining and informative presentation on ZFS in the Trenches.

I was lucky enough to get a chance to talk with David Miner over a quick lunch later in the week and talk about the new opportunities and challenges with the OpenSolaris installation technologies.

On Wednesday through Friday I attended a mix of presentations, met with a bunch of vendors, and also sat in some of the ‘Guru is in’ sessions and talked with a number of conference attendees.  The highlights for me were:

• Werner Vogel (CTO of Amazon) gave a fascinating talk on the history of Amazon’s IT philosophy and infrastructure and how they evolved from a humble internal IT shop to adding a business which is the dominant  cloud computing provider.
• Elizabeth Zwicky’s talk on “Searching for Truth, or at Least Data: How to Be an Empiricist Skeptic”
• Bryan Cantrill’s talk on “Visualizing DTrace: Sun Storage 7000 Analytics”
  
• Talking with the folks from Splunk (awesome log searching analysis tool)

I have a lab machine running OpenSolaris 2009.06 (updated to snv_117) and had created an LDIF file with about 100k small entries in it (file size was ~ 63 megs).  I wanted to get a count of the exact number of entries so I ran:

grep -c ^dn:

I expected it to take a second or two.  I was wrong.  It was painfully slow.

I used the time command to re-run the grep and saw it clocked in at just over a minute.

This was weird, so I though it was time to investigate further.  I used the DTrace Toolkit’s hotuser command to see what the hot functions were:

pfexec /opt/DTT/hotuser -c "grep -c ^dn: /var/tmp/search.out"
Sampling... Hit Ctrl-C to end.

FUNCTION                                                     COUNT   PCNT
...<snipped out smaller functions>...
ggrep`check_multibyte_string                                    5480   8.9%
methods_unicode.so.3`__mbrtowc_dense_utf8                      12328  20.1%
libc.so.1`mbrlen                                               13566  22.1%
libc.so.1`memset                                               23014  37.5%

Hmm, interesting to see the calls to mbrlen and methods_unicode among the hot functions.  Lets check my $LANG setting:

echo $LANG

en_US.UTF-8

Bingo!  Lets try it again with a non multi-byte LANG setting.

LANG=C time grep -c ^dn: /var/tmp/search.out

99987

real        0.1
user        0.0
sys         0.0

That looks normal.  Now lets try one more time with a multi-byte LANG to be sure:

LANG=en_US.UTF-8 time grep -c ^dn: /var/tmp/search.out
99987

real     1:01.4
user     1:01.3
sys         0.0

Yep, the problem is confirmed.

Notes

For those unfamiliar with OpenSolaris,  the default path has /usr/gnu/bin first.  The grep I was using was:

grep -V
grep (GNU grep) 2.5

Copyright 1988, 1992-1999, 2000, 2001 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

If you use the non-GNU grep available at /usr/xpg4/bin/grep it doesn’t have the big slowdown regardless of the LANG.

I also tried the same test on the GNU wc command and saw about a 25x difference when using a multi-byte LANG.

For both grep and wc, I re-ran the tests multiple times to make sure that file system caching played no role in the results.

I think these performance differences are way higher than they should be, I’m going to dig further when I have a chance.

-XX:+PrintGCTimeStamps

this option tells the JVM to print a time stamp in the front of each GC line that is the number of seconds since the Java process started.  You end up with lines like:

1.449: [GC 76756K->5892K(504320K), 0.0226648 secs]
1.472: [Full GC 5892K->5752K(504320K), 0.1099638 secs]
4.276: [GC 137848K->14895K(504320K), 0.0333021 secs]
5.653: [GC 146991K->63707K(504320K), 0.0858191 secs]
38.536: [GC 195803K->77393K(636416K), 0.0683676 secs]
144.875: [GC 341585K->97580K(637440K), 0.1131954 secs]

While these are better than nothing, it can be a PITA to translate these to something useful, like say wall clock time, such as when you want to correlate a GC time to an external alarm.

Starting in Sun’s JDK version 1.6u4, there is another option that can be enabled that actually prints both the wall clock time as well as the seconds since process start.  This is way more user-friendly.  I suspect there is a extremely minute performance penalty to add this extra info, but I’d be shocked if it is actually perceptable.  If anyone knows differently, please let me know.  This option is:

-XX:+PrintGCDateStamps

If you have it enabled, the output looks like:

2009-09-02T23:02:07.151-0400: 1.463: [GC 79398K->5887K(504320K), 0.0236701 secs]
2009-09-02T23:02:07.175-0400: 1.486: [Full GC 5887K->5753K(504320K), 0.1095894 secs]
2009-09-02T23:02:10.028-0400: 4.339: [GC 137849K->14987K(504320K), 0.0345331 secs]
2009-09-02T23:04:31.707-0400: 146.018: [GC 147083K->23143K(504320K), 0.0455040 secs]
2009-09-02T23:14:28.502-0400: 742.813: [GC 155239K->28406K(504320K), 0.0912850 secs]

As you can see, this is much more easily digestable to  humans.  Since you still have the ‘seconds since program start’ info handy, you can continue to use scripts/programs to easily calculate the time deltas between GCs.

Some useful links I’ve found for Java tuning in general are:

• Resources from Matty’s presentation:

http://prefetch.net/blog/index.php/2008/02/05/java-performance-presentation/

• Java performance forum:

http://forums.java.net/jive/forum.jspa?forumID=60

• Jon Masamitsu’s blog: (the last post was over a year ago, starting to age, but some good material)

http://blogs.sun.com/jonthecollector/

• Garbage Collection Tuning in the Java HotSpot Virtual Machine presentation at Java One 2009

http://developers.sun.com/learning/javaoneonline/sessions/2009/pdf/TS-4887.pdf

The first thing I did was read through the automated installer docs.  There isn’t a lot there yet, so it is a quick read, but it will help you get the basics.  Another good place to look for information is the OpenSolaris forum for installer technology (aka project Caiman).

There appear to be several components involved, at least for x86 based clients.  I haven’t yet tried SPARC so am not sure how it differs.

1) DHCP server – to hand out an address and the PXE boot parameters to a client

2) TFTP server – to serve the PXE boot image

3) Install server – an Apache instance that hands back the XML configuration files and the mini root.  In my case it was running on port 5555.

4) Package repository – for fetching the actual packages to install. By default it is pkg.opensolaris.org/release, but you could change it to a different repository (including a mirror hosted locally if you had one).

Note that there is no NFS service needed, this should make firewall admins very happy.

The lab

I built a lab environment consisting of two virtual machines inside VMWare on my desktop.

To keep things simple, the first VM was called “server”, and the second “client”.  The purpose of my lab environment was to configure the AI environment on the server machine and complete a hands-off install on the client machine.

The VMs were configured as follows:

Server

• RAM – 800M
• Disk – 16GB
• NIC1 (e1000g0) – bridged to public network
• NIC2 (e1000g1) – host-internal network

I also went into the VMWare networking tool and disabled VMWare’s built-in DHCP server on the host-internal network to ensure that my server would be handing out any DHCP responses.

Client

• RAM 800M
• NIC1 (e1000g0) – host-internal network
• Disk – 8GB   Note: when I first tried an 8 GB disk  AI complained that it couldn’t find any suitable disks because  it wanted at least a 12.5GB disk.  You can work around it by explicitly specifying which disk you want to install on, in which case the default minimum size limit won’t be  triggered.

OpenSolaris Auto Installer Lab

Setting up the server

1) Installed OpenSolaris 2009.06

2) Installed the automated install software

I saw from the docs that I needed the installadm utility, which wasn’t on my system.  I wasn’t sure which package this was from, so I ran:

pkg  search installadm

this told me I wanted the SUNWinstalladm-tools package.  I installed that using:

pfexec pkg install SUNWinstalladm-tools

3) Download the automated install ISO image

The installer needs an architecture (x86 or SPARC) specific ISO image for each type of client that will be supported.  Since I was going to install on x86, I downloaded the appropriate image from genunix: http://genunix.org/distributions/indiana/osol-0906-ai-x86.iso

4) Create an install environment under /auto-install named ai-x64 on the 192.168.72.0 network starting at .10 and using 5 addresses

pfexec installadm create-service -n ai-x64 -i 192.168.72.10 -c 5 -s /export/home/wdh/Downloads/osol-0906-ai-x86.iso /auto-install

5) Configure dhcpd to run on the appropriate interface

The installadm command configured dhcpd, but it was running on the e1000g0 interface by default.  For my environment, I needed to switch that to e1000g1 so it would see the requests from the client VM.

pfexec dhcpconfig -P INTERFACES=e1000g1
svcadm disable dhcp-server
svcadm enable dhcp-server

6) Install squid

We will need squid (or some other proxy) since we aren’t running a local repository server and the client machine will need to be able to fetch packages from pkg.opensolaris.org.  We will tell the client machine to use the proxy on the server.

pkg search squid

figure out the package name I am looking for is SUNWsquid

pfexec pkg install SUNWsquid

svcadm enable http:squid

I was pleasantly surprised how easy that was.  If you are on a non-NATed network, you will likely need to edit the squid configuration file to allow access to your clients.

7) Customize the default AI manifest (I’ll call mine ai_proxy.xml)

cd /auto-install/auto_install

make a copy of the default manifest and name it something more specific

pfexec cp default.xml ai_proxy.xml

added <ai_target_device><target_device_name>c8t0d0</target_device_name> </ai_target_device>

so I could use a disk that was smaller then the auto-installer default

added  <ai_http_proxy url=”http://192.168.72.2:3128″/> so it would use the proxy and be able to reach the internet

changed the ai_auto_reboot setting to true, and changed the default user and password from jack to my normal values.

ran installadm to let the AI service know it should use the custom version of the file

pfexec /usr/sbin/installadm add -m ai_proxy.xml -n ai-x64

8) Register the target system as a client

Started the client virtual machine and retrieved the MAC address (  00:0c:29:b6:43:bf )

On the server use installadm to register the client

pfexec installadm create-client -e 00:0c:29:b6:43:bf -t /auto-install -n ai-x64

9) Started the client system in network boot mode

The install succeeded, but it took about 1.5 hours.  I suspect if I had a local repository and was installing on a non-emulated hard disk it would have gone substantially faster.

Overall thoughts

I was happy that it was relatively straightforward to get working, but I think it will be a while before the system has as much flexibility for customizing installs as Jumpstart.  Based on all the traffic I see on the forum, it seems like the AI project has a lot of momentum behind it, so I am looking forward to giving another spin in a few months.  I’d also like to try this with a local mirror of the pkg repository and see how quick the installer will run.

Update on June 24th

I saw this morning that a functional spec for the AI client has been submitted and the project team is asking for comments.  Please read the thread/document and give any feedback you might have.

# pkgrm SUNWfontconfig-root

It was returning:

## Removing pathnames in class <manifest>
svc:/application/font/fc-cache:default remains enabled; aborting
pkgrm: ERROR: class action script did not complete successfully

Removal of <SUNWfontconfig-root> failed.

To troubleshoot it, we added the -v option to pkgrm so it would show the scripts it was running:

# pkgrm -v SUNWfontconfig-root

+ [  !=  -a  != / ]
+ [ -r /etc/svc/volatile/repository_door ]
smf_alive=yes
SVCPROP=/usr/bin/svcprop
+ read mfst
+ [ yes = yes ]
+ /usr/sbin/svccfg inventory /var/svc/manifest/application/font/fc-cache.xml
ENTITIES=svc:/application/font/fc-cache:default
svc:/application/font/fc-cache
+ /usr/bin/svcprop -p general/enabled svc:/application/font/fc-cache:default
en_p=true
+ /usr/bin/svcprop -p general_ovr/enabled svc:/application/font/fc-cache:default
en_o=
+ [ true = true -o  = true ]
+ echo svc:/application/font/fc-cache:default remains enabled; aborting
svc:/application/font/fc-cache:default remains enabled; aborting
+ exit 1
pkgrm: ERROR: class action script did not complete successfully

Ok, now we can see the problem.  We need to change the general/enabled property of the service to false and refresh the service.

# svccfg -s   svc:/application/font/fc-cache:default setprop general/enabled = false

# svcadm refresh font/fc-cache:default

Verify that it took:

# /usr/bin/svcprop -p general/enabled svc:/application/font/fc-cache:default

false

That looks good.  Now we can retry the package removal:

# pkgrm SUNWfontconfig-root

and it worked!

pkg install SUNWmemcached

and then tried to get it running with:

svcadm enable memcached

It kept on dieing and respawning and I saw the log file (/var/svc/log/application-database-memcached\:default.log) growing with lines like the following

[ Jun  1 14:42:34 Enabled. ]
[ Jun  1 14:42:34 Executing start method ("/lib/svc/method/memcached start"). ]
can’t run as root without the -u switch
[ Jun  1 14:42:35 Method "start" exited with status 0. ]

So basically memcached was complaining that it was starting as root and not being told to switch to another user.  To fix this I had to tell it to switch to the ‘nobody’ user when starting.  Here are the steps I used:

# tell memcached to run as the user ‘nobody’ and set the max memory to 1024M, you need to at least have

# -u nobody (or some other account that exists on the system)

svccfg -s memcached setprop memcached/options= ‘(“-u” “nobody” “-m” “1024″)’

svcadm refresh memcached

svcadm disable memcached

svcadm enable memcached

The memcached man page in OpenSolaris sort of mentions you need to do this towards the end, but I think it is poor form for the server to be unable to run by using the default SMF configuration.  I will try to get an RFE to at least have the ‘-u nobody’ option set by default.

While there is always room for improvement, I think given OpenSolaris’ design, feature set,  and maturity it is now in a place where I’d consider it a viable option for production deployments on x64 systems.  I’d still hold off for a little while on SPARC since I think it may take a bit for all the auto-install and boot-related code to gain maturity there.

Here is what I did:

create a ZFS file system mounted at /zones

# zfs create -o mountpoint=/zones -o compression=on -o atime=off data/zones

create a zone ‘goldenzone’ that I will use as a target for cloning.  If I wanted to add certain software or other setup (like security hardening) to all zones I would do that here.

# zonecfg -z goldenzone

goldenzone: No such zone configured
Use ‘create’ to begin configuring a new zone.

zonecfg:goldenzone> create -b

zonecfg:goldenzone> set zonepath=/zones/goldenzone

zonecfg:goldenzone> exit

# zoneadm -z goldenzone install

Creating list of files to copy from the global zone.

Copying <130143> files to the zone …

# zoneadm -z goldenzone boot (so we get the SMF importing out of the way)

# zlogin -C goldenzone (verify the SMF import finishes and answer sysid questions)

This is also a good time to do any customization steps that you would like done everywhere, such as disabling services, changing root’s shell to /bin/bash, using a custom apache configuration, etc.

# zoneadm -z goldenzone halt

Now we will create a new full-root zone ‘zone1′ that we will clone from goldenzone

# zonecfg -z zone1

zonecfg:zone1> create -b

zonecfg:zone1> set zonepath=/zones/zone1

zonecfg:zone1> add net

zonecfg:zone1:net> set physical=e1000g0

zonecfg:zone1:net> set address=zone1/24

zonecfg:zone1:net> end

zonecfg:zone1:> exit

Now that we have finished the main configuration, lets see how long it takes to make the new zone:

time zoneadm -z zone1 clone goldenzone
Cloning snapshot data/zones/goldenzone@SUNWzone1
Instead of copying, a ZFS clone has been created for this zone.
grep: can’t open /a/etc/dumpadm.conf

real    0m4.088s

user    0m0.439s

sys    0m0.187s

We will also take a look at the space used.  The goldenzone took 1.7GB (ZFS compression was on) and we can see that the snapshot & clone used for zone1 took us less than a meg of extra space.  Sweet!

NAME                              USED  AVAIL  REFER  MOUNTPOINT
data/zones                             1.70G   263G  42.0K  /zones
data/zones/goldenzone        1.70G   263G  1.70G  /zones/goldenzone
data/zones/goldenzone@SUNWzone1   405K      -  1.70G  -
data/zones/zone1                  285K   263G  1.70G  /zones/zone1