williamhathaway.com

OpenSolaris.org zones-discuss forum

Zones at forums.sun.com

Zones FAQ

Jeff Victor’s blog

BigAdmin’s zones section

zonemanager – a script to automate zone creation and post-install tasks

zonestat – print zone resource usage statistics

Solaris Resouce Manager and Zones documentation at docs.sun.com

I recently ran across a storage benchmark called Blogbench and decided to give it a quick whirl on my lab machine.

My environment consists of a Sun x4150 (32G RAM and dual quad-core Xeons @ 2.93 Ghz) running Solaris 10 05/2008 with (8) 73GB 10k RPM SAS drives and an LSI SAS RAID controller with 256M of memory.

For this test I created the following 6 disk zpool:

Two options I decided to test for ZFS using BlogBench were atime and compression.  For those unfamiliar with it, the atime of a file is changed when a file is accesed.  All UNIX file systems I’m aware of have atime updating enabled by default. As a sysadmin, it can be very handy to have the atime available to see when a file was last accessed, but  atime updating can add a significant amount of overhead in some access patterns, so a lot of sites disable it on mounts that need performance. For ZFS to disable atime you use:

zfs set atime=off $datasetname

for most UNIX file system types there is noatime or similar mount option that can be used.

I ran 5 iterations of BlogBench (using ./blogbench –directory=/data/blogbench ) for each permutation of atime and compression settings. I had the script sleep for 60 seconds between runs to make sure any background activity for memory or ZFS housekeeping had finished before the next run started.  Averaging the 5 runs together for each permutation gave me the following results:

When atime was on (which is the default), there was very little difference in the non-compressed versus compressed results.  With atime disabled there was a 30-50% increase in read transactions performed and about a 250% increase in write operations.  Note that the data size of the benchmark ( ~ 3.6G) was significantly smaller than the memory on the machine (32G) , so all reads were satisfied out of file system cache.

These results are only applicable to this specific test and software/hardware, so your environment may vary significantly but I would like people to be aware of the atime setting so they can be aware of another potential knob to turn in their environment.

Adam Leventhal just blogged about a new form factor for solid state drives that will be announced soon called the ‘Open Flash Module’.  From the picture, it looks like the form-factor is based around SO-DIMM (commonly used in laptops).  I’m excited to see this since the most common form factor now is 2.5 inch or 3.5 inch drives that connect to SATA/SAS/SCSI/FC bays, which is great for compatibility (physical form-factor as well as drivers), but is a huge waste of potential density.  The new form-factor should allow you to have much greater density and boost bandwidth/reduce latency compared to current implementations.  One thing I am curious about though is the ability to hot-plug a form-factor like this which seems like it will be plugging straight in to a motherboard.

Update: Sun just posted more information about it here.  Initial capacity will be 24G with a 64M DRAMM buffer.

I haven’t updated in a while, here is the blitz:

Running – I managed to somehow get a minor muscle strain in my back which took me out of running for about 10 days (possibly by overdoing in on the Wii Fit ‘Super Hula Hoop’ game).  I came back with a 6 miler @ Conewago on Saturday with the gang, took Sunday off, ran 5.5 miles with Jon, Tex, and Cali on Monday, and then ran 3 miles with Nate tonight. The Geisinger Humdinger race is coming up in about a week and a half. Hopefully I’ll be able to get in five or six solid runs in before then.  I definitely won’t be able to beat my time from last year, but I will be happy if I am able to push hard for the entire length of the race.

Finance – mint.com has added some nice new features, including being able to track off-line assets like houses, vehicles, or just accounts that can’t be accessed automatically.  They also prettied up the user interface a bit.

Zombie Killing - I started playing the video game Left4Dead with some friends online, which has been a lot of fun.

Programming – I’ve been checking out a new Java SDK for LDAP from the Unbound ID folks.  If you have been using the ancient Netscape/Mozilla LDAP SDK before, give Unbound’s a shot, it has a lot of interesting features.

Solaris – if you are troubleshooting anything relating to tcp listen drops, check out the ndd -get /dev/tcp tcplistenhash command.   One quirk I noticed was that if you aren’t root, it has some limit to the frequency you can run the command.  I assume the command is a bit expensive and they are trying to prevent a DOS attack, but the observability is excellent.

Here is a cool video showing that the sonic vibration from yelling at a high-speed disk can cause the latency of IOs to go up.
They are using the analytics from a Sun Unified Storage Server to show the info.

Update: I did some more research on the effects of vibration on hard drive performance.  It looks like a lot of vendors are offering technology to attempt to compensate in certain drives models, although it gets tougher when you are using 15k RPM drives.  Storage Mojo has a short post from about two years ago on the topic.

I’m working on a project with a customer who is migrating from Sun’s older 5.2 based Directory Server to the latest version (6.3) of  Directory Server Enterprise Edition.  I’m excited about the project because I get to work with some good people I haven’t been in touch with lately, and we are using a number of cool technologies, including:

• Solaris 10 on x4600 servers (16 Opteron cores and 128G of memory)
  • ZFS – provides compression and snapshot capabilities on top of storage
  • DTrace – used for look at performance and troubleshooting
  • Privileges – allows our application user to start processes running on a low port
• Directory Server 6.3
  • I’m loving the dsconf tool for setting almost all config parameters via the command line.  So far there have been only two things I’ve been forced to hack via perl instead of using the supported interface to edit  (db-page-size and db-path).  This is a huge improvement over DS 5.2′s command-line manageability.
• Directory Proxy Server
  • Provides an application level proxy for load balancing LDAP requests and allowing us to perform data distribution.  This is really powerful stuff, giving us a lot more flexibility than if we just had traditional hardware load balancers at our disposal (we are still using hardware based LBs, just in front of our DPS instances).

My plan on blog entries related to this project is to not mention any customer identifying information, but to pass along some general info which might be helpful to other organizations using all or some of the components in similar ways.

Tip of the day:  We had a note from the sysadmin team asking about memory usage on one of our directory servers. There was a bit of confusion about where the memory on the box had gone since according to vmstat about 10 gigs were free and we could only see about 10 gigs used by user processes.  The answer is that since we are using ZFS, the memory consumed by ZFS’s adaptive replacement cache (ARC) doesn’t show up as free memory (unlike UFS, where it does).  This explains why systems using ZFS will have a very different memory usage profile than UFS systems.   In order to give visibility into ZFS’s memory usage, we pulled down arc_stat.pl and used that to show that ZFS was caching about 107 gigs of file system data (which is what we trying to do).  For cases like ours where you have a known memory demand for applications, you might want to consider limiting ZFS’s cache.

If you are using ZFS consider reading Ben Rockwood’s excellent posts on the topic.

Check out the whats new doc (mostly ZFS updates, but there is some other good stuff) and then download.

Note: if you want to use ZFS as your root file system, you’ll need to either run the installer in ‘console’ mode or use jumpstart.  The GUI installer still only has the UFS code.

I recently spent some time in a Sun class “Solaris System Performance Management“  and wanted to give a brief rundown on my experience.

Pros

• I think the class was helpful in incorporating some performance concepts that I knew about but hadn’t fully baked into my mental model.
• A few of the lab exercises were really good.
• I got to use SWAT (a disk and tape workload analyzer) again and it definitely seemed a smoother experience than when I used it a about a year and a half ago. I’ll definitely put it my bag of tricks.  Follow Henk’s blog to stay up to date.

Cons

• There was a lot of material to cover, especially for students without some level of a computer science background
• Some of the labs didn’t work
• The books were last updated in 2006 and while it had some modern touches (DTrace scripts were sprinkled in a few times), you could tell it wasn’t a modern course.  I think most of the material was probably written in the late 90s and incrementally updated.
• The instructor wasn’t good about letting students know what parts of the book contained practical advice that they would use frequently versus other aspects which were much more theoretical.
• The instructor wasn’t following the non-OS tools mentioned in the book.  For example, SWAT and vdbench are available to all users now (previously it was just Sun employeers and resellers).

Overall Rating
Based on my experience, I wouldn’t give a recommendation for people to take this course.  I’m sure there are some instructors that deliver more value than the one I had, but I also think there is much stronger material out there now than the aging textbook.  There was a lot of material to cover, and for people without some level of computer science background, it seemed tough.  Covering things such as the difference between a “direct-mapped” versus a “set-associative” cache seemed not to be a good use of time when there was still a lot of student misunderstandings about the basic info provided by tools such as vmstat.

What I Recommend

I think a better use of time is to get a copy of the excellent Solaris Performance and Tools book (and potentially the Solaris Internals book as well) and be prepared to spend some time reading through it and trying the tools out on a test machine that has some load on it. Also, take a gander at the Solaris Internals Wiki, which has a lot of great information about performance and tuning of Solaris for a wide variety of sub-systems and situations.  For a quick tour around Solaris 10 performance and observability, check out Jim Mauro’s latest presentation.

For the readers who are interested in Solaris/systems technical content, I have started posting that on my work blog:

http://blogs.versatile.com/systems

I stumbled across a web form that lets you easily create Solaris SMF manifest definitions.  While it only works for simple services, it is good idea since probably 90% of custom services fit this model.

Step 1. Go to the easy SMF website and put in your values

Step 2: Copy and paste the xml output into a file (a quick set of instructions for managing your new service is also provided)

Step 3: Start the service