williamhathaway.com

I have a lab machine running OpenSolaris 2009.06 (updated to snv_117) and had created an LDIF file with about 100k small entries in it (file size was ~ 63 megs).  I wanted to get a count of the exact number of entries so I ran:

grep -c ^dn:

I expected it to take a second or two.  I was wrong.  It was painfully slow.

I used the time command to re-run the grep and saw it clocked in at just over a minute.

This was weird, so I though it was time to investigate further.  I used the DTrace Toolkit’s hotuser command to see what the hot functions were:

pfexec /opt/DTT/hotuser -c "grep -c ^dn: /var/tmp/search.out"
Sampling... Hit Ctrl-C to end.

FUNCTION                                                     COUNT   PCNT
...<snipped out smaller functions>...
ggrep`check_multibyte_string                                    5480   8.9%
methods_unicode.so.3`__mbrtowc_dense_utf8                      12328  20.1%
libc.so.1`mbrlen                                               13566  22.1%
libc.so.1`memset                                               23014  37.5%

Hmm, interesting to see the calls to mbrlen and methods_unicode among the hot functions.  Lets check my $LANG setting:

echo $LANG

en_US.UTF-8

Bingo!  Lets try it again with a non multi-byte LANG setting.

LANG=C time grep -c ^dn: /var/tmp/search.out

99987

real        0.1
user        0.0
sys         0.0

That looks normal.  Now lets try one more time with a multi-byte LANG to be sure:

LANG=en_US.UTF-8 time grep -c ^dn: /var/tmp/search.out
99987

real     1:01.4
user     1:01.3
sys         0.0

Yep, the problem is confirmed.

Notes

For those unfamiliar with OpenSolaris,  the default path has /usr/gnu/bin first.  The grep I was using was:

grep -V
grep (GNU grep) 2.5

Copyright 1988, 1992-1999, 2000, 2001 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

If you use the non-GNU grep available at /usr/xpg4/bin/grep it doesn’t have the big slowdown regardless of the LANG.

I also tried the same test on the GNU wc command and saw about a 25x difference when using a multi-byte LANG.

For both grep and wc, I re-ran the tests multiple times to make sure that file system caching played no role in the results.

I think these performance differences are way higher than they should be, I’m going to dig further when I have a chance.

I need to upload  more, but you can see the pics at Smugmug.

I got a new camera this week and went up to  Wildwood Park near HACC to take some pictures today.   I really like the 20x zoom lense (the main reason I bought it), and the image stabilization seemed to work pretty well.   I think it is cool that I was able to see the details of the pollen on the bee.

Bee on a flower

Goose swimming in green alge

We all live downstream

Overrun lake

Main trail

My wife and I have decided to move to the San Francisco/SiliconValley area.  We haven’t set a date yet,  but we are definitely moving sometime in 2010.

Just wanting google to start indexing my wiki.

I’ve had my Prius for about 16 months now (got it in early May 2008).  So far I have been largely happy.  I’ve averaged about 44-47 MPG in the warmer months and 42-44MPG in the winter with a roughly  50% highway/50% city mix of driving.  I’ve only had a few  minor nits/issues:

1. The gas tank is made of a (sort of) flexible membrane and it becomes stiff during colder months and won’t hold as much fuel.  During the winter I can often only get about 75% as much gas in the tank as I can during the spring/summer/fall months.   More info available in a hybrid cars forum.
2. The default behavior is to beep inside the car when in reverse (versus trucks which beep outside to warn people behind it).  It turns out it is fairly easy to disable the beeping using these instructions.  I just followed the instructions this morning.
3. It is difficult to jump start other cars.  The positive terminal is really small and when I tried to help a friend get his car started, the metal “jaw” on the jumper cables were too big to fit in the space near the positive terminal.
4. I wish there was better iPod integration by default (I am currently just using the AUX jack).  A friend of mine purchased the factory unit for his Lexus and the interface is dissapointing.  I am considering getting a Vaistech unit.

Other notes:

1. I’ve had no maintenance problems.
2. The leg/headroom seem fine.  While I am not tall, I regular drive 3 other people to lunch, and the bigger guys in the office can fit comfortably (although the 6’6″ guy does always take shotgun).
3. By folding down the back seats I can easily put my mountain bike in the trunk if I take off the front tire.

I’ve been spending a bunch of time lately looking at Java garbage collection (GC) performance & tuning.  One of the old standby options for GC logging is:

-XX:+PrintGCTimeStamps

this option tells the JVM to print a time stamp in the front of each GC line that is the number of seconds since the Java process started.  You end up with lines like:

1.449: [GC 76756K->5892K(504320K), 0.0226648 secs]
1.472: [Full GC 5892K->5752K(504320K), 0.1099638 secs]
4.276: [GC 137848K->14895K(504320K), 0.0333021 secs]
5.653: [GC 146991K->63707K(504320K), 0.0858191 secs]
38.536: [GC 195803K->77393K(636416K), 0.0683676 secs]
144.875: [GC 341585K->97580K(637440K), 0.1131954 secs]

While these are better than nothing, it can be a PITA to translate these to something useful, like say wall clock time, such as when you want to correlate a GC time to an external alarm.

Starting in Sun’s JDK version 1.6u4, there is another option that can be enabled that actually prints both the wall clock time as well as the seconds since process start.  This is way more user-friendly.  I suspect there is a extremely minute performance penalty to add this extra info, but I’d be shocked if it is actually perceptable.  If anyone knows differently, please let me know.  This option is:

-XX:+PrintGCDateStamps

If you have it enabled, the output looks like:

2009-09-02T23:02:07.151-0400: 1.463: [GC 79398K->5887K(504320K), 0.0236701 secs]
2009-09-02T23:02:07.175-0400: 1.486: [Full GC 5887K->5753K(504320K), 0.1095894 secs]
2009-09-02T23:02:10.028-0400: 4.339: [GC 137849K->14987K(504320K), 0.0345331 secs]
2009-09-02T23:04:31.707-0400: 146.018: [GC 147083K->23143K(504320K), 0.0455040 secs]
2009-09-02T23:14:28.502-0400: 742.813: [GC 155239K->28406K(504320K), 0.0912850 secs]

As you can see, this is much more easily digestable to  humans.  Since you still have the ‘seconds since program start’ info handy, you can continue to use scripts/programs to easily calculate the time deltas between GCs.

Some useful links I’ve found for Java tuning in general are:

• Resources from Matty’s presentation:

http://prefetch.net/blog/index.php/2008/02/05/java-performance-presentation/

• Java performance forum:

http://forums.java.net/jive/forum.jspa?forumID=60

• Jon Masamitsu’s blog: (the last post was over a year ago, starting to age, but some good material)

http://blogs.sun.com/jonthecollector/

• Garbage Collection Tuning in the Java HotSpot Virtual Machine presentation at Java One 2009

http://developers.sun.com/learning/javaoneonline/sessions/2009/pdf/TS-4887.pdf

30 second summary for those that don’t want to read the troubleshooting details

When using replicated LDAP servers in a Sun Communications Suite deployment, it is important that every connection from a given Convergence (webmail component) instance go to the same LDAP server ,otherwise address book creation can partially fail causing some user logins to webmail to hang.  To fix this, use one of the following techniques:

1) Configure Convergence’s application level failover to point to individual LDAP servers (be sure to switch the host order on alternating Convergence instances to spread the load)

/opt/sun/comms/iwc/sbin/iwcadmin -u admin -W pwdfile  -o ugldap.host -v ldap1:$port ,ldap2:$port

(you will also need to restart the web container for this to take effect)

2) Use Directory Proxy Server to route writes to a preferred master

3) If pointing at a HW load balancer virtual IP, use a distribution algorithm that has backend server persistence based on originating IP.  Note that with a few machines this might not actually balance out well, so verify you aren’t overloading one LDAP instance.

Background

A customer of mine is deploying Sun’s Communications Suite (aka Messaging, Calendar, and IM servers) and was testing their custom provisioning tool.  A few accounts had been created that worked fine but one of the accounts would just hang when trying to login to webmail.  The screen would show the application initialization progress bar stuck at 84% and indicate it was dealing with the address book.

I verified that the account would hang and then took a look at the account’s main LDAP entry, which looked fine.  I then checked the account’s LDAP address book data.  The bad account had 3 LDAP entries in the address book branch, but a good account should have 4. Taking a look at the iwc.log from Convergence, I could see an error:

ADDRESS_BOOK: ERROR from com.sun.comms.client.ab.wabp.WABPEngineServlet  Thread httpSSLWorkerThread-80-4 at 2009-06-27 14:33:05,586 – pstore object couldn’t be created for user :baduser

At this site we have a pair of replicated LDAP servers behind a pair of load balancers that are used by the messaging components. Sun’s Directory Server has a loose replication model that usually works fine, but you can run into a rare race condition when applications are adding inter-related entries to different masters in a rapid fire succession.  Convergence was initially pointing at the load balancer virtual IP to reach the LDAP servers.

When I checked the logs on the LDAP servers, I could see that  Convergence had tried to create address book entries when the user first logged in, but had done so over several different LDAP connections which via the load balancer went to different LDAP servers. It created a parent entry on ldap1, then in a connection to ldap2 tried to create a dependent entry, which failed.  Convergence then created  another version of the parent entry on ldap2 (which worked, but caused a replication conflict).  Later attempts to login ended up adding some dependent entries, but it was still in an usuable state.

When things are working correctly, you will see a LDAP operation pattern that looks like:

[27/Jun/2009:16:21:39 -0400] conn=566625 op=5 msgId=948 – ADD dn=”piPStoreOwner=$user,o=$domain,o=PiServerDb”
[27/Jun/2009:16:21:39 -0400] conn=566636 op=1 msgId=950 – ADD dn=”piEntryID=random1,piPStoreOwner=$user,o=$domain,o=PiServerDb”
[27/Jun/2009:16:21:39 -0400] conn=566636 op=2 msgId=951 – ADD dn=”piEntryID=random2,piPStoreOwner=$user,o=$domain,o=PiServerDb”
[27/Jun/2009:16:21:39 -0400] conn=566636 op=3 msgId=952 – ADD dn=”piEntryID=random3,piPStoreOwner=$user,o=$domain,o=PiServerDb”

The fix

In order to fix the account and the problem in general, we ended up deleting the skeleton address book entries for the user in question and used iwcadmin to change Convergence to point to individual LDAP servers in a failover mode. Since we had two Convergence instances and two LDAP instances, it was easy to flip the perferred order so that LDAP load will be well-balanced.

Things that could be improved

1) Convergence could give a better error experience to the user instead of a just hanging.  Perhaps timing out after 30 seconds with a message “There is a problem with initializing your address book, please ask your administrator to investigate”.

2) Convergence could use a single LDAP connection when performing address book creation for any given user

3) Sun’s Directory Server could have an assured replication model (this is available in the OpenDS 2.0 release candidates)

I took a test drive of the OpenSolaris automated installer (AI) utility today.  This is the replacement for the venerable Solaris jumpstart technology and is the only way to install OpenSolaris in a hands-off approach.  Based off my 2 hours of so of perusing the documentation and working with it, I think it is still a work in progress (e.g. I didn’t see any way of having jumpstart-like custom finish scripts).

The first thing I did was read through the automated installer docs.  There isn’t a lot there yet, so it is a quick read, but it will help you get the basics.  Another good place to look for information is the OpenSolaris forum for installer technology (aka project Caiman).

There appear to be several components involved, at least for x86 based clients.  I haven’t yet tried SPARC so am not sure how it differs.

1) DHCP server – to hand out an address and the PXE boot parameters to a client

2) TFTP server – to serve the PXE boot image

3) Install server – an Apache instance that hands back the XML configuration files and the mini root.  In my case it was running on port 5555.

4) Package repository – for fetching the actual packages to install. By default it is pkg.opensolaris.org/release, but you could change it to a different repository (including a mirror hosted locally if you had one).

Note that there is no NFS service needed, this should make firewall admins very happy.

The lab

I built a lab environment consisting of two virtual machines inside VMWare on my desktop.

To keep things simple, the first VM was called “server”, and the second “client”.  The purpose of my lab environment was to configure the AI environment on the server machine and complete a hands-off install on the client machine.

The VMs were configured as follows:

Server

• RAM – 800M
• Disk – 16GB
• NIC1 (e1000g0) – bridged to public network
• NIC2 (e1000g1) – host-internal network

I also went into the VMWare networking tool and disabled VMWare’s built-in DHCP server on the host-internal network to ensure that my server would be handing out any DHCP responses.

Client

• RAM 800M
• NIC1 (e1000g0) – host-internal network
• Disk – 8GB   Note: when I first tried an 8 GB disk  AI complained that it couldn’t find any suitable disks because  it wanted at least a 12.5GB disk.  You can work around it by explicitly specifying which disk you want to install on, in which case the default minimum size limit won’t be  triggered.

OpenSolaris Auto Installer Lab

Setting up the server

1) Installed OpenSolaris 2009.06

2) Installed the automated install software

I saw from the docs that I needed the installadm utility, which wasn’t on my system.  I wasn’t sure which package this was from, so I ran:

pkg  search installadm

this told me I wanted the SUNWinstalladm-tools package.  I installed that using:

pfexec pkg install SUNWinstalladm-tools

3) Download the automated install ISO image

The installer needs an architecture (x86 or SPARC) specific ISO image for each type of client that will be supported.  Since I was going to install on x86, I downloaded the appropriate image from genunix: http://genunix.org/distributions/indiana/osol-0906-ai-x86.iso

4) Create an install environment under /auto-install named ai-x64 on the 192.168.72.0 network starting at .10 and using 5 addresses

pfexec installadm create-service -n ai-x64 -i 192.168.72.10 -c 5 -s /export/home/wdh/Downloads/osol-0906-ai-x86.iso /auto-install

5) Configure dhcpd to run on the appropriate interface

The installadm command configured dhcpd, but it was running on the e1000g0 interface by default.  For my environment, I needed to switch that to e1000g1 so it would see the requests from the client VM.

pfexec dhcpconfig -P INTERFACES=e1000g1
svcadm disable dhcp-server
svcadm enable dhcp-server

6) Install squid

We will need squid (or some other proxy) since we aren’t running a local repository server and the client machine will need to be able to fetch packages from pkg.opensolaris.org.  We will tell the client machine to use the proxy on the server.

pkg search squid

figure out the package name I am looking for is SUNWsquid

pfexec pkg install SUNWsquid

svcadm enable http:squid

I was pleasantly surprised how easy that was.  If you are on a non-NATed network, you will likely need to edit the squid configuration file to allow access to your clients.

7) Customize the default AI manifest (I’ll call mine ai_proxy.xml)

cd /auto-install/auto_install

make a copy of the default manifest and name it something more specific

pfexec cp default.xml ai_proxy.xml

added <ai_target_device><target_device_name>c8t0d0</target_device_name> </ai_target_device>

so I could use a disk that was smaller then the auto-installer default

added  <ai_http_proxy url=”http://192.168.72.2:3128″/> so it would use the proxy and be able to reach the internet

changed the ai_auto_reboot setting to true, and changed the default user and password from jack to my normal values.

ran installadm to let the AI service know it should use the custom version of the file

pfexec /usr/sbin/installadm add -m ai_proxy.xml -n ai-x64

8) Register the target system as a client

Started the client virtual machine and retrieved the MAC address (  00:0c:29:b6:43:bf )

On the server use installadm to register the client

pfexec installadm create-client -e 00:0c:29:b6:43:bf -t /auto-install -n ai-x64

9) Started the client system in network boot mode

The install succeeded, but it took about 1.5 hours.  I suspect if I had a local repository and was installing on a non-emulated hard disk it would have gone substantially faster.

Overall thoughts

I was happy that it was relatively straightforward to get working, but I think it will be a while before the system has as much flexibility for customizing installs as Jumpstart.  Based on all the traffic I see on the forum, it seems like the AI project has a lot of momentum behind it, so I am looking forward to giving another spin in a few months.  I’d also like to try this with a local mirror of the pkg repository and see how quick the installer will run.

Update on June 24th

I saw this morning that a functional spec for the AI client has been submitted and the project team is asking for comments.  Please read the thread/document and give any feedback you might have.

A lot of times I hear “don’t worry, we have support for product X” when talking to customers and I find a lot of people consider the ability to get support for a product to be more of a  yes/no checkbox versus a what I feel tends to be more a continuum.

To start off with, I’d like to layout my primary definition of product support as “the ability to resolve problems encountered with the product”.  I know some people may add “the ability to get new versions”, but I’m putting that to the side for now, since that isn’t a very nuanced problem (essentially solved with $ alone for a support agreement if commercial-only software).  I also know some people just use support as “someone to blame so I can slip out of all responsibility despite my job”, but I’ll be ignoring those folks.

For hardware products where you might need to get a replacement for failing disks, motherboards, etc, having support can certainly be a yes/no type answer (although another option might be just buying some extra components that you know are likely need replacing).

For software, besides the standard vendor support contracts available, I think of the following aspects of support:

• internal employees/contractors – their knowledge of the overall problem space as well as the specific products used.
• software maturity – the important thing here isn’t just how long it has been around, but how widely used is the functionality you want to implement.
• platform choice – this doesn’t matter for some generic Java services, but if the product is a compiled program that interacts with the operating system a lot,  make sure there are enough other customers using the same operating system as you.  I’ve seen cases where a product was extremely widely used, but if you were one of the handful running the product on an unpopular platform (e.g. AIX), your chance of having an undiscovered problem and your time to problem resolution was on average much higher than if you were using one of the more popular platforms.  If you can’t find much evidence of people running a product on your platform of choice, consider choosing a different product or if the specific product is key to your organization, consider running it on the vendor’s platform of choice.
• community – are there active mailing lists/forums/blogs/irc channels for people using this product?
• documentation (both official and unofficial) – how comprehensive in breadth and depth?
• boutique consultants – smaller organizations like Percona (MySQL) or people like Richard Elling (ZFS/Storage) may be able to provide expert assistance in a targeted and quick manner, often going much deeper than typical vendor support and much faster to engage than  larger professional service organizations

I think your real support level is made up by all the factors above and they should be considered when evaluating a product’s overall suitability for your organization, don’t think of it just as a checkbox.