williamhathaway.com

I’m working on a project with a customer who is migrating from Sun’s older 5.2 based Directory Server to the latest version (6.3) of  Directory Server Enterprise Edition.  I’m excited about the project because I get to work with some good people I haven’t been in touch with lately, and we are using a number of cool technologies, including:

• Solaris 10 on x4600 servers (16 Opteron cores and 128G of memory)
  • ZFS – provides compression and snapshot capabilities on top of storage
  • DTrace – used for look at performance and troubleshooting
  • Privileges – allows our application user to start processes running on a low port
• Directory Server 6.3
  • I’m loving the dsconf tool for setting almost all config parameters via the command line.  So far there have been only two things I’ve been forced to hack via perl instead of using the supported interface to edit  (db-page-size and db-path).  This is a huge improvement over DS 5.2′s command-line manageability.
• Directory Proxy Server
  • Provides an application level proxy for load balancing LDAP requests and allowing us to perform data distribution.  This is really powerful stuff, giving us a lot more flexibility than if we just had traditional hardware load balancers at our disposal (we are still using hardware based LBs, just in front of our DPS instances).

My plan on blog entries related to this project is to not mention any customer identifying information, but to pass along some general info which might be helpful to other organizations using all or some of the components in similar ways.

Tip of the day:  We had a note from the sysadmin team asking about memory usage on one of our directory servers. There was a bit of confusion about where the memory on the box had gone since according to vmstat about 10 gigs were free and we could only see about 10 gigs used by user processes.  The answer is that since we are using ZFS, the memory consumed by ZFS’s adaptive replacement cache (ARC) doesn’t show up as free memory (unlike UFS, where it does).  This explains why systems using ZFS will have a very different memory usage profile than UFS systems.   In order to give visibility into ZFS’s memory usage, we pulled down arc_stat.pl and used that to show that ZFS was caching about 107 gigs of file system data (which is what we trying to do).  For cases like ours where you have a known memory demand for applications, you might want to consider limiting ZFS’s cache.

If you are using ZFS consider reading Ben Rockwood’s excellent posts on the topic.

I’ve been following the (sometimes controversial) OpenDS project for a while now.  This morning I woke up a bit earlier than usual so I decided to try out the latest build.

It is very straightforward to use, and took about two minutes to download, extract, setup, and start running.  I needed a resonable number of entries to test with, so I created a million synthetic users (via makeldif) and then kicked off the import process (which ran at about 3-4k a second).  After the import finished, I ran a modrate job and forgot about it.  When I returned home this evening, I saw in an open SSH session that it was still crunching away, averaging about 7,000 modifications a second.  I accidentally performed a 13 hour stress test and it held up just fine (after approximately 327 million mods!).

This was on a 2 processor quad-core Sun x4150 running Solaris 10.  No settings were changed from the default.

./modrate  -p 1389 -D cn=directory\ manager -w password -b uid=%d,ou=people,dc=example,dc=com -M ‘description:60:[a-z]‘ -r 1000000 -K -t 16

….
(this is the output I came home to)
Avg r=1955.44/thr (6257.40/sec), total= 31287
Avg r=2005.94/thr (6419.00/sec), total= 32095
Avg r=2035.00/thr (6512.00/sec), total= 32560
Avg r=2040.62/thr (6530.00/sec), total= 32650
Avg r=2013.12/thr (6442.00/sec), total= 32210
Avg r=2301.25/thr (7364.00/sec), total= 36820
Avg r=2397.00/thr (7670.40/sec), total= 38352
Avg r=2443.69/thr (7819.80/sec), total= 39099
Avg r=2748.69/thr (8795.80/sec), total= 43979
Avg r=2899.12/thr (9277.20/sec), total= 46386
Avg r=2475.25/thr (7920.80/sec), total= 39604
Avg r=2333.06/thr (7465.80/sec), total= 37329
Avg r=2673.19/thr (8554.20/sec), total= 42771
Avg r=2652.88/thr (8489.20/sec), total= 42446
Avg r=2371.06/thr (7587.40/sec), total= 37937
Avg r=2065.69/thr (6610.20/sec), total= 33051
Avg r=2228.38/thr (7130.80/sec), total= 35654
Avg r=2326.81/thr (7445.80/sec), total= 37229
^C

Not shabby at all!

I was working with dsadm on a Sun Directory Server 6.2 install yesterday and ran into a snag with using the cert-export/cert-import functionality. Here are my notes:

1) When using the dsadm cert-export command, you provide a cert alias name, but it actually exports all your certificates.

2) When using dsadm cert-import, I had a problem where it re-imported the serverCert and caCert but the trust permissions on caCert weren’t correct. This caused outbound SSL connections to fail since it wasn’t able to validate the other server’s certificate. (this scenario was for an in-house certificate authority, so it may not apply for people using certificates issued by Verisign and the like)

You can read about the NSS trust flags at the certutil man page.

Steps to show/fix the caCert trust issue (using the certutil located in /usr/sfw/bin on Solaris 10)

——————————————————————————————————-

# look at the trust settings for caCert, which should be C,,

/var/directory/master1/alias> certutil -L -d . -P slapd-
defaultCert CTu,u,u
serverCert u,u,u
caCert c,c,

# modify the trust settings to the right values

/var/directory/master1/alias> certutil -M -n caCert -t C,, -d . -P slapd-

# list the trust settings again, now they are correct

/var/directory/master1/alias> certutil -L -d . -P slapd-
defaultCert CTu,u,u
serverCert u,u,u
caCert C,,