williamhathaway.com

I spent last week at the LISA Conference in Baltimore MD.  if you aren’t familiar with LISA, it is a conference focused on system administration.  This is the 4th  LISA I’ve attended in the last 12 years.

On Monday I attended a tutorial by Richard Elling on ZFS: A Filesystem for Modern Hardware.

On Tuesday I attended two tutorials.  The first was Jacob Farmer’s Disk-to-Disk Backup and Eliminating Backup System Bottlenecks.  The second was Tom Limoncelli’s Design Patterns for System Administrators.

Unfortunately on both Monday and Tuesday I had to spend a significant amount of time on conference calls helping to troubleshoot some work related issues, but the time I spent in all 3 sessions and viewing their materials was helpful.  I would definitely recommend attending tutorials by any of the 3 people above if they are teaching a topic of interest to you.

On Tuesday night I attended some (Open)Solaris birds-of-a-feather sessions.  There were a few times that people in the crowd were being belligerent towards a speaker (mostly complaining about the difficulty of finding information of various types), even though the speaker certainly had no sway over what the person in the crowd was upset about.  I don’t care how much money your company spends with a vendor, there is never a reason to be rude.   David Miner gave a talk about whats coming in Solaris.next and Ben Rockwood gave an entertaining and informative presentation on ZFS in the Trenches.

I was lucky enough to get a chance to talk with David Miner over a quick lunch later in the week and talk about the new opportunities and challenges with the OpenSolaris installation technologies.

On Wednesday through Friday I attended a mix of presentations, met with a bunch of vendors, and also sat in some of the ‘Guru is in’ sessions and talked with a number of conference attendees.  The highlights for me were:

• Werner Vogel (CTO of Amazon) gave a fascinating talk on the history of Amazon’s IT philosophy and infrastructure and how they evolved from a humble internal IT shop to adding a business which is the dominant  cloud computing provider.
• Elizabeth Zwicky’s talk on “Searching for Truth, or at Least Data: How to Be an Empiricist Skeptic”
• Bryan Cantrill’s talk on “Visualizing DTrace: Sun Storage 7000 Analytics”
  
• Talking with the folks from Splunk (awesome log searching analysis tool)

For anyone that has spent much time looking at MySQL performance, you will be familiar with the ‘slow query log’.  This basically is a log where queries that took over some amount of time would get recorded.   For kicks, I tried implementing a similar hook for OpenDS.  My current version is in pretty rough shape (not very efficient or configurable), but seems to work.  I started from a copy of the TextAccessLogPublisher.java file and created a new one called TextSlowAccessLogPublisher.java.  My logic is basically:

• create a hash table
• emptied out all the log XYZIntermediateMessage and connect/disconnect methods
• when a request comes in, store the text to log in the hash table (keyed off connectionID and opNumber) instead of outputting it (changed the logSearchRequest, logModifyRequest, … methods)
• when a request is finished processing, we check the elapsed time (etime)
  • if the elapsed time greater  than our or equal to our threshold
    • print the request info we stashed in the hash table and delete it
    • print the response info
  • if the elapsed time is less than our threshold
    • delete the request info from the hash table, don’t print anything

There are a few more things I want to do:

• Make the ‘slow operation threshold time’ dynamically changeable (looks like I will need to mess with configuration objects since I want to add an additional parameter not in the standard access log type)
• Add extra information to the output format such as authorization DN (and potentially client connection info if not too hard to retrieve)
• Instead of all the text formatting for every request, just put the Operation object into the hash table, since the majority of operations won’t ever get printed we shouldn’t burn CPU formatting them.  The operations would only be formatted to text if the operations end up being slow and printed.

Files

• TextSlowAccessLogPublisher.java (currently quick-n-dirty quality)
• LDIF to enable the slow query log