williamhathaway.com

After a frustrating 15 months or so of very sporadic running,  I’m happy to report that I’ve been able to run fairly consistently for the last 3 months.  I am still getting some annoying BFS based symptoms, but it seems like I am trending towards getting better.  I also had a EMG in December to help make sure there was nothing else amiss (I had been given a misdiagnosis of muscular dystrophy from a neurologist in early 2009, which really scared me).

For the last 3 weeks I’ve been averaging 30 miles a week and would like to run in the Garden Spot Village Marathon on April 10th with Nate.   My certainty level about the race isn’t too high due to the impending California move and lack of clarity about my work/travel schedule.  If I can’t make that race for some reason, I’ll try to keep my mileage consistent and then look for a marathon more local to San Jose.

I’m definitely not in prime shape yet, but at this point, I am thrilled to be running again on a regular basis.

I borrowed “Release It!” from Bill Kratzer a few weeks ago and have really enjoyed reading it.  My favorite quote from the book is “Feature complete does not mean production ready”.  I think this sums up a lot of large software projects, especially when there is a disconnect between the development team and the group responsible for deployment and operations.

The book covers 4 main topics:

• Stability
• Capacity
• General Design Issues
• Operations

In most of the sections the author breaks his advice down into an introduction (with a real example showing a problem), a set of anti-patterns that encourage the problem and a set of patterns to help  software cope with the various stresses placed on it and make it manageable.

The book stays at a relatively high level of discussion and is easy to follow.  If you are looking for lots of low-level coding examples you will be disappointed, but I think the book offers good advice that can be consumed by a wide range of people ranging from developers, to system administrators, and to project managers.

Last year I was involved in a project that struggled with a lot of the issues mentioned in this book and I think that hundreds of thousands of dollars and countless hours of stress and frustration could have been saved if this book had been required reading at the start of the project.

I recommend this book to anyone involved in developing or operating software services, or managing the people that do.

Photo by John @ ThinkHole.com

On Tuesday night I gave a presentation on ZFS to the Central PA Linux User Group. Since the audience was a Linux user group, I wasn’t expecting too many in the crowd to be familiar with ZFS, but I was pleasantly surprised that about 40% of the ~ 20 people in attendance had used ZFS in some capacity. If you are already a seasoned ZFS user, I would highly recommend Richard Elling’s ZFS presentation which he uses in his day-long tutorials.

A friend of mine’s son, Alex, has recently accepted a software development job at a financial trading company.  He is starting his final semester of school, so he won’t begin working at the company for another 4-5 months.  When I was over visiting at their house last week, Alex showed me a small stack of  books that the company had sent him.  The books covered  a mix of technical and business topics that would help him build up an understanding of the software tools, development philosophies, and business concepts specific to the organization so that when he arrived at work he will be productive much quicker.

I think this is a fantastic investment by the company, and should be considered by organizations hiring for any but senior positions.  You obviously don’t want to overwhelm new hires with an onslaught of 10,000 pages of recommended reading, but having a small package from Amazon show up at their door containing a few books most appropriate to their position and your culture is a great way to help new hires get up to speed, even before they hit the door.

I’ve been taking advantage of my vacation time during the past week and a half by reading a motley collection of technology, science, career, and military history books.  One of the books (“Ship It!”) I had to read on the computer via the Safari on-line book service since it isn’t available in paper form yet.  It has been nice to spend a lot of time focused and really absorbing material versus the tons of short-form articles/blog entries that I tend to read while doing interrupt driven surfing on the computer.  I know my reading pace will slow down substantially when works starts up again, but I  want to keep tackling at least one new book a week.

Over the holiday vacation my brother mentioned he was a big fan of a library search service called WorldCat.  If you give WorldCat a book or media title/ISBN, it will search thousands of library catalogs and let you know if it finds any libraries nearby that have it.  Not all libraries are indexed by WorldCat, so depending on where you live (and your library system) mileage may vary, but it is worth checking out.

While looking at the Reductive Labs’  Puppet on Solaris page I saw there was a repository which hosts Puppet in a pkg format.  This makes installing a Puppet server on OpenSolaris pretty easy.

pkg set-publisher -O http://pkg.codenursery.com/ codenursery.com
pkg install puppet

groupadd puppet
useradd -g puppet puppet

mkdir /etc/puppet /var/puppet

/usr/ruby/1.8/sbin/puppetd  --genconfig > /etc/puppet/puppet.conf

svcadm enable puppet/master

I spent last week at the LISA Conference in Baltimore MD.  if you aren’t familiar with LISA, it is a conference focused on system administration.  This is the 4th  LISA I’ve attended in the last 12 years.

On Monday I attended a tutorial by Richard Elling on ZFS: A Filesystem for Modern Hardware.

On Tuesday I attended two tutorials.  The first was Jacob Farmer’s Disk-to-Disk Backup and Eliminating Backup System Bottlenecks.  The second was Tom Limoncelli’s Design Patterns for System Administrators.

Unfortunately on both Monday and Tuesday I had to spend a significant amount of time on conference calls helping to troubleshoot some work related issues, but the time I spent in all 3 sessions and viewing their materials was helpful.  I would definitely recommend attending tutorials by any of the 3 people above if they are teaching a topic of interest to you.

On Tuesday night I attended some (Open)Solaris birds-of-a-feather sessions.  There were a few times that people in the crowd were being belligerent towards a speaker (mostly complaining about the difficulty of finding information of various types), even though the speaker certainly had no sway over what the person in the crowd was upset about.  I don’t care how much money your company spends with a vendor, there is never a reason to be rude.   David Miner gave a talk about whats coming in Solaris.next and Ben Rockwood gave an entertaining and informative presentation on ZFS in the Trenches.

I was lucky enough to get a chance to talk with David Miner over a quick lunch later in the week and talk about the new opportunities and challenges with the OpenSolaris installation technologies.

On Wednesday through Friday I attended a mix of presentations, met with a bunch of vendors, and also sat in some of the ‘Guru is in’ sessions and talked with a number of conference attendees.  The highlights for me were:

• Werner Vogel (CTO of Amazon) gave a fascinating talk on the history of Amazon’s IT philosophy and infrastructure and how they evolved from a humble internal IT shop to adding a business which is the dominant  cloud computing provider.
• Elizabeth Zwicky’s talk on “Searching for Truth, or at Least Data: How to Be an Empiricist Skeptic”
• Bryan Cantrill’s talk on “Visualizing DTrace: Sun Storage 7000 Analytics”
  
• Talking with the folks from Splunk (awesome log searching analysis tool)

For anyone that has spent much time looking at MySQL performance, you will be familiar with the ‘slow query log’.  This basically is a log where queries that took over some amount of time would get recorded.   For kicks, I tried implementing a similar hook for OpenDS.  My current version is in pretty rough shape (not very efficient or configurable), but seems to work.  I started from a copy of the TextAccessLogPublisher.java file and created a new one called TextSlowAccessLogPublisher.java.  My logic is basically:

• create a hash table
• emptied out all the log XYZIntermediateMessage and connect/disconnect methods
• when a request comes in, store the text to log in the hash table (keyed off connectionID and opNumber) instead of outputting it (changed the logSearchRequest, logModifyRequest, … methods)
• when a request is finished processing, we check the elapsed time (etime)
  • if the elapsed time greater  than our or equal to our threshold
    • print the request info we stashed in the hash table and delete it
    • print the response info
  • if the elapsed time is less than our threshold
    • delete the request info from the hash table, don’t print anything

There are a few more things I want to do:

• Make the ‘slow operation threshold time’ dynamically changeable (looks like I will need to mess with configuration objects since I want to add an additional parameter not in the standard access log type)
• Add extra information to the output format such as authorization DN (and potentially client connection info if not too hard to retrieve)
• Instead of all the text formatting for every request, just put the Operation object into the hash table, since the majority of operations won’t ever get printed we shouldn’t burn CPU formatting them.  The operations would only be formatted to text if the operations end up being slow and printed.

Files

• TextSlowAccessLogPublisher.java (currently quick-n-dirty quality)
• LDIF to enable the slow query log

I had a good time attending the CPOSC event  today, which was held at Harrisburg University.  Got to see lots of old friends and acquaintances and enjoyed the speakers.  At only $35 (which included a t-shirt and food), it was an awesome bargain.  Thanks to John and Eric for doing such a great job organizing the event and all the presenters for sharing their knowledge.