williamhathaway.com

I borrowed “Release It!” from Bill Kratzer a few weeks ago and have really enjoyed reading it.  My favorite quote from the book is “Feature complete does not mean production ready”.  I think this sums up a lot of large software projects, especially when there is a disconnect between the development team and the group responsible for deployment and operations.

The book covers 4 main topics:

• Stability
• Capacity
• General Design Issues
• Operations

In most of the sections the author breaks his advice down into an introduction (with a real example showing a problem), a set of anti-patterns that encourage the problem and a set of patterns to help  software cope with the various stresses placed on it and make it manageable.

The book stays at a relatively high level of discussion and is easy to follow.  If you are looking for lots of low-level coding examples you will be disappointed, but I think the book offers good advice that can be consumed by a wide range of people ranging from developers, to system administrators, and to project managers.

Last year I was involved in a project that struggled with a lot of the issues mentioned in this book and I think that hundreds of thousands of dollars and countless hours of stress and frustration could have been saved if this book had been required reading at the start of the project.

I recommend this book to anyone involved in developing or operating software services, or managing the people that do.

For anyone that has spent much time looking at MySQL performance, you will be familiar with the ‘slow query log’.  This basically is a log where queries that took over some amount of time would get recorded.   For kicks, I tried implementing a similar hook for OpenDS.  My current version is in pretty rough shape (not very efficient or configurable), but seems to work.  I started from a copy of the TextAccessLogPublisher.java file and created a new one called TextSlowAccessLogPublisher.java.  My logic is basically:

• create a hash table
• emptied out all the log XYZIntermediateMessage and connect/disconnect methods
• when a request comes in, store the text to log in the hash table (keyed off connectionID and opNumber) instead of outputting it (changed the logSearchRequest, logModifyRequest, … methods)
• when a request is finished processing, we check the elapsed time (etime)
  • if the elapsed time greater  than our or equal to our threshold
    • print the request info we stashed in the hash table and delete it
    • print the response info
  • if the elapsed time is less than our threshold
    • delete the request info from the hash table, don’t print anything

There are a few more things I want to do:

• Make the ‘slow operation threshold time’ dynamically changeable (looks like I will need to mess with configuration objects since I want to add an additional parameter not in the standard access log type)
• Add extra information to the output format such as authorization DN (and potentially client connection info if not too hard to retrieve)
• Instead of all the text formatting for every request, just put the Operation object into the hash table, since the majority of operations won’t ever get printed we shouldn’t burn CPU formatting them.  The operations would only be formatted to text if the operations end up being slow and printed.

Files

• TextSlowAccessLogPublisher.java (currently quick-n-dirty quality)
• LDIF to enable the slow query log

I had a good time attending the CPOSC event  today, which was held at Harrisburg University.  Got to see lots of old friends and acquaintances and enjoyed the speakers.  At only $35 (which included a t-shirt and food), it was an awesome bargain.  Thanks to John and Eric for doing such a great job organizing the event and all the presenters for sharing their knowledge.

I just saw a post from Bill Krazter about searching for his first post on usenet.  This got me thinking about my activities and I google stalked the resume I sent out on misc.jobs.resumes on June 9th 1995.

Kind of interesting to look back at how I thought of myself 14 years ago.  The text formatting got a little dorked up in the translation.

Hi,
I've just gotten a Bachelor's degree in Computer Information Science,
and am looking for employment as a programmer or assistant sysadmin
in a UNIX shop.  I have very good C skills, have done a pretty fair
amount of "lower-level" unix coding for both the computer services dept
at Kutztown, and for my personal use (i.e. wrote a finger utility when we
switched OS's, and Dynix's finger was broken, wrote a user watch program
that notified me when friends logged onto any of the networked machines
here, and a process reaper to help kill runaway programs).  I'm a quick
learner and always enjoy exploring new areas. I have good verbal/written
communication skills, enjoy beating expectations, and am a team player.
I'd prefer a dress-down environment, and to work in the Philadelphia/Princeton
area.   If it sounds like I might be a decent match for your business,
please get in touch.  Thanks for your time.
-Bill
hathXXXX@kutztown.edu
+++++++++++++++++++++++++++++++++++++++++++
++
William D. Hathaway
4XXX George School
Newtown, PA 18940
215/860-XXXX
OBJECTIVE   An entry level position as a system adminstrator/programmer.
EDUCATION   Kutztown University, Kutztown Pennsylvania
Bachelor of Science in Computer Information Science
Concentration: Business
GPA: 3.34/4.00
May 1995
COMPUTER
KNOWLEDGE   Operating Systems: UNIX (SunOS/Solaris,Dynix,OSF/1,Linux),
MSDOS,MacOS,Windows
Languages        : C,C++,Ada,pascal,COBOL,80x86 assembler,
perl,shell scripts,lex,yacc
Programmed in C since September 1992.
Extensive working with UNIX since January 1992.
Adminstered own Linux machine since February 1993.
RELEVANT
COURSES     Internetwork Programming (independent study)
Data Communications          File Structures
Compiler Design              Algorithm Analysis
EXPERIENCE  Student Programmer/Consultant September 1993 - May 1995
Kutztown University Computer Services, Kutztown PA.
* helped network adminstrator pinpoint/fix problems
* individualized tutoring about UNIX/internet
* wrote various utility programs
ACTIVITIES  Lacrosse Team
Kutztown University Computer Association
REFERENCES  Available upon request

I haven’t updated in a while, here is the blitz:

Running – I managed to somehow get a minor muscle strain in my back which took me out of running for about 10 days (possibly by overdoing in on the Wii Fit ‘Super Hula Hoop’ game).  I came back with a 6 miler @ Conewago on Saturday with the gang, took Sunday off, ran 5.5 miles with Jon, Tex, and Cali on Monday, and then ran 3 miles with Nate tonight. The Geisinger Humdinger race is coming up in about a week and a half. Hopefully I’ll be able to get in five or six solid runs in before then.  I definitely won’t be able to beat my time from last year, but I will be happy if I am able to push hard for the entire length of the race.

Finance – mint.com has added some nice new features, including being able to track off-line assets like houses, vehicles, or just accounts that can’t be accessed automatically.  They also prettied up the user interface a bit.

Zombie Killing - I started playing the video game Left4Dead with some friends online, which has been a lot of fun.

Programming – I’ve been checking out a new Java SDK for LDAP from the Unbound ID folks.  If you have been using the ancient Netscape/Mozilla LDAP SDK before, give Unbound’s a shot, it has a lot of interesting features.

Solaris – if you are troubleshooting anything relating to tcp listen drops, check out the ndd -get /dev/tcp tcplistenhash command.   One quirk I noticed was that if you aren’t root, it has some limit to the frequency you can run the command.  I assume the command is a bit expensive and they are trying to prevent a DOS attack, but the observability is excellent.

I read a copy of Steve Souder’s book Higher Performance Web Sites: Essential Knowledge for Front-End Engineers about two months ago.  I had enjoyed the book but had forgotten to blog about it until I heard a podcast last night that was talking about measuring site performance.

Souder’s book is a fairly quick read.  He provides a series of rules websites should follow to provide  good user response times and then goes into detailed descriptions about the techical nuisances and provides examples of how different choices affect real-world performance.  Steve’s rules have been embedded in the YSlow plugin for Firefox which can be ran against a site to get a list of which best practices are used and ends up giving a letter grade for overall site optimization.

You can read a short article that gives a high level overview of some best practices that are mentioned.

Recommendation: If you are a website developer, I think this is a fantastic book to read.   If you are short on time, just read the article above and download the YSlow plugin.

Additional notes: KITE is a free website performance tool (Windows only) offered by Keynote that was discussed in the podcast.

I found a web site called slideshare that is focused around sharing presentations.  One of the interesting collections for me was a  bunch of presos from people who discuss various scaling issues involved with building large web sites.

I just tried registering a Solaris 10 11/06 machine using sconadm and here is the resulting output:

/usr/sbin/sconadm register -a -r /tmp/regprofile
sconadm is running
javax.management.remote.JMXProviderException: Connection refused
at com.sun.cacao.rmi.impl.CacaoRMIConnectorProvider.newJMXConnector(Caca oRMIConnectorProvider.java:415)
at javax.management.remote.JMXConnectorFactory.getConnectorAsService(JMX ConnectorFactory.java:415)
at javax.management.remote.JMXConnectorFactory.newJMXConnector(JMXConnec torFactory.java:307)
at com.sun.scn.client.MSCConnection.doMscConnect(MSCConnection.java:155)
at com.sun.scn.client.MSCConnection.mscConnectWithKeystorePassword(MSCCo nnection.java:67)

(about 30 lines snipped out by me)

at com.sun.cns.basicreg.BasicReg.<init>(BasicReg.java:100)
at com.sun.cns.basicreg.BasicRegCLI.main(BasicRegCLI.java:523)

How about something more helpful like:

ERROR 68: Can’t connect to the fuzzbat service on server server99.

Please check that the fuzzbat service is enabled with ‘svcadm enable fuzzbat; svcs fuzzbat’.

While stack traces are helpful to developers, they can largely obfuscate the problem to the end-user and certainly don’t provide guidance on what the likely problem is and what the next step should be. If you want to print stack traces, at least try to provide some actionable information to the end user.

(My error was apparently generated because I had renamed the machine, but hadn’t bounced some services. After I rebooted, the sconadm command worked fine.)