• The software generally works as advertised
• It is very easy to get up and running, and the architecture is easy to change if your requirements shift
• Updates occur on a pretty quick cadence
• There is a free license version of the product that has a reasonable amount of functionality and works fine for people that don’t have a huge data need
• All of their documentation is freely available (I dislike vendors that make you spend $ or jump through hoops to access docs)
• They use the excellent StackOverflow engine for their community driven support site answers.splunk.com, making it easy to contribute and floating good answers to the top

Another company that I hold in a similar light is Atlassian (although they don’t have free versions, they have licenses for the first 10 users available for $10, which is close enough).

I bought a wifi based iPad pretty soon after it was released.  Here is my quick take:   The iPad definitely won’t replace my laptop or cell phone, but it is pretty handy as an entertainment and media consuming device.  The screen based keyboard isn’t as bad as I thought, but still isn’t very efficient if I need to bang out a lot of text.    I can see the form factor working really well for a variety of cases where someone wants to run specialized apps, such as the medical field using it for patient records.

Pros

• I’m using  DropBox + GoodReader to transfer and view PDFs which is working well for reading on planes.
• Netflix streaming of videos is great.
• Kindle app means lots of ebook availability (Apple’s iBook store is fairly sparse now in comparison, but you get access to both).
• I’m really happy with the long battery life.
• Web browser is very quick and crisp (although I wish it had Flash support or more sites switch to HTML 5)
• Mail client/Gmail is very usable, although I still prefer a laptop keyboard for long messages.

Cons

• Wifi support is persnickety.  I’ve had trouble connecting to some access points when both my iPhone and MacBook Pro were getting full signal.  I hope this is firmware fixable.  I thought about getting a 3G iPad but couldn’t stomach a single device broadband fee.
• Most apps originally targeted at the iPhone look either wimpy at original iPhone screen size or don’t scale up nicely when you switch to 2x magnifcationn mode

Puppet (configuration management) – I’ve used CFengine a lot on the past for configuration management, but I now prefer Puppet due to its wide scope of manageable resources, ranging the gamut from standard file distribution to  packages, services, users, network configurations, etc.   We are currently using Puppet to bootstrap systems from a base CentOS image to having all the components needed for their role and it is working out really well. I read the book Pulling String with Puppet to get ramped up, and while it is helpful, it lacks an index, so I’d recommend getting an electronic copy.

Splunk (log/event parsing) – I haven’t seen any other locally deployed tool that can come anywhere close to Splunk for taking various log files and making the information immediately usable on a wide scale.  The search interface is super shiny and has a lot of very interesting capabilities for creating dashboard, performing data mining, and alerting.  Splunk came out with a new release (4.1) last week.  If you last looked at Splunk more than a year ago, check out whats new.  If you are looking for a cloud-based solution, Paglo (recently bought by Citrix)seems to be an interesting option.

Func (command & control) – while almost all of the management in the environment is done via Puppet, there are times when we want to run commands across a set of hosts. Func fills the niche nicely by providing the ability to use wildcards and grouping for host selection and running against multiple targets in parallel.  For folks wanting to get a quick overview of Func, I recommend Dan Hank’s Managing Your Minions With Func presentation

A colleague and I were  working with a new Seagate Black Armor 420 NAS appliance today and struggling with NFS.  It was often taking 20 or more seconds for local NFS mounts to occur, and this was below the timeout of our main clients (VMWare ESXi 4.0).  While the NAS has a feature that will automatically tell you if there are newer versions of the firmware available, this wasn’t working properly.  The device indicated that no new firmware was available, but after some forum searching, we saw a newer version (4000.0631) was downloadable.  The manual upgrade went fine and then NFS started working like a champ.   If you have a tight ( < $1000) budget for a Network Attached Storage device, the Black Armor line seem to be pretty feature rich for the price.

The book covers 4 main topics:

• Stability
• Capacity
• General Design Issues
• Operations

In most of the sections the author breaks his advice down into an introduction (with a real example showing a problem), a set of anti-patterns that encourage the problem and a set of patterns to help  software cope with the various stresses placed on it and make it manageable.

The book stays at a relatively high level of discussion and is easy to follow.  If you are looking for lots of low-level coding examples you will be disappointed, but I think the book offers good advice that can be consumed by a wide range of people ranging from developers, to system administrators, and to project managers.

Last year I was involved in a project that struggled with a lot of the issues mentioned in this book and I think that hundreds of thousands of dollars and countless hours of stress and frustration could have been saved if this book had been required reading at the start of the project.

I recommend this book to anyone involved in developing or operating software services, or managing the people that do.

]]> http://www.williamhathaway.com/wordpress/2010/01/21/thumbs-up-for-release-it/feed/ 0 http://www.williamhathaway.com/wordpress/2010/01/14/zfs-presentation/ http://www.williamhathaway.com/wordpress/2010/01/14/zfs-presentation/#comments Thu, 14 Jan 2010 14:40:22 +0000 William Hathaway http://www.williamhathaway.com/wordpress/?p=527

Photo by John @ ThinkHole.com

On Tuesday night I gave a presentation on ZFS to the Central PA Linux User Group. Since the audience was a Linux user group, I wasn’t expecting too many in the crowd to be familiar with ZFS, but I was pleasantly surprised that about 40% of the ~ 20 people in attendance had used ZFS in some capacity. If you are already a seasoned ZFS user, I would highly recommend Richard Elling’s ZFS presentation which he uses in his day-long tutorials.

A friend of mine’s son, Alex, has recently accepted a software development job at a financial trading company.  He is starting his final semester of school, so he won’t begin working at the company for another 4-5 months.  When I was over visiting at their house last week, Alex showed me a small stack of  books that the company had sent him.  The books covered  a mix of technical and business topics that would help him build up an understanding of the software tools, development philosophies, and business concepts specific to the organization so that when he arrived at work he will be productive much quicker.

I think this is a fantastic investment by the company, and should be considered by organizations hiring for any but senior positions.  You obviously don’t want to overwhelm new hires with an onslaught of 10,000 pages of recommended reading, but having a small package from Amazon show up at their door containing a few books most appropriate to their position and your culture is a great way to help new hires get up to speed, even before they hit the door.

pkg set-publisher -O http://pkg.codenursery.com/ codenursery.com
pkg install puppet

groupadd puppet
useradd -g puppet puppet

mkdir /etc/puppet /var/puppet

/usr/ruby/1.8/sbin/puppetd  --genconfig > /etc/puppet/puppet.conf

svcadm enable puppet/master

On Monday I attended a tutorial by Richard Elling on ZFS: A Filesystem for Modern Hardware.

On Tuesday I attended two tutorials.  The first was Jacob Farmer’s Disk-to-Disk Backup and Eliminating Backup System Bottlenecks.  The second was Tom Limoncelli’s Design Patterns for System Administrators.

Unfortunately on both Monday and Tuesday I had to spend a significant amount of time on conference calls helping to troubleshoot some work related issues, but the time I spent in all 3 sessions and viewing their materials was helpful.  I would definitely recommend attending tutorials by any of the 3 people above if they are teaching a topic of interest to you.

On Tuesday night I attended some (Open)Solaris birds-of-a-feather sessions.  There were a few times that people in the crowd were being belligerent towards a speaker (mostly complaining about the difficulty of finding information of various types), even though the speaker certainly had no sway over what the person in the crowd was upset about.  I don’t care how much money your company spends with a vendor, there is never a reason to be rude.   David Miner gave a talk about whats coming in Solaris.next and Ben Rockwood gave an entertaining and informative presentation on ZFS in the Trenches.

I was lucky enough to get a chance to talk with David Miner over a quick lunch later in the week and talk about the new opportunities and challenges with the OpenSolaris installation technologies.

On Wednesday through Friday I attended a mix of presentations, met with a bunch of vendors, and also sat in some of the ‘Guru is in’ sessions and talked with a number of conference attendees.  The highlights for me were:

• Werner Vogel (CTO of Amazon) gave a fascinating talk on the history of Amazon’s IT philosophy and infrastructure and how they evolved from a humble internal IT shop to adding a business which is the dominant  cloud computing provider.
• Elizabeth Zwicky’s talk on “Searching for Truth, or at Least Data: How to Be an Empiricist Skeptic”
• Bryan Cantrill’s talk on “Visualizing DTrace: Sun Storage 7000 Analytics”
  
• Talking with the folks from Splunk (awesome log searching analysis tool)

• create a hash table
• emptied out all the log XYZIntermediateMessage and connect/disconnect methods
• when a request comes in, store the text to log in the hash table (keyed off connectionID and opNumber) instead of outputting it (changed the logSearchRequest, logModifyRequest, … methods)
• when a request is finished processing, we check the elapsed time (etime)
  • if the elapsed time greater  than our or equal to our threshold
    • print the request info we stashed in the hash table and delete it
    • print the response info
  • if the elapsed time is less than our threshold
    • delete the request info from the hash table, don’t print anything

There are a few more things I want to do:

• Make the ‘slow operation threshold time’ dynamically changeable (looks like I will need to mess with configuration objects since I want to add an additional parameter not in the standard access log type)
• Add extra information to the output format such as authorization DN (and potentially client connection info if not too hard to retrieve)
• Instead of all the text formatting for every request, just put the Operation object into the hash table, since the majority of operations won’t ever get printed we shouldn’t burn CPU formatting them.  The operations would only be formatted to text if the operations end up being slow and printed.

Files

• TextSlowAccessLogPublisher.java (currently quick-n-dirty quality)
• LDIF to enable the slow query log