williamhathaway.com

Here are a couple of systems tools I’ve been using lately and have been very happy with:

Puppet (configuration management) – I’ve used CFengine a lot on the past for configuration management, but I now prefer Puppet due to its wide scope of manageable resources, ranging the gamut from standard file distribution to  packages, services, users, network configurations, etc.   We are currently using Puppet to bootstrap systems from a base CentOS image to having all the components needed for their role and it is working out really well. I read the book Pulling String with Puppet to get ramped up, and while it is helpful, it lacks an index, so I’d recommend getting an electronic copy.

Splunk (log/event parsing) – I haven’t seen any other locally deployed tool that can come anywhere close to Splunk for taking various log files and making the information immediately usable on a wide scale.  The search interface is super shiny and has a lot of very interesting capabilities for creating dashboard, performing data mining, and alerting.  Splunk came out with a new release (4.1) last week.  If you last looked at Splunk more than a year ago, check out whats new.  If you are looking for a cloud-based solution, Paglo (recently bought by Citrix)seems to be an interesting option.

Func (command & control) – while almost all of the management in the environment is done via Puppet, there are times when we want to run commands across a set of hosts. Func fills the niche nicely by providing the ability to use wildcards and grouping for host selection and running against multiple targets in parallel.  For folks wanting to get a quick overview of Func, I recommend Dan Hank’s Managing Your Minions With Func presentation

Photo by John @ ThinkHole.com

On Tuesday night I gave a presentation on ZFS to the Central PA Linux User Group. Since the audience was a Linux user group, I wasn’t expecting too many in the crowd to be familiar with ZFS, but I was pleasantly surprised that about 40% of the ~ 20 people in attendance had used ZFS in some capacity. If you are already a seasoned ZFS user, I would highly recommend Richard Elling’s ZFS presentation which he uses in his day-long tutorials.

I spent last week at the LISA Conference in Baltimore MD.  if you aren’t familiar with LISA, it is a conference focused on system administration.  This is the 4th  LISA I’ve attended in the last 12 years.

On Monday I attended a tutorial by Richard Elling on ZFS: A Filesystem for Modern Hardware.

On Tuesday I attended two tutorials.  The first was Jacob Farmer’s Disk-to-Disk Backup and Eliminating Backup System Bottlenecks.  The second was Tom Limoncelli’s Design Patterns for System Administrators.

Unfortunately on both Monday and Tuesday I had to spend a significant amount of time on conference calls helping to troubleshoot some work related issues, but the time I spent in all 3 sessions and viewing their materials was helpful.  I would definitely recommend attending tutorials by any of the 3 people above if they are teaching a topic of interest to you.

On Tuesday night I attended some (Open)Solaris birds-of-a-feather sessions.  There were a few times that people in the crowd were being belligerent towards a speaker (mostly complaining about the difficulty of finding information of various types), even though the speaker certainly had no sway over what the person in the crowd was upset about.  I don’t care how much money your company spends with a vendor, there is never a reason to be rude.   David Miner gave a talk about whats coming in Solaris.next and Ben Rockwood gave an entertaining and informative presentation on ZFS in the Trenches.

I was lucky enough to get a chance to talk with David Miner over a quick lunch later in the week and talk about the new opportunities and challenges with the OpenSolaris installation technologies.

On Wednesday through Friday I attended a mix of presentations, met with a bunch of vendors, and also sat in some of the ‘Guru is in’ sessions and talked with a number of conference attendees.  The highlights for me were:

• Werner Vogel (CTO of Amazon) gave a fascinating talk on the history of Amazon’s IT philosophy and infrastructure and how they evolved from a humble internal IT shop to adding a business which is the dominant  cloud computing provider.
• Elizabeth Zwicky’s talk on “Searching for Truth, or at Least Data: How to Be an Empiricist Skeptic”
• Bryan Cantrill’s talk on “Visualizing DTrace: Sun Storage 7000 Analytics”
  
• Talking with the folks from Splunk (awesome log searching analysis tool)

I had a good time attending the CPOSC event  today, which was held at Harrisburg University.  Got to see lots of old friends and acquaintances and enjoyed the speakers.  At only $35 (which included a t-shirt and food), it was an awesome bargain.  Thanks to John and Eric for doing such a great job organizing the event and all the presenters for sharing their knowledge.

I have a lab machine running OpenSolaris 2009.06 (updated to snv_117) and had created an LDIF file with about 100k small entries in it (file size was ~ 63 megs).  I wanted to get a count of the exact number of entries so I ran:

grep -c ^dn:

I expected it to take a second or two.  I was wrong.  It was painfully slow.

I used the time command to re-run the grep and saw it clocked in at just over a minute.

This was weird, so I though it was time to investigate further.  I used the DTrace Toolkit’s hotuser command to see what the hot functions were:

pfexec /opt/DTT/hotuser -c "grep -c ^dn: /var/tmp/search.out"
Sampling... Hit Ctrl-C to end.

FUNCTION                                                     COUNT   PCNT
...<snipped out smaller functions>...
ggrep`check_multibyte_string                                    5480   8.9%
methods_unicode.so.3`__mbrtowc_dense_utf8                      12328  20.1%
libc.so.1`mbrlen                                               13566  22.1%
libc.so.1`memset                                               23014  37.5%

Hmm, interesting to see the calls to mbrlen and methods_unicode among the hot functions.  Lets check my $LANG setting:

echo $LANG

en_US.UTF-8

Bingo!  Lets try it again with a non multi-byte LANG setting.

LANG=C time grep -c ^dn: /var/tmp/search.out

99987

real        0.1
user        0.0
sys         0.0

That looks normal.  Now lets try one more time with a multi-byte LANG to be sure:

LANG=en_US.UTF-8 time grep -c ^dn: /var/tmp/search.out
99987

real     1:01.4
user     1:01.3
sys         0.0

Yep, the problem is confirmed.

Notes

For those unfamiliar with OpenSolaris,  the default path has /usr/gnu/bin first.  The grep I was using was:

grep -V
grep (GNU grep) 2.5

Copyright 1988, 1992-1999, 2000, 2001 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

If you use the non-GNU grep available at /usr/xpg4/bin/grep it doesn’t have the big slowdown regardless of the LANG.

I also tried the same test on the GNU wc command and saw about a 25x difference when using a multi-byte LANG.

For both grep and wc, I re-ran the tests multiple times to make sure that file system caching played no role in the results.

I think these performance differences are way higher than they should be, I’m going to dig further when I have a chance.

Venn diagram of UNIX and graffiti enthusthiasts

I came across this on a couple of buildings, fence posts, and the sidewalk while walking around Morgantown.

Has anyone seen UNIX-like graffiti tagging elsewhere?

UNIX graffiti in Morgantown, WV

Here are the steps I have used in the past to configure some Centos boxes being used as LoadRunner load generators to get better performance when trying to generate a lot of traffic.

Note: these were machines that were only accessible from within a very limited environment, I don’t think these suggestions are good from a security perspective.

1) Increase the available TCP ports used for outbound connections
edit etc/sysctl.conf and add
net.ipv4.ip_local_port_range = 1024 65000

To apply the change immediately:
root@host# sysctl net.ipv4.ip_local_port_range=”1024 65000″

2) Disable IP Tables

use the service and chkconf commands to stop the service and disable it from starting at boot

root@host# service iptables save
root@host# service iptables stop
root@host# chkconfig iptables off

3) Disable Selinux
edit /etc/selinux/config and change the selinux line to be:

SELINUX=disabled

and reboot.

To apply the change immediately, use:

root@host# echo 0 >/selinux/enforce

1) Increase the available TCP ports used for outbound connections
edit etc/sysctl.conf and add
net.ipv4.ip_local_port_range = 1024 65000

To apply the change immediately:
root@host# sysctl net.ipv4.ip_local_port_range=”1024 65000″