I have a lab machine running OpenSolaris 2009.06 (updated to snv_117) and had created an LDIF file with about 100k small entries in it (file size was ~ 63 megs).  I wanted to get a count of the exact number of entries so I ran:

grep -c ^dn:

I expected it to take a second or two.  I was wrong.  It was painfully slow.

I used the time command to re-run the grep and saw it clocked in at just over a minute.

This was weird, so I though it was time to investigate further.  I used the DTrace Toolkit’s hotuser command to see what the hot functions were:

pfexec /opt/DTT/hotuser -c "grep -c ^dn: /var/tmp/search.out"
Sampling... Hit Ctrl-C to end.

FUNCTION                                                     COUNT   PCNT
...<snipped out smaller functions>...
ggrep`check_multibyte_string                                    5480   8.9%
methods_unicode.so.3`__mbrtowc_dense_utf8                      12328  20.1%
libc.so.1`mbrlen                                               13566  22.1%
libc.so.1`memset                                               23014  37.5%

Hmm, interesting to see the calls to mbrlen and methods_unicode among the hot functions.  Lets check my $LANG setting:

echo $LANG

en_US.UTF-8

Bingo!  Lets try it again with a non multi-byte LANG setting.

LANG=C time grep -c ^dn: /var/tmp/search.out

99987

real        0.1
user        0.0
sys         0.0

That looks normal.  Now lets try one more time with a multi-byte LANG to be sure:

LANG=en_US.UTF-8 time grep -c ^dn: /var/tmp/search.out
99987

real     1:01.4
user     1:01.3
sys         0.0

Yep, the problem is confirmed.

Notes

For those unfamiliar with OpenSolaris,  the default path has /usr/gnu/bin first.  The grep I was using was:

grep -V
grep (GNU grep) 2.5

Copyright 1988, 1992-1999, 2000, 2001 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

If you use the non-GNU grep available at /usr/xpg4/bin/grep it doesn’t have the big slowdown regardless of the LANG.

I also tried the same test on the GNU wc command and saw about a 25x difference when using a multi-byte LANG.

For both grep and wc, I re-ran the tests multiple times to make sure that file system caching played no role in the results.

I think these performance differences are way higher than they should be, I’m going to dig further when I have a chance.

My Garmin RSS feed

I was pretty excited about racing in this 5k since it has been about 4 months since my last 5k.

I arrived at City Island about 40 minutes before the 8 AM start. The island was buzzing with activity as lots of armed forces personnel were working on getting the vehicles (tanks, helicopters, armored personnel carriers, etc) and other displays ready. Picking up the bib was painless because the race was pretty small (about 215 people). I jogged around for about 15 minutes, saw a few people from the Harrisburg Road Runners and then went over to the starting area.

I had lined up about 5 rows back from the front figuring I’d would probably be in the top 20 or 30 finishers. As people were getting ready to start, a helicopter came down and was landing about 100 feet away. This was extremely loud and I was wondering if people would actually be able to hear the race director over the noise. We waited for a minute or two and the noise started getting decreasing as the engine spun down. The race directory spoke loudly, spent 10 seconds of thanking the armed forces, we had a quick round of applause and then were off.

The race starts on a curvy section of the City Island road (next to the mini train station) and heads counter-clockwise around the island. It was a fairly congested for the first 100 yards as people negotiated the curves and a few very large puddles. After about three minutes of the “is this too fast? too slow? too fast?” internal monologue I started settling into a good rhythm.

We hit the Walnut Street Bridge at about the three quarters of a mile. I noticed everyone was running on the small concrete walkway (as opposed to some people running on the iron lattice which happens in most Harrisburg races). I looked ahead and saw a vehicle parked on the other end of the bridge that was blocking most of the non-walkway part so figured I would stay on the concrete too. There were a few people walking from Harrisburg to City Island across the bridge who I felt bad for. Due to a timing misfortune, they were being passed by hundreds of people flying past them in the opposite direction all crowded into 5 feet of space. I crossed the 1 mile mark right as we exited the bridge in 6:26.

The course took an immediate right turn and we ran down to the walkway on the lower-level. I don’t really like this turn in races since you have a nice downhill and then need to make an immediate150 degree turn to head up-river which means you burn some time slowing for the turn.

We ran about 3/4 of a mile up-river. I was feeling good here and keeping a relatively even pace hovering around 6:20/mile. We next had to run up a 40 foot or so hill to get back to the upper-level. As I was starting up the hill the leaders came zooming down past me. I tried to keep an even effort going up and just after the hill, but I could tell the pace was dropping off. We ran a quick loop around the garden and then it was my turn to head back down the hill to the lower-level again. I hit the two mile mark just before heading down the hill with a 6:29 split.

The course then headed back down the river to the Walnut Street Bridge. I was definitely starting to fatigue and was concentrating on maintaining good form and cadence. I passed a handful of people here and had a mini-battle with one runner where I passed her and then she passed me back a hundred yards later. I finally passed her again a short distance from where we had to climb back up to street level. I was trying to push as hard as I could while running back across the bridge, but the pace and the last hill had taken most of the fight out of me. I was able to pick it up a little bit and ended up finishing in 19:51, good for 18th place out of roughly 215 people.

My Garmin measured the course at 3.08 miles, so it lost about 150 feet, which isn’t surprising given the sharp turns. What really surprised me was that my auto-recorded splits for miles two and three were both 6:29.36. Now I know that if the Garmin had been able to measure the corners correctly I would have not had splits identical to within a hundredth of a second, but it was kind of funny to see those times. Apparently I’ve got even pacing down to a science, now just need to work on getting faster.

Overall I was pretty happy with the race effort.

Motionbased

Sat: ran with Jon Laman, had to cut it off at 4.5 miles due to hamstring tightness (pain level 5 out of 10)

Sun: 20 mins easy on elliptical at home + stretching.  No pain.
Mon: off

Tues: off + PT appointment. Performed a bunch of leg exercises. Felt uncomfortable that the only PT had to split time between me and another patient.

Wed: bike 30 mins at LA Fitness + stretching.

Thu: bike 45 mins at LA Fitness + stretching. Ran slowly for 5 mins w/o pain.

Fri: racquetball for an hour was the plan but stopped at 15 mins due to mild hamstring tightness (pain level 2 out of 10).  I was annoyed at myself for neglecting a warm-up.  After stretching the hamstring tightness pretty much went away. Later that night I  came across a Runner’s World article on recovering from hamstring problems.

On Saturday morning I was supposed to run with Jon and a running group (called “Crispy Balance“) he recently discovered. I was fairly nervous about running since it felt tight the night before, but ended up deciding to give it a shot.

I met up with Jon and the “Crispies” at City Island at 9. The group was welcoming, laid-back, and well-organized. Meg, the woman hosting the run, had written route instructions for runs of a variety of distances, ranging from 5.7 to 11 miles and had stashed water and Gatorade along the course in several locations. Introductions were made, then we all posed for a group picture and headed off. I wasn’t sure how far I was going to go since I was worried about my hamstring tightening up again. I made sure to do a bunch of movement to help warm-up before starting and concentrated on keeping my pace really slow.

I enjoyed both the running route (there were a couple of awesome views of Harrisburg from the hills), and the nice group of people. Several were really into ultra-marathon and trail racing. One woman mentioned she celebrated turning 50 by running in the JFK 50 mile trail race.  Very inspiring.  A lot of group members were training for the Flying Pig Marathon in Cincinnati, which I thought was really cool since it meant a bunch of people on a similar training schedule.

I felt much better than I had feared earlier, and ended up running 10 miles. My leg felt complete fine until the last 1/2 mile or so, when I could feel a little bit of fatigue/tightness.  After the run I stretched for about 10 minutes and  felt great after that.

Sunday: 1 hour 10 mins on bike + stretching.

Plan for the week:  Bike 3  days and try to get in two 5-6 mile easy runs and a good long run on Sat or Sun.  I’m also going to slowly ease into some hamstring specific exercises to build back strength and flexibility.