SAN project complete!

Bryan asked me to post this for him today. He's in the hospital with his wife and their new baby boy! Congratulations to Bryan and his family!

Welcome to the year 2010. The year had a very exciting start for me. On Dec. 30 we finished migrating the last database to our new SAN. This was the largest and most complex part of the project. Thanks to excellent planning by our database administrator, Lee Crain, the move went smoothly.

The completion of this SAN migration project brings increased reporting and inbound speeds, better redundancy and virtually unlimited expansion capabilities. I am really excited by the results we are seeing. We have observed 72,000 IO requests processed in a single second. Page life expectancy is considered adequate with a value of 300. At one point recently, it was at 300,000 (higher is better). Other performance metrics indicate our servers spend much of their day waiting for more work to do.

There was an issue with our hourly statistics rollups that developed after the migration. We’ve been dealing with an intermittent lock contention issue during the rollups. We hoped the new servers and SAN would improve the situation. Unexpectedly, we found the increased performance caused the system to trip over itself because it was running so fast. The net result was a tortoise and hare problem. Sometimes slow and steady wins the race.

We caught the issue quickly and made the necessary changes to resolve it. By changing how the rollup works, we are taking full advantage of the performance our new systems give us. This process is running 18 times faster than it did previously. We are speedy and steady. Several of our engineers: Kacie, Art, Myong and Lee put in a heroic effort to make these changes happen so quickly.

I’m very excited to start the New Year with the completion of this project.