Bryan asked me to post this for him today. He's in the hospital with his wife and their new baby boy! Congratulations to Bryan and his family!
Welcome to the year 2010. The year had a very exciting start for me. On Dec. 30 we finished migrating the last database to our new SAN. This was the largest and most complex part of the project. Thanks to excellent planning by our database administrator, Lee Crain, the move went smoothly.
The completion of this SAN migration project brings increased reporting and inbound speeds, better redundancy and virtually unlimited expansion capabilities. I am really excited by the results we are seeing. We have observed 72,000 IO requests processed in a single second. Page life expectancy is considered adequate with a value of 300. At one point recently, it was at 300,000 (higher is better). Other performance metrics indicate our servers spend much of their day waiting for more work to do.
There was an issue with our hourly statistics rollups that developed after the migration. We’ve been dealing with an intermittent lock contention issue during the rollups. We hoped the new servers and SAN would improve the situation. Unexpectedly, we found the increased performance caused the system to trip over itself because it was running so fast. The net result was a tortoise and hare problem. Sometimes slow and steady wins the race.
We caught the issue quickly and made the necessary changes to resolve it. By changing how the rollup works, we are taking full advantage of the performance our new systems give us. This process is running 18 times faster than it did previously. We are speedy and steady. Several of our engineers: Kacie, Art, Myong and Lee put in a heroic effort to make these changes happen so quickly.
I’m very excited to start the New Year with the completion of this project.
Showing posts with label SAN. Show all posts
Showing posts with label SAN. Show all posts
Wednesday, January 6, 2010
Thursday, December 17, 2009
SAN Project Update
I am pleased to post that the second phase of our SAN project was completed successfully Tuesday night. After a full day of monitoring, things are running great. Everything went well with the move and the performance gains are awesome.
Coordinating everything within a short maintenance window can be a challenge. Our careful planning paid off though, and we had the Partner Center, Affiliate Center and Merchant Center back up within 45 minutes. This will be the last phase of the SAN project we complete until after Christmas.
Coordinating everything within a short maintenance window can be a challenge. Our careful planning paid off though, and we had the Partner Center, Affiliate Center and Merchant Center back up within 45 minutes. This will be the last phase of the SAN project we complete until after Christmas.
Tuesday, December 15, 2009
LinkTrust SAN Project
I'm excited to share some details about my latest project at LinkTrust, something we've been calling the SAN Project. Hopefully this will be the first of many posts on what is going on behind the scenes at LinkTrust.
My brain is always thinking about something. About a year ago it started pondering how we store all the data in LinkTrust. At the time we had about a terabyte of data and it was growing fast. I realized we needed to start looking at a new storage solution.
The problem for us isn’t so much that we have a lot of data. Our problem is we need to access the vast amount of data quickly. Performance and reliability are key to keeping LinkTrust running well for our partners.
These considerations lead me to begin looking at a SAN. In a nutshell, a SAN is a big black box full of hard drives that work together. It provides high-performance data storage with a high degree of reliability and redundancy.
We began the project by evaluating different SAN vendors. We had to separate what the sales people told us from what their products actually do. The importance of this decision meant we had to carefully look at all our options. In the end we decided to go with Compellent. Their product offers some very unique flexibility and optimizations that I felt made it our best choice.
Once we decided on a vendor the real work began. There are a myriad of configuration choices to be made when implementing a SAN. I really wanted to make sure we got this right. It was important to me we didn’t have any single points of failure.
To ensure every piece of the system worked well, we also upgraded our switching fabric to a new set of Juniper switches. I purchased new Cat 6 network cables to ensure we weren’t bitten by an old cable starting to give out. The devil is in the details as they say. Every little piece must be considered to ensure success.
Whenever we embark on a project I want it to make things better for our partners. It needs to be worth the time effort and money we spend on it. I also don’t want there to be a lot of pain associated with upgrades or changes we make.
I work hard and plan meticulously to ensure successful project implementation. With the efforts of my great team we successfully moved our first database to the SAN last week.
To ensure success we started relatively small. The database that was moved was selected because it allowed for a seamless move with no visible impact to our partners. At the same time, it had enough load on it we could see a visible benefit when it was moved to the SAN.
Even I was surprised to see we were able to index more than 20 million records in a matter of a few seconds with no impact to user performance. This was something our previous infrastructure couldn’t do.
There is still much work to be done. We’ve grouped remaining tasks into a series of phases to allow for minimal impact to our partners and maximum success. The next phase, which will begin this evening, will require we take the Partner Center, Affiliate Center and Merchant Center offline for about an hour. This is to ensure no changes are made to configuration settings while the move is occurring.
This project really started about nine months ago. Looking back it has been a long road. We can finally see the light at the end of the tunnel. We can tangibly see the benefits that this new infrastructure will bring to the LinkTrust system. We’ve been able to make the upgrades with minimal impact to our partners. I’m thrilled to see the success we’ve had with this project thus far. I’m sure I’ll only be more excited by the final outcome.
My brain is always thinking about something. About a year ago it started pondering how we store all the data in LinkTrust. At the time we had about a terabyte of data and it was growing fast. I realized we needed to start looking at a new storage solution.
The problem for us isn’t so much that we have a lot of data. Our problem is we need to access the vast amount of data quickly. Performance and reliability are key to keeping LinkTrust running well for our partners.
These considerations lead me to begin looking at a SAN. In a nutshell, a SAN is a big black box full of hard drives that work together. It provides high-performance data storage with a high degree of reliability and redundancy.
We began the project by evaluating different SAN vendors. We had to separate what the sales people told us from what their products actually do. The importance of this decision meant we had to carefully look at all our options. In the end we decided to go with Compellent. Their product offers some very unique flexibility and optimizations that I felt made it our best choice.
Once we decided on a vendor the real work began. There are a myriad of configuration choices to be made when implementing a SAN. I really wanted to make sure we got this right. It was important to me we didn’t have any single points of failure.
To ensure every piece of the system worked well, we also upgraded our switching fabric to a new set of Juniper switches. I purchased new Cat 6 network cables to ensure we weren’t bitten by an old cable starting to give out. The devil is in the details as they say. Every little piece must be considered to ensure success.
Whenever we embark on a project I want it to make things better for our partners. It needs to be worth the time effort and money we spend on it. I also don’t want there to be a lot of pain associated with upgrades or changes we make.
I work hard and plan meticulously to ensure successful project implementation. With the efforts of my great team we successfully moved our first database to the SAN last week.
To ensure success we started relatively small. The database that was moved was selected because it allowed for a seamless move with no visible impact to our partners. At the same time, it had enough load on it we could see a visible benefit when it was moved to the SAN.
Even I was surprised to see we were able to index more than 20 million records in a matter of a few seconds with no impact to user performance. This was something our previous infrastructure couldn’t do.
There is still much work to be done. We’ve grouped remaining tasks into a series of phases to allow for minimal impact to our partners and maximum success. The next phase, which will begin this evening, will require we take the Partner Center, Affiliate Center and Merchant Center offline for about an hour. This is to ensure no changes are made to configuration settings while the move is occurring.
This project really started about nine months ago. Looking back it has been a long road. We can finally see the light at the end of the tunnel. We can tangibly see the benefits that this new infrastructure will bring to the LinkTrust system. We’ve been able to make the upgrades with minimal impact to our partners. I’m thrilled to see the success we’ve had with this project thus far. I’m sure I’ll only be more excited by the final outcome.
Thursday, December 10, 2009
Hello from the LinkTrust Technology Department
Meet Bryan.
Bryan Call is the VP of technology at LinkTrust. He's responsible for overseeing the implementation and architecture of the LinkTrust system. When he's not working on the latest version of LinkTrust, he watches reruns of Star Trek or takes his wife ballroom dancing. We're excited to have him become our first contributor to our blog... and bring some exciting news that will help LinkTrust perform at an even higher level. You'll definitely want to keep an eye out for his posts about the new SAN and what that means for you.
Welcome to the blog, Bryan!
Subscribe to:
Posts (Atom)
Posts
