churchIT

Microsoft iSCSI Initiator v2.04

I just noticed that the Microsoft iSCSI Initiator v2.04 has been released. You can find it at http://www.microsoft.com/downloads/details.aspx?familyid=12CB3C1A-15D6-4585-B385-BEFD1319F825&displaylang=en

iSCSI Questions from a reader

I received an email from Alvin from Asia with a couple questions regarding our EqualLogic iSCSI SAN implementation. Although, I feel a bit like Dear Abby; I'm always glad to share what I've learned.

From Alvin:

This is Alvin from Asia Pacific region. I enjoyed reading your blog on the Church IT section. And it is good to read the happy news on the new baby. Congratulation!!

One particular interest of mine is to read your successful IP SAN implementation. Me and my friends would like to learn more from you.

Would it be possible for you to share with us:

- The reasons you choose IP SAN? (other than cost, e.g. what are the existing pains in existing IT infrastructure)

- Do you have any network/infrastructure diagrams for the pre-SAN environment? If yes, can you share with us?

- Is you did update the network diagram with the post-SAN implementation, can you share with us as well?

- Last, what is your plan to make good use of your IP SAN? Can you share with us your next 6 months plan?

Hope this email doesn't cause you too much trouble. As you might not aware, IP SAN is very slow in Asia, it is hard for us to find a relevant experience on our side. Any help we can get is greatly appreciated.

I chose an IP SAN because of its flexibility, feature set and growth potential compared to Fibre Channel (FC) or traditional Direct Attached Storage (DAS). FC is really expensive and does not offer the growth potential of a Lefthand or EqualLogic system. DAS is inexpensive but offers no flexibility and it makes disaster recovery harder to implement. An EqualLogic iSCSI box is a good middle ground that allows me to centrally manage the storage, expand as needed, virtualize servers easily, and move data offsite easily.

I do not have any diagrams of before and after the iSCSI SAN but the implementation is pretty simple. I added two HP gigabit switches (for redundancy) to the server rack and created a new VLAN that supported Jumbo frames. I added two Intel Pro 1000/PT network cards to the servers that will use the SAN. I then connected the EqualLogic box and the server network cards to the switches. Basically, I created a separate storage network that will only handle storage I/Os.

My plan for the next 6+ months is to virtualize everything in the server rack and to solidify a backup strategy for the SAN.

iSCSI and Dynamic Disks

A little heads up to those who are implementing an iSCSI SAN with the Microsoft iSCSI Initiator. Do not make the iSCSI volumes Dynamic Disks on your Windows Computers unless you are using an iSCSI HBA. I accidentally made 2 volumes Dynamic disks and spent many hours troubleshooting why those volumes would not mount on boot-up and would go offline, without notice. We are using Intel PRO 1000/PT adapters and I had a ton of problems with keeping a stable iSCSI connection. After some hardcore Googling, I stumbled onto this info from the "Microsoft iSCSI Software Initiator 2.x Users Guide":

Dynamic disks (applies to Windows 2000 and Windows Server 2003)

Configuring volumes on iSCSI disks as Dynamic disk volumes using the Microsoft software iSCSI initiator is not currently supported. It has been observed that timing issues may prevent dynamic disk volumes on iSCSI disks from being reactivated at system startup.

Hardware-based iSCSI initiators (iSCSI Host Bus Adapters or "HBAs") can typically make the devices that it connects to available much earlier during the system startup process than the iSCSI software initiator can. Therefore, iSCSI HBAs may provide support for dynamic disk volumes.

I missed those details on my first reading for the Microsoft iSCSI Users Guide but I hope this helps someone else avoid this same problem!

SAN Installed

We received the new EqualLogic PS100E SAN this Monday. We immediately installed it in the rack and powered it up. Physical installation was easy, just like every other piece of equipment in the rack, we did have to reroute some cables though. I had to setup the new Vlan for the SAN and have setup the IPs for all the adapters. The software setup was very easy and I did the setup twice. I tried the Windows Setup application first and then wiped the configuration and setup from the terminal console. To be honest, I liked the console setup better, but I think that is because I use telnet fairly often. Either way, setup is a breeze. I setup the SAN with RAID 50 and there are 2 hot spare drives, so my 3.5TB SAN has a usable capacity of about 2.5TB.

I connected a server to the SAN using one its built-in adapters, but it turns out that the server adapter does not support Jumbo frames and was very slow at file transfers. I will be purchasing new NICs that have TOE (TCP Offload Engine) and Jumbo frame support shortly. I will also take Alacritech up on their offer of trying one of their NICs versus an Intel or HP NIC that has TOE. Has anyone else done performance testing on TOE NICs on their SAN?

I am attaching a couple pictures of the SAN, racked and ready.

SAN Decision

After months of research and a month of talks with vendors and manufacturers, l I have made a decision. Drum roll please…… EqualLogic. Our PO was submitted Friday and we are expecting it to be delivered late this week or early next week.

Like I've stated in past posts, this has been a difficult decision but I feel very confident about it. I have ranked the solutions on order of appeal EqualLogic, LeftHand MPC, LeftHand NSM, D-Link and Siafu. The last week, I was only looking at the EqualLogic and MPC solutions and each has their own merits. EqualLogic could be setup by us and has the best internal redundancy and LeftHand has their Campus SAN solution and thin provisioning. I really liked the LeftHand Campus SAN solution, but the more I thought about it, the more I didn't think it was that big of an advantage, as it looked on paper. Any SAN solution, for the foreseeable future, would be installed the main server room, so physical separation for disaster recovery (DR) is not possible for us, this negated part of their advantage. I'd rather have my DR replicate to an offsite data center and both units support this requirement. To make the LeftHand solution redundant in the Campus SAN, we must synchronously replicate data between two units. This requires a 50% storage overhead on top of the RAID overhead inside the units; this never seemed like a storage efficient solution. Lastly, I was worried about supporting the system. Because it was highly recommended that the manufacturer install the solution and train us, I was very concerned about transferring knowledge in the future or someone else supporting it if I was on the other side of the country. EqualLogic on the other hand has multiple controller boards, fan units, hard drives and power supplies within each unit, so the odds of a single unit going down is minimal, and I am comfortable with that. I also was much more comfortable with the support requirements of the EqualLogic solution, so someone could come it behind me and manage the system.

Now it's time to prepare for the installation!

SAN Update 4

I've spent the week checking references and weighing the different SAN options available. This has been a much tougher decision than I thought it would be a few weeks ago.

I did talk with Debi Cooper with M&A Technology, along with Jeff Horne of D-Link. We had a good conversation about their technology. The D-Link DSN-3200 has 15 storage bays for SATA drives and a whopping 8 1GB ports. Redundancy is provided by a second DSN-3200 whose replication is handled by Symantec VERITAS Storage Foundation. You can setup volumes on individual disks, place a volume on a single RAID array, or place multiple volumes on a RAID array; so it is fairly flexible. As I understood, the Veritas Storage Foundation could asynchronously and synchronously replicate data to a remote and/or local unit for data protection and disaster recovery. I am a little apprehensive about the Veritas software, which will run on another computer and perform the synchronizing. I did talk to a customer who implemented a DSN-3200 about 3 weeks ago. They also looked at a Siafu solution but chose this solution because of its overall value. They have only 7 of the drive bays populated with 750GB drives and love the solution. This will probably not be the solution that I choose for our implementation because of expandability concerns and concerns about Veritas Storage Foundation. The DSN-3200 is very economical and I would still recommend it be placed on your research list if looking for a SAN.

I also asked for references from EqualLogic and Left Hand. I talked with Justin Smith at Second Baptist Church in Houston and also talked with Jason Powell of Granger Community Church. Both had great things to say about EqualLogic and both have had great experiences with the product. Second is only using their PS300E for databases and Granger has moved most of their data and 20 virtual machines to their PS100E. Both have had great experiences with the product and will be expanding in the future. Jason also talked about EqualLogic's responsiveness to the drive problems they encountered several months ago.

I also talked with Patrick Trela of Prestonwood Baptist Church; they are running a 19TB total storage LeftHand solution. Most of their units are NSM160s and they are running in a Campus SAN environment for disaster recovery. Patrick had great things to say about Left Hand and we had a good conversation. I didn't think Prestonwood was an apple to apple comparison since they have 3 times everything that we have so I asked for another reference. I was directed to Bob Samson IT Director of UT – Arlington Campus Library. They have been a Left Hand customer for 3 years and have about 10TB raw storage. They have had a few problems over the years but have had a great impression of Left Hand's support.

Tomorrow's the decision day. I have received about 10 phone messages the last 2 days in regards to the RFP. I know which direction I am going but I'll wait until I have a PO in-hand before announcing the decision.

SAN Update 3

The bid due date has come and gone, and I have bids from 5 vendors, for 5 different manufacturers, and 4 software platforms. These are EqualLogic, Left Hand, MPC, Siafu, and D-Link. Actually I am waiting final pricing from 2 vendors but I pretty much know the general price range of their bids. Aside from the final pricing, I am now at T-minus 7 days for a decision.

I spent an hour with my Tech Depot reps, Dan Kalagher and several of his colleagues from Siafu Software. According to their company history, this a new company, with their first installation 9 months ago. It is based off of Windows Storage Server. Since it is a new platform, it looks like they are still building out the feature set but they have several features in the SAN that are interesting. The first is the ability to encrypt some or all of the volumes on the SAN. I can see this being useful in some scenarios but it brings about a 20% overhead. The next feature is the ability to backup snapshots directly to tape, this is possible since there is an accessible Windows OS running the SAN. Although, I would worry about the backup software crashing the SAN. Lastly, there is a PXE boot server available to boot your diskless servers directly from the SAN. As I pointed out my SAN Update 2 post, booting from a non-HBA adapter would require a separate PXE server with DAS. This adds a layer of complexity, but an integrated PXE boot server would reduce that complexity. The system Siafu and Tech Depot are quoting is a SWARM 3000-HA.

I also received the D-Link bid today. It is a combination of a D-Link DSN-3200 and Symantec VERITAS Storage Foundation Software. I don't really have much to say about this at this point. I think this might be an "old fashioned" iSCSI SAN and that would explain why it is a least $10K less expensive than the other bids. I am waiting on the vendor to provide an explanation of the bid, before further review.

Lastly, I talked with EqualLogic for a little while and followed up with them on a few details, about their system. I wanted to make sure that I had an accurate impression of the functionality, they provide.

SAN Update 2

Last Thursday was a busy and very productive day. I spent about 2 hours with Rick Bruder and Keith Kincer of EqualLogic, in the morning. We discussed our current and future needs, their technology, other technology and did some whiteboarding. They brought a PS100E and did a demo from power up to managing the unit. I must say that I am really impressed with the PS100E hardware and the ease of use, of the software. I came away mostly impressed by the overall package.

I spent about an hour on the phone with Mike Hanes and Bill Horton of Alacritech. Alacritech specializes in TCP Offload Engine (TOE) NIC and iSCSI adapters. It was made clear to me that, this is not an iSCSI HBA, which presents itself as an SCSI adapter to the system. These are pure TOE NICS and they say that they get much better performance than most internal NICs or HBAs. I am really looking at them because of their combination of performance and the ability to boot from their card. I would like most of my new servers to be diskless, so booting from the SAN is a must. The only downside is that there boot is PXE enabled and would require a PXE boot server, and that is another moving part that can fail. They've offered a 30 day live trial of one of their adapters, so when I get the SAN running, I'll take them up on the offer.

I also had a conference call with PC Mall and Left Hand. I had a few questions about their bid of two or three NSM160s. The three NSM160 solution was over budget and lacks a 3 year warranty, so it will not work. I had a concern about the two NSM160 solution only having 8 SATA spindles in the SAN and the quote wasn't clear on the warranty options. I found out during our conversation, the quoted SAN did not come with the a 3 year warranty but a 1 year warranty. I've done the math and adding the warranty will put the project over budget. Tony Lagera of Left Hand tried to address my performance question but I am sold not on 8 SATA spindles. Tony will be in Houston Wednesday, March 21 and will meet with me to discuss my concerns.

A note to SAN buyers, do your homework. I know that is one of those "Duh!" statements but be thorough in your research, before budgeting. When I budgeted this project a few months ago, I looked at iSCSI solutions, but I never fully grasped the available technologies. I was really set on an "old school" EMC, HP or IBM iSCSI solution. I've seen the Left Hand and EqualLogic solutions and either one, with their faults, is still light years ahead of, most of, the competition. This includes Fibre Channel.

SAN Update 1.1

This is a quick mini update. MPC computer's regional sales representative coming out Wednesday morning to talk SAN and the new Micron. Thursday we have EqualLogic coming out to do an on-site demo of their product, with their regional sales representative and his engineer. I love when vendors make a good effort like EqualLogic is doing this week and it'll be fun talking technology with them.

Speaking of EqualLogic, Jason Powell reminded me of his posts detailing Granger's SAN research and deployment last year. He also has follow-up post about EqualLogic's excellent support. His excellent write-ups can be found here http://jpowell.blogs.com/jason_powell_church_it/san/index.html. I would recommend reviewing them if a SAN is in your future.

SAN Update

I've been on the phone and seen more of WebEx than I ever thought possible.  I have been in communication with 5 different vendors.  So far, I have looked at solutions from LeftHand, EqualLogic, MPC Computers, and HP.  I have two other vendors that have not given me any information on their offerings and recommendations, just an "I'm Interested".

Of the offerings that I've seen, LeftHand has the best overall software package.  Their Campus SAN is first rate, because you can place synchronously redundant SANs in multiple places, for limited disaster recovery. For full disaster recovery, we could place another SAN in a remote data center and asynchronously replicate data and snapshots to it. If these snapshots are virtualized servers, then we could

EqualLogic has the better internal hardware redundancy inside the box and the easiest setup, by far.

MPC is a custom hardware box that is running LeftHand software.

With all of the options I really like the virtualization options that are available, like being to move a VM fairly easily between servers. I also like that our Media Macs will be able join the SAN and performance will scale up, as we add storage units.