How to use PVS without having to wait for file copies

By:Rich Brumpton

---

As I have mentioned before I love PVS. The entire concept of live booting a large number of XenApp servers off of a single image simply makes sense. Works great for VDI or Desktop OS streaming too and one of the facts of life on all three of these scenarios is that you are going to have to do updates to your images. No matter how good a job you do a separating your apps from your image using application virtualization tools, you will still have to deal with OS patches and other required changes.

Typically, this means that you are versioning anywhere from once a month up to several times a week depending on the life-cycle stage and stability of the requirements for the image. In this post I will outline an approach that I have started using in my lab and show how to make these updates simple and without requiring a wait for file copies.

The general principle behind this update method is the same as what Citrix uses themselves with XenServer. "If you have to copy something big, you might as well ask the storage system to do it for you." This way the data never has to leave the SAN to be copied, and if you have a SAN that supports it you can even do thin cloning for instant writable copies of data.

Because that is what I wanted, the ability to update a vDisk image without having to copy the file. In my lab environment I often have to version a vdisk several times in one day while working through several generations of a build and the file copy process was limiting how many versions I could iterate through in a day. I also have in my lab right now a NetApp SAN that provides me with thin cloning, and one day during a demo I hit on the way to make this all easier.

During a presentation that included a demo the NetApp SnapDrive component (more on this later) I was asked by a member of the audience "How hard is it to take a snapshot and mount it?" I was demoing SnapDrive on my PVS server so I said "easy, like this" and demoed taking and mounting a snapshot about as fast as I could run through the wizard. After this demo I got to thinking, I had just created a copy of 12 different PVS images in seconds, not the ~1hour copy time I usually faced for a full versioning of all of my vDisks. This in turn led me to investigate how I could make this work in production, and the result is:

How to use NetApp and PVS to make instant version changes
The Key to making this work for me is the ability of the NetApp FAS to take a snapshot and make it writable. There are two mechanisms that can do this for an iSCSI LUN (which is where I will focus) hosted on NetApp storage, FlexClone and LUN Clone. While it's possible to do what I'm about to describe with LUN Clone, it will take several more steps or more extensive scripting to get the same ease of use as FlexClone coupled with SnapDrive, which is the solution I'll focus on here.

FlexCloneas mentioned above has the ability to take a snapshot of point in time data and make a writable version of it that only grows by the deltas between the original and modified version. The other piece to the solution from NetApp is a piece of software that you install on your servers that connect to the SAN called SnapDrive. This software component snaps into computer management and allows you to manage the storage attached to the server from the server itself. This is a very easy way for server administrators to take advantage of features like storage snapshots, provisioning, and cloning without having to understand the underlying SAN operations.

With these components in place you can manage versions of images on provisioning server like so:

Preparation
We have to build the environment properly to allow for this update method, so we begin by setting up a new store. Note that in this model you may have a large number of LUNs per server which is why we are using folder mount points instead of drive letters.

1. Create a thinly provisioned FlexVol on the NetApp Array for use by PVS.
2. Create a folder on the PVS server to server as a container for multiple vDisk stores (i.e. c:\stores)
   
3. Create an iSCSI LUN and attach it to your PVS Server using SnapDrive and mount the LUN in a subfolder of your stores folder (i.e. c:\stores\WindowsXP.001)
   
4. Use the PVS Console to create a new store pointed to this new LUN
5. Create a new vDisk or copy an existing one to this location
6. Assign this initial version to devices and use normally

Versioning
The actual versioning process is used for each version of the vDisk throughout it's lifecycle. The longest part of this process will be the actual updating process in the VM.

1. Use SnapDrive to clone this LUN and mount under a new subfolder (i.e. c:\stores\WindowsXP.002)
2. Create a new PVS store and import existing vDisks
3. Change vDisk properties to Private mode, change version number (if using automatic updates) and assign to an update VM.
4. Use update VM to apply updates
5. Change vDisk back to standard mode and then assign to dev, test or prod machines as appropriate

Retirement
Because the child clone relies on it's parent, we do have to take steps to remove old versions cleanly.

1. "Split" the child clone from the parent
2. Ensure that all vDisks in the store are unused and unlocked
3. Delete the old store
   
4. Once this split is complete, delete the parent LUN and mount folder.

As you can see this update method, is not too unlike a traditional file copy update method in the number of steps required, but is much faster. The other really cool thing is that many of these steps can be scripted. Both PVS and SnapDrive provide scripting interfaces that can be used to perform many of these commands either on demand or on a regular basis. In fact I'm working on a variation of this for a customer who is looking for a way to script the behavior of PVS to allow scheduled updates trough SCCM. This process is to be completely automated up until it is tested and released into production by the administrators. The process for them would be very simular except instead of cycling ever upward through version number for their folders, they would have a series of folders that are re-used on a monthly basis (i.e. vDisks.1 to vDisks.4 for a weekly update schedule.)

The other cool thing about this solution is that you don't have to sacrifice performance in order to save space and time. Because the NetApp system is aware that the 2+ (virtual) copies of a given block in different thin clones are really the same physical block, it can simply cache it once, and avoid having to go to the disk for new instances of the same vDisk. This means that not only are you not slowing down the SAN by making a big file copy, your new Disk image is already in the cache of the storage array!

This kind of true synergy between vendors really gets be excited. To be able to take these features and tie them together ourselves without relying on them to do the integration for us by using a product like WorkFlow Studio is simply amazing.

More information on Provisioning Server


Read more!

NetApp is Generation V storage!!!

By:Rick Rohne

---

It seems that everyone has a favorite hypervisor, and the tech battles between which hypervisor is better will probably go on for the next few years.

One detail that seems to be overlooked in almost every environment that is the storage that hosts the VM's. NetApp is the only major player in that market "that I can think of" which (by luck or maybe some talented forward thinkers) has invested a lot of research and development around virtualization. NetApp uses De-Duplication and Flex Volumes to reduce the VM storage for your favorite hypervisor. I recently finished a project that involved hosting XENServer VM's using NetApp storage and I thought I would share my experiences...

Project Plan

---

The goal was to virtualize all 30 servers into 5 Dell Servers and a single Storage Unit. To accomplish this, I went with XENServer 5.0 embedded edition using a NetApp FAS2050 to store all of the data.

The Netapp FAS2050 is an Active/Active Clusterable Storage Appliance. When I scoped the project, I went with a total of 12 disks. Of course, at the time, I kinda forgot that each Filer will have to have it's own disks. So I had to split the disks in half 6 for each filer. At first, I was a little taken back by this but it actually worked out for me in the long run. Since I was limited on the disk space, I went with a RAID 4 on each aggregate instead of RAID DP. This gave me 4 usable disks, with one parity and one spare per cluster. So in essence, I had approximately 908 GB of usable space on each cluster.

The plan was to separate the VM's from the usable data, this is of course recommended by NetApp and Citrix because it allows your data de-duplication and your VM de-duplication top be treated independently. It also splits the Disk I/O in a way that data read and writes do not affect Server read and writes and vice versa. To accomplish this plan, I would actually have to have two seperate aggregates.

Filer configurations

---

I created one filer on node1, Aggregate1 to hold all of the CIFS file share data along with the Exchange databases. The connectivity to the data was served up using the CIFS protocol and built-in iSCSI. The placement of the Exchange Database on the same disks as the CIFS file systems basically leveled out the I/O requirements of that Filer. This also made backing up the data easy to do because all of the changed data was sitting on a single filer that was directly attached to an HP Tape Library. The total available disk space for the aggregate was about 900Gb. I was working with a little under 900 GB of data, so getting it to fit on that small chunk of disk was going to be tricky (so it seemed until I ran the Deduplication jobs).

The second filer was used to host only VM images. I had well over 1.5 TB of data requirements, and had to fit all of that into a 900GB Aggregate. To fit all those VDI's into that small space, I decided to use the XenServer adapter for NetApp Data ONTAP. This gave me the option to do thin provisioning, integrated snap shots and FAS de-duplication which would later reduce my storage on this aggregate down to 700 GB of used space (over 50% reduction in storage utilization).

XENServer Configuration

---

When you setup the XENServer adaptor for NetApp, you have the option to select the number of FlexVols to use on your aggregate. Here are the basic guidelines for setting up FlexVols:

The Number of FlexVols to use is by default 8. This can be changed to a number in the range 1 - 32. The number of FlexVols to be used is based on:

Increase the number of FlexVols if the data center will have:

• A lot of VMs that will be Snapshot often



• A lot of VMs with multiple VDIs

Decrease the number of FlexVols if the data center will have:

• SnapMirror replication to a Disaster Recovery Site
  

Now, I started with the default FlexVol size, which basically created a single Storage repository which consisted of 8 FlexVols on the NetApp. When using the XENServer adaptor for NetApp, you are given the decision to use thin provisioning and to turn on FAS de-duplication or use thick provisioning only. I chose to use thin provisioning and FAS de-duplication, which allow you to use the following features:

• Fast cloning of VDIs



• Fast Snapshot of VDIs



• FAS data de-duplication

I created a few Windows Server 2003 and 2008 templates to use when deploying new servers and deployed new servers using these templates (Which basically performs a fast clone in the back ground). One thing I noticed right away was the Fast Cloning gave me well over 75% data deduplication. So in essence, I wasn't really using too much disk space to add all these new servers.

P2V and Disk Allocation

---

The next part of the project was to perform a P2V on all of the remaining servers that were already running in the environment.

When you P2V a server into the environment, XENserver allocates all of the space on the NetApp while you are performing the import. You have to rely on Data De-Duplication to actually go around after the fact and perform the de-duplication job. There are a couple things that I found out when performing P2V's into XENServer.

1. If possible, try to keep the P2V's disks on their own storage repository. P2V's will not de-dup as well as fast clones. This is because the machines have less in common with each other and the fact that the blocks on the disk may actually be misaligned. (you can read the Best Practices Guide to determine if your VM's are misaligned.



2. Manually run a ASIS job (De-Duplication Job) after every P2V. This will gain the actual space back so that you have more room for the next P2V.



3. Try to eliminate white space (Free Space) if possible. Free Space actually gets allocated to the VDI, which basically grows your allocated space very fast (more on this in a second). XENConvert does not include a way to reduce the free space on a drive, so for some servers, I actually used VMWare Workstation to perform the P2V so that I could actually reduce the drive size. I recommend using PlateSpin for this process as it integrates directly into XENserver (And it reduces the downtime during the migration).



4. XENServer allocates all of the space in the VDI, even if you perform data de-duplication. This was a little shocking at first, because by my 15th P2V job, it appeared that I was out of space and I had 15 more to go!!! XenServer will actually prevent you from allocating more space than what is physically available on a per-storage repository basis. You can easily work around this by creating another Storage Repository. XENserver will see this Storage repository as a clean slate. Now of course, remember that once you start doing this, you are actually in the RED zone. The new data that you are adding is being allocated to previously freed up blocks that have already been de-duplicated. You should monitor the allocated space using NetApp monitoring tools once you get into the RED zone.


5. You can check the size of the allocated space using XENCenter. If you click on the SR, you will note the size of the repository reported in the Right pane. "Size: 672 GB used of 908 GB total (802 GB allocated)"What this means is you have 802 GB allocated to a 908 GB aggregate, however you are only using 672 GB. Once you have allocated 908 GB you will no longer be able to add new disks (however writes can continue to happen on the already allocated disks). If you must add additional disks, you will have to create a new SR or add physical disks.

NOTE: When the New SR is created now, it should read...Size: 672 GB used of 908 GB total (0 GB allocated)

Project wrap up

---

In wrapping up the project, I was able to get more than 50% data duplication on the XENServer VDI's and about 30% deduplication on the CIFS volume. The whole project reduced the datacenter footprint from 6 racks to one rack. The systems were more manageable and more importantly, the servers were portable which allowed for the entire data center to be backed up and shipped off-site or moved to different physical servers running XENServer 5.0.
I am impressed at how easy it was to setup, but there were some speed bumps that I did run into. Overall, the project was a sucess.

Now let's examine Data Deduplication

---

NetApp De-Duplication is basically just that... Data Deduplication. Similar to the way that compression works, De-Duplication works at the Block level to basically remove duplicate blocks in the file system and create pointers to a single block in its place.

To accomplish this, NetApp provides a process called ASIS which provides data de-duplication within the volume. In short, Netapp uses the WAFL file system to look for any 4kb blocks of data that is stored in more than one location "within a single VM or across all VM's" and consolidates them into a single 4kb Block instance.

Now that's all fancy talk, but the reality is this... Almost every VM that is hosted on your SAN is generally running the same operating system, has the same base applications such as Anti-virus and management software, and has very typical update cycles. Because of this, you are generally storing the same data over and over on every VM and probably adding more and more disk to accommodate for the VM sprawl!

By eliminating redundant data objects and referencing just the original object, an immediate benefit is obtained through storage space efficiencies. This reduces the initial storage cost and allows more time before you have to add capacity to your existing storage unit.

The de-duplication job is run on a schedule on a per-volume basis. The de-duplication process itself can negatively impact the performance of a production environment, so you may choose to run it over the weekend or manually. The job may take anywhere from 30 minutes to a few hours depending on the amount of duplicated data that is on the volume. I found that the best time to run the de-dup job is on a weekend after a major P2V conversion and approximately once a month after the fact.

What About thin Provisioning

---

Thin provisioning uses the same concept as data deduplication, however, there is no process that runs in the back ground. When you take a snap shot or take a fast clone, you are basically creating new pointers to the exsisting system and locking those pointers until the data is deleted or changed. Netapp still uses the WAFL file system to consolidate any 4kb blocks of data that is stored in more than one location. The main difference, is there is no process that runs in the background to clean this up.

For performance reasons, you may choose to separate volumes and aggregates so that the physical disks that hold some VM's do not impact the performance of others. The best way to get the most de-duplication in your environment and still have some kind of separation is to ensure that all VM's on a volume share a common installation source. For instance, Thin provisioned clones should be hosted on a one volume, while P2V VM's should be hosted on their own volume. If you have different flavors of operating systems, you can place each group of OS's on separate volumes. Finally, try to host the read write data outside of the Operating system image and on its own volume.

Both Fast Clones and FAS deduplicaiton can be used together to get the best storage consolidation possible.


50% Less Storage Guarantee

---

NetApp has also announced a 50% less storage guarantee! If you don’t use 50% less storage with NetApp or reduce your data by 35% on non-NetApp storage with NetApp V-Series, then NetApp will provide the additional capacity to meet the shortfall at no additional charge. Read the details here:
http://www.netapp.com/us/solutions/infrastructure/virtualization/guarantee.html

NetApp Best Practices Docs

---

Finally, I thought I would share some of the best practices documents related to Virtualization and NetApp!

Microsoft Hyper-V Storage Best Practices
http://media.netapp.com/documents/tr-3702.pdf

XENServer Storage Best Practices
http://media.netapp.com/documents/tr-3732.pdf

VMWare VI3 Storage Best Practices
http://media.netapp.com/documents/tr-3428.pdf

NetApp does many great things for a Virtual environment whether you are using VMWare, XENServer, or Hyper-V. There is a nice blog that I've been watching that really dives deep into the technical details of pretty much everything in the NetApp Virtualization solution offerings. You can find that blog here http://blogs.netapp.com/virtualstorageguy/.

Thanks for reading!

Read more!

Why NetApp is a great choice for VDI

By:Rich Brumpton

---

While working with a customer on a large VDI architecture recently we were comparing the required storage across several vendors and after looking at the proposed solutions from several I was asked the question:
In regard to the configs I’m really surprised at the low number of spindles relative to the IOPS req[uirement]s. Can you please help me understand the PAM a little more?"

The short answer is that the PAM can greatly increase performance in an environment that is heavy on small random reads like VDI, but that is not the only technology that NetApp uses to help optimize the storage of VDI.


There are a few things working in NetApp’s favor to keep the spindle count low. To begin with on the NetApp system one or more large pools of disks (Aggregates) are created that allow thinly provisioned volumes to be created that are striped across the entire aggregate. These volumes then contain one or more LUNs, more on this part later.

The Raid Groups that make up these aggregates use RAID-DP which offers double disk failure protection like RAID6, but because the NetApp storage system always writes full stripes and never has to do the read, read, write, write operation that gives RAID5 it’s 4:1 overhead and a 6:1 overhead for RAID6. In fact since NetApp can write it’s metadata anywhere in the file system the RAID write overhead is 1:1, in fact the only place I have to calculate overhead on RAID-DP is for IOPS (I=P(N-2), where I is total raid group IOPS, P is single disk IOPS and N is the number of disks in the array) to account for parity disks.

Other technologies on the NetApp storage system combine to reduce the physical size of the working set including thin cloning and primary storage deduplication.

The first of these, thin cloning, allows a snapshot of a single master copy of a volume (FlexClone) or LUN (LUN clone) to be presented read-write to hosts. This appears to hosts as a separate full copy of the data, but in fact only the deltas between the old and new blocks are written to disk, all the common OS components that make up a good portion of the working set for VDI actually remain in the same, single location. This technology can be used using the NetApp Rapid Cloning Utility (RCU) for VMware View or when using Citrix XenServer as the host for a XenDesktop machine. In either case this allows the working set to be decreased from N*W to W+((N-1)*D)W (where N is the number of clones, W is the working set size, and D is the delta percent of change from the master.)

Data Deduplication also plays a role when more than one VM is stored in the same volume. This feature looks through a volume for duplicate data blocks and removes all but a master copy and places metadata pointers back to this copy for each other copy of the block. Like thin cloning this feature is available because of NetApp’s ability to store metadata anywhere within the file system and creating pointers is nothing unusual given the structure of the WAFL file system. In a VDI scenario Data Dedupe helps contain the size of the deltas between VM’s by removing duplicate blocks created by OS or software updates, but this is a batch process so short lived data structures may not benefit.

So now that we have reduced the size of the working set, let’s talk about the PAM card which is 16GB of DRAM on a PCI-E card coupled with FlexScale software to act as an intelligent read cache. For folks from the server world the PAM operates like an L2 cache on a processor as an accelerator between the controller RAM and data on disks. There are 2 modes that are interesting to us in this discussion, default mode and metadata-only mode. In default mode the PAM card caches ONLY small random reads and metadata. This allows a large majority of the pointers used for deduplication and thin cloning to be stored very close to RAM which will already be used to cache the MOST frequently used data and metadata. If thin cloning and deduplication are used intelligently with an optimized configuration this mode can be used to retain a large number of the random read blocks in the PAM, greatly reducing the amount of time that users have to wait for blocks to come all the way from disk. This is the mode that I would use with VMware View or persistent XenDesktop VM’s and this is the mode that helps the most with events like boot-storms in View and persistent VDI scenarios. Metadata-only mode is used when there is a large working set and there is no way that it can fit enough of it in the PAM to avoid simply churning through the cached data. Metadata is cached in the PAM while data blocks are not allowing instant access to metadata blocks and a shorter access time for data stored on disk. This mode is the one I would use with XenDesktop in which each VM can be configured to store its own Provisioning Services write cache on the SAN, but this cache will be unique to each VM.


RAID-DP: http://media.netapp.com/documents/wp_3298.pdf
VMware RCU: http://blogs.netapp.com/virtualization/2009/03/netapp-and-vmware-view-vdi-best-practices-for-solution-architecture-deployment-and-management-part-8.html
Deduplication: http://media.netapp.com/documents/tr-3505.pdf
XenDesktop on NetApp: http://www.citrix.com/site/resources/dynamic/partnerDocs/CitrixXD2.0withNetAppStoragePilotDeploymentOverview.pdf
PAM: http://blogs.netapp.com/storage_nuts_n_bolts/2008/08/performance-acc.html


Read more!