Enterprise Cloud and All-You-Can-Eat Buffets

Enterprise Cloud and All-You-Can-Eat Buffets

What does Enterprise Cloud have in common with an all-you-can-eat buffet?

Delegation and Self-Service

As well as automating tasks to be run on a regular schedule to avoid manual handling, many tasks are being automated to allow the delegation of executing certain tasks to other humans.

Consider the case where you have 10 VDI desktops deployed. As common tasks come up, such as restoring files from snapshots, diagnosing performance issues or provisioning new desktops, it’s easy to jump in and take care of matters by hand. Take that number to 1000 and you’re likely going to start to see issues maintaining those by hand as you scale. Get to 10,000 or more and it’s an entirely different class of problem.

This doesn’t just apply to VDI — DevOps deployments and Enterprise server farms are seeing the same kinds of challenges as they scale too.

In order to scale past a few systems, you need to start to delegate some number of tasks to someone else. Whether that be a helpdesk team of some kind, or a developer or application owner, or even potentially the end user of a VDI desktop.

However, delegation and self-service are not just a case of dumping a bunch of tech in front of folks and wishing them luck. In most cases, these folks won’t have the technical domain knowledge required to safely manage their portion of infrastructure. We need to identify the tasks that they need to be able to perform and package those up safely and succinctly.


Consider a restaurant with an all-you-can-eat buffet. One of the nice ones — we’re professionals here. Those buffets don’t have a pile of raw ingredients, knives and hotplates, yet they’re most definitely still self-service.

You’re given a selection of dishes to choose from. They’ve all been properly prepared and safely presented, so that you don’t need to worry about the food preparation yourself. There is the possibility of making some bad decisions (roast beef and custard), but you can’t really go far enough to actually do yourself any great harm.

They do this to scale. More patrons with fewer overhead costs, such as staff.

DIY Self-Service

As we deploy some kind of delegation or self-service infrastructure, we need to:

  1. Come up with a menu of tasks that we wish to allow others to perform,
  2. Work out the safety constraints around putting them in the hands of others, and
  3. Probably still having staff to pour the bottomless mimosas instead of simply a tap.

We did introduce these two things in previous series’ of articles. In particular, #1 is a case of listing and defining one or more business problems, as we saw in the automation series.. For example, users that accidentally delete or lose an important file, might need a way to retrieve files from a snapshot from a few days ago. #2 above is referring to taking and validating very limited user input. In the restore example above, we’d probably only allow the user to specify the day that contains the snapshot they’re looking for and maybe the name of their VM.

Public Cloud

Self-service and autonomy are one of the things that Public Cloud have brought to the table at a generic level. By understanding the specifics of your own Enterprise, you can not only meet, but exceed that Public Cloud agility within your own data centre. This can also be extended to seamlessly include Public Cloud for the hybrid case.

Next Steps

As with each of these series, we’re starting here with a high level overview and will follow that up with an illustrative example over the coming articles. We’ll build on what we’ve learned in those previous series and we’ll again use the common System Center suite to get some hands-on experience. As always, the concepts and workflow apply quite well to tools other than System Center too.

To summarise, delegation and self-service are essential for most organisations as they scale. When used to safely allow autonomy of other groups, it can save you and your team significantly.

[Buffet picture by Kenming Wang and used unmodified under SA2.0]



Enterprise Cloud Orchestration recap

Enterprise Cloud Orchestration recap

This brief article hopes to summarise and collect the recent set of articles published around orchestration in the Enterprise Cloud.

  1. In our first article, we gave an overview of orchestration in the context of the larger automation umbrella and looked at is as a way to simplify the safe execution of automated tasks.
  2. Part two in the series looked at orchestration workflows (runbooks in System Center speak), using System Center Orchestrator 2016 as an example.
  3. Article #3 looked at a Microsoft PowerShell template for calling complex PowerShell functionality from within a System Center Orchestrator runbook.
  4. In our next article, number four, we looked at the use-case specific code. Our example used Tintri SyncVM to perform some innovative and efficient data-copy management for our Test/Dev folks.
  5. Finally, article five in the series pulled it all together and allowed us to execute the orchestration runbook, and our PowerShell activity, and see the results.

This series extended upon our automation series to take a business problem and create an agile and automated solution suitable for safely delegating to folks outside our core infrastructure group. This could also be scheduled for regular execution within Orchestrator.

Keep your eye out for the next series, which will look at putting this in the hands of the end user through a simple self-service portal.

[La grande salle de la Philharmonie de Paris image by Jean-Pierre Dalbera and used unmodified under CC2.0]

Orchestration for Enterprise Cloud part 5

Orchestration for Enterprise Cloud part 5

We’ve spent the past four installments in this series putting together a System Center Orchestrator runbook workflow to call into PowerShell to call Tintri Automation Toolkit cmdlets to do a bunch of stuff.

The stuff that it’s doing is solving a real business need for us — we want our developers to be able to test their code against a current copy of production data. Dump and restore operations are very expensive and error prone, so we’re taking advantage of Tintri’s SyncVM functionality to handle the data synchronisation for us. As we’ll see, this is going to take less than a minute to perform!

In this article, we’ll walk through executing this runbook and show how easy it makes the task. This simplicity makes it a great candidate for a task that can be delegated to someone with less in-depth knowledge (or access to) the cloud infrastructure. This is a big step forward toward self-service.

Orchestrator Web Console

If we now point our web browser at port 82 of our Orchestrator server (for example, http://scorch-2016.vmlevel.com:82/), you should be presented with the Orchestrator Web Console and you should see our new Runbook.


Select the runbook and click the Start Runbook button.


It will prompt you for the required input — simply the name of the developer’s virtual machine. It doesn’t request any information about the VMstore that the VM is stored on, it doesn’t ask for the production VM name, it doesn’t ask which snapshot to sync from and it doesn’t ask which virtual disks to synchronise. All of that is taken care of inside the runbook. This drastically reduces the number of places we could accidentally mess something up when we’re in a hurry or if we delegate this task to someone else.

Should something go wrong with the destination VM as part of this process, the SyncVM process we’re using takes a safety snapshot automatically, so at worst, we can easily roll it back.

We’ll enter our VM name (vmlevel-devel) and kick off the runbook job.


Next we’ll click on the Jobs tab and should see a running job.


If the job doesn’t have an hour glass (indicating it’s running) or a green tick (indicating success), it’s worth checking that the Orchestrator Runbook service is started on your runbook servers (check your Services applet). I’ve noticed that at times it doesn’t start correctly by itself despite being set to be automatically started:


After a little while (it takes about 45 seconds in my lab), hit refresh and the job should have succeeded. At that point, click on View Instances and then View Details to view the details of the job.


If we click on the Activity Details tab and scroll down, we can see the parameters of the Run .Net Script activity that calls our PowerShell code. If you look closely, you’ll see the variables we have defined. This especially includes out TraceLog variable, which you can see in the above output gives us a very detailed run-down of the process executed.

Given that this has succeeded, we’ve achieved our goal. Our developer VM has our developer code and OS on it, but has a copy of the latest production data snapshot. The whole process took less than 60 seconds and the developer is now up and running with recent production data — all without costly dumps and restores.

Try it for yourself and see.

[Ovation image by Joi Ito and used unmodified under CC2.0]

Orchestration for Enterprise Cloud part 2

Orchestration for Enterprise Cloud part 2

In our previous post, we set the scene for being able to use System Center Orchestrator for being able to bundle up automated functionality in a way that’s more consumable by others. We also defined the business problem we’re trying to solve and worked out which inputs we would rely on the user to provide and which things we will decide on their behalf.

In this article, we’ll start to look at the Orchestrator workflow and how to include some PowerShell automation as part of that. We’ll then follow up with some specifics around example PowerShell code, which has its share of peculiarities when being run under Orchestrator.

Orchestrator Workflows

System Center Orchestrator, like many orchestration tools, allows us to take automation modules, called Activities, and glue them together in a workflow called a Runbook. There are a bunch of pre-defined activities that ship with Orchestrator, and Microsoft provides extra activity bundles in a set of Integration Packs that are available for download. We’ll be keeping things quite simple in our example and will rely primarily on the Run .NET Script activity that comes standard.

Open Orchestrator’s Runbook Designer tool, select your Orchestrator runbook server and we’ll create a new runbook. In my example, I’ve called mine Prod-Dev Data Sync. Using the activities on the left, drag and drop the following activities into the runbook and connect them:

  • Initialize Data — this is where our runbook will start
  • Run .NET Script — this activity is where we’ll call out to PowerShell to do Tintri SyncVM magic
  • Return Data — at the end of the runbook, any returned data will be handled here and passed back to Orchestrator

It will end up looking like this:


We can far more complex runbooks than this, and indeed ours could be improved, but this will work as a starting point.

Next, right-click the Initialize Data activity, select Properties and click the Details tab. This is where we set the inputs that this runbook will take from the end user. We’ll add one string input parameter and we’ll call it VMname. It should look as follows:


We’re going to blindly pass this along to the next activities in the runbook, but it would be better to perform some kind of basic checks here to make sure that we’re operating on a developer VM and not a production VM. We’ll come back to that.

Next, right-click the Run .NET Script activity, hit Properties and again select the Details tab. This is where we’ll add our PowerShell script shortly. For now, set the language to PowerShell and leave the the script pane blank. Click on the Published Data tab. This is where we take any values returned our script and pass them to the next activity in the runbook.

We’re going to add three variables to be returned:

  1. ResultStatus as an integer
  2. ErrorMessage as a string
  3. TraceLog as a string

The names don’t matter too much, but the name you use in the Variable field will need to match the variable names we use in our script, and the Name field will be plumbed into the Return Data activity shortly. The result should look like this:


The plumbing for our runbook is almost done. We just need to take care of the data that we want to return and we’re then ready to start adding our PowerShell code.

First, right-click on the runbook’s title in the tab along the top, and click Properties, and select the Returned Data tab. This is where we’ll declare which parameters we plan to return. To begin with, these will match what we returned from the Run .NET Script activity earlier.


This defines the data we plan to return, but we still need to link that up with the previous activities in the runbook.

Right-click the Return Data activity, click Properties and select the Details tab. You’ll notice that it has already populated with the list of return parameters we defined a moment ago.

For each parameter to be returned, we need to subscribe to a piece of data published by earlier activities. In our case, we’ll right-click the text field, select Subscribe and Published Data. You should be able to find each of the variables we defined as output from the Run .NET Script activity earlier. Orchestrator will then fill in the text fields with template text that looks like ‘{ResultStatus from “Run .Net Script”}’ as seen below:


At this point, our runbook is pretty much complete. To summarise what we’ve done:

  1. Declared the VM name as an input parameter and subscribed to it
  2. Passed that VM name to a PowerShell script (still to come)
  3. Took the returned data from the script and returned that at the end of our runbook.

We’ll cover the PowerShell script in two separate articles — our next will cover the plumbing needed for PowerShell scripts in Orchestrator, and then the one after will look at the use-case specific code.

Ideas for improvement

The runbook we have will be pretty functional once we’re doing with the PowerShell component. But it is about as simple as it could be. Here are a couple of ways that we could insert some more activities into the runbook to make things more robust:

  1. Have an activity after the Initialize Data activity that checks that the virtual machine name is a developer VM, rather than production or some other application.
  2. Extend that to check that the user requesting the activity owns the VM being operated on. Most hypervisors don’t have a concept of an owner of a VM in this context, so we’d need to think of ways to map that ourselves.
  3. Check the output of the script and if the operation failed, automatically email us with the trace log data and anything else helpful.

Time permitting, we’ll try to tackle one or more of these once we have the initial project up and running.

[Orchestra image by aldern82 and used without modification under SA2.0]

Orchestration For Enterprise Cloud

Orchestration For Enterprise Cloud

In our last series, we looked at taking a business problem or need and turning into a piece of automated code. We started out by breaking the problem down into smaller pieces, we then put together a piece of sortamation to demonstrate the overall solution, and then made it more modular and added some error handling.

In this series, we’re going to extend upon this and integrate our automation into an orchestration framework. This approach will apply to any orchestration framework, but we’ll use System Center Orchestrator 2016 in our examples.

Why Orchestration?

Primarily for delegation of tasks. We may have written a script that we can run to perform some mundane task, but for us to be able to successfully scale, we need to start putting some of these tasks into the hands of others.

As a dependency for delegation, we also want to use automation and orchestration as a way to guarantee us consistent results and general damage prevention. Sure, we could just allow everyone access to SCVMM or vCenter to manage their virtual machines, but that’s a recipe for disaster. Orchestration gives us a way to safely grant controlled access to limited sets of functionality to make other groups or teams more self-sufficient.

The Process

Much like in our earlier automation series, we want to start by defining a business problem and breaking it down to smaller tasks from there. We want to extend this too to include the safe delegation of this task and this will include thinking carefully about input we’ll accept from the user, information we’ll give back to the user, and what kind of diagnostic information we’ll need to collect so that if something goes wrong, we can take a look later.

The fictitious, but relevant, business problem that we’re going to solve in this series is a common DevOps problem:

Developers want to be able to test their code against real, live production data to ensure realistic test results.

The old approach would be to have someone dump a copy of the production application database and restore the dump to each of the developers’ own database instances. This is expensive in time and capacity and can adversely impact performance. It’s also error-prone.

Instead, we’ll look at making use of Tintri’s SyncVM technology to use space-efficient clones to be able to nearly-instantly make production data available to all developers. We’ll do this with some PowerShell and a runbook in System Center Orchestrator.

We can then either schedule the runbook to be executed nightly, or we can make the runbook available to the helpdesk folks, who can safely execute the runbook from the Orchestrator Web Console. [Later, we’ll look at another series that shows us how to make this available to the developers themselves — probably through a Service Manager self-service portal — but let’s not get too far ahead of ourselves]

Core Functionality

Our production VM and developer VMs all have three virtual disks:

  1. A system drive that contains the operating system, any tools or applications that the developer needs, and the application code either in production or in development.
  2. A disk to contain database transaction logs.
  3. A disk to contain database data file logs.

In our workflow, we’ll want to use SyncVM to take vDisks #2 and #3 from a snapshot of the production VM, and attach them to the corresponding disk slots on the developer’s VM. We want this process to not touch vDisk #1, which contains the developer’s code and development tools.


For us to use SyncVM (as we saw previously), we need to pass in some information to the Tintri Automation Toolkit about what to sync. Looking at previous similar code, we probably need to know the following:

  • The Tintri VMstore to connect to.
  • The production VM and the snapshot of it that we wish to sync
  • The set of disks to sync
  • The destination developer VM

In order to limit potential accidents, we probably want to limit how much of this comes from the person we’re delegating this to. In our example, we’ll assume a single VMstore and a single known production VM. We’ve also established earlier that there is a consistent and common pattern for the sets of disks to sync (always vDisk #2 and #3). The only parameter where there should be any user input is the destination VM. The rest can all be handled consistently and reliably within our automation.


After this has been executed, we’ll need to let the user know if it succeeded or failed, along with any helpful information they’ll need. We’ll also want to track a lot of detailed information about our automation so that if an issue arises, we have some hope of resolving it.


So far, we have again defined a business need or business problem (synchronisation of production data to developer VMs), defined the set of inputs we’ll need and where those will come from, and we’ve defined the outputs.

In the next installment, we’ll start to get our hands dirty with System Center Orchestrator, followed by PowerShell and the Tintri Automation Toolkit for PowerShell.

[Orchestra image by Sean MacEntee and used unmodified under CC2.0]

Hyper-V I/O Multi-tenancy for Private Cloud part II

Hyper-V I/O Multi-tenancy for Private Cloud part II


Last week, we looked at the use of user-based access controls to keep storage tenants on separate SMB shares for Hyper-V hosted private clouds. This week, we’ll expand on that through separation using VLANs.

VLANs themselves are pretty straightforward as a way to partition a physical network into multiple virtual, isolated networks. Where this can get a little murky, is when we put Kerberos and SMB into the mix. That’s the area that we’ll aim to clear up here this week.


[Freecell image by mahemoff under CC by 2.0]

Networks, Subnetworks and Names

We can create separation by creating a distinct VLAN for each tenant’s compute. Alice might be on VLAN 11 and Bob might be on VLAN 12. The Tintri VMstore serving I/O for these tenants would have its data interface on each tenant’s VLAN.

Each of these VLANs would be assigned a different subnet and the VMstore would have an IP address on each VLAN. Each tenant can communicate with the VMstore’s data interface, but can’t communicate with each others’. Easy.

So what’s the problem?

As we saw in Kerberos Fundamentals a few weeks ago, we have a DNS name that maps to a data IP address on the VMstore and has an associated Service Principal Name (SPN) in Active Directory. It might be tempting to have that DNS record resolve to all of the data IPs assigned to the VMstore (for each VLAN), but that won’t work in a lot of cases. Most DNS servers by default will return those IP addresses in a round-robin rotation. Using our Alice and Bob example, Bob’s compute trying to resolve the DNS name of the Tintri VMstore would have a high likelihood of getting an IP address on Alice’s VLAN/subnet, which is not reachable by Bob.

Split-DNS or a DNS server that responds differently to queries based on source IP address (such as views in ISC bind) are valid ways to get past this, but can be more unwieldy to manage and troubleshoot.

Extra SPNs

By creating a DNS record for each tenant data IP address and an associated SPN, we have a solution to the problem where all components (DNS records and SPNs) are kept with each other. The following table shows an example:

Data path DNS name IP address and netmask VMstore SPN
alice-data.vmlevel.com cifs/alice-data.vmlevel.com
bob-data.vmlevel.com cifs/bob-data.vmlevel.com


You’ll notice that when deploying the VMstore, we assign the SMB data path hostname in the UI so that the VMstore knows the name used by default for its SPN. It is not necessary to tell the Tintri VMstore about the subsequent SPNs or data path names for Alice and Bob above.

So as you can see, it isn’t necessarily obvious, but it isn’t very difficult to use SMB and Kerberos in a multi-tenant environment where VLAN and subnet separation has been employed.

Hyper-V I/O Multi-tenancy for Private Cloud part I

Hyper-V I/O Multi-tenancy for Private Cloud part I


This is the first of a two part series of articles that will talk a little about multi-tenancy in for Hyper-V in a Hosted Private Cloud deployment. Not all Private Clouds need this level of separation, but for the cases where it’s a requirement, this series aims to help.

In the context of Hyper-V, I/O Multi-tenancy is where we serve storage for VMs over SMB 3 to two or more clients (tenants) where they are not aware that the storage is shared with others. Essentially there needs to be some degree of security and privacy between tenants — especially when it comes to traversing the data network.

This first article will deal with SMB shares for user-based access control as separation, and we’ll extend upon that in the second article, which will discuss VLANs and how they impact Kerberos authentication.

In the case of VM-aware storage, once inside the storage array, each tenant’s I/Os are kept separate from each other through either implicit or explicit VM-level QoS. That might be a topic for another day.


[Public Domain image taken by Bart Lumber]

An Example

To keep this topical, let’s say that we’re a hosting service for US Presidential Candidates and their campaign folks. Using standard fictional security characters, we’ll call these two candidates Alice and Bob.

Each has access to a Hyper-V compute platform and manages their own VMs, which are stored on their Tintri VMstore. Both groups need the ability to manage and maintain their own virtual machines and applications, but neither group should be able to see each other’s, which might contain sensitive emails or tax returns. Bob should see Bob’s stuff and Alice should see Alice’s. Neither should see anything else.

Separate Shares

By creating an SMB share (or more) each, we can grant access to only the parties that need it. Probably the easiest way to do this is with the (free) Tintri PowerShell Toolkit.

Consider the following illustrative example of creating a share each that is accessible by the right compute nodes for both Alice and Bob. The VMstore in this case being vmstore01.vmlevel.com.

PS:\> Connect-TintriServer vmstore01.vmlevel.com
PS:\> New-TintriSmbShare -Name AliceVMs
PS:\> Grant-TintriSmbShareAccess -Name AliceVMs -User alice-hv01$ -Access FullControl
PS:\> Grant-TintriSmbShareAccess -Name AliceVMs -User alice-hv02$ -Access FullControl

PS:\> New-TintriSmbShare -Name BobVMs
PS:\> Grant-TintriSmbShareAccess -Name BobVMs -User bob-hv01$ -Access FullControl
PS:\> Grant-TintriSmbShareAccess -Name BobVMs -User bob-hv02$ -Access FullControl

Above, we create a connection to the VMstore, we create a new SMB share for each tenant (Alice and Bob) and grant each tenant’s compute nodes access to the correct share.

VM Administrators

In single-tenant deployments, in addition to granting hosts/clusters of compute nodes access to the storage, it’s necessary to grant your VM Administrators access to the VMstore to manage storage and create VMs. This is most-simply done by assigning their Active Directory group(s) the Tintri Super Admins RBAC role, which allows them the administrative access that they need.

It might seem like a straightforward case of simply assigning both Bob and Alice’s VM admins the same role. However, that would grant both parties access to management functionality over the VMstore, and would grant each other access to the other’s VM data — the very thing we’re trying to avoid.

Instead, we just grant both parties explicit access to their own share using the same PowerShell cmdlets above, but naming Active Directory groups containing Alice and Bob’s VM administrators:

PS:\> Grant-TintriSmbShareAccess -Name BobVMs -User "Bob Admins" -Access FullControl
PS:\> Grant-TintriSmbShareAccess -Name AliceVMs -User "Alice Admins" -Access FullControl

And there you have it. Bob’s team can now create VMs against the share \\vmstore01-data.vmlevel.com\BobVMs and Alice’s team can do the same from their compute nodes against \\vmstore01-data.vmlevel.com\AliceVMs. Neither canĀ  see each other and our own admins can manage both should the need arise.


It’s possible with SMB 3 shares and access control lists to implement the foundation for a multi-tenant environment. In the next article, we’ll expand on this to include network-level multi-tenancy without breaking Kerberos.