Enterprise Cloud and All-You-Can-Eat Buffets

Enterprise Cloud and All-You-Can-Eat Buffets

What does Enterprise Cloud have in common with an all-you-can-eat buffet?

Delegation and Self-Service

As well as automating tasks to be run on a regular schedule to avoid manual handling, many tasks are being automated to allow the delegation of executing certain tasks to other humans.

Consider the case where you have 10 VDI desktops deployed. As common tasks come up, such as restoring files from snapshots, diagnosing performance issues or provisioning new desktops, it’s easy to jump in and take care of matters by hand. Take that number to 1000 and you’re likely going to start to see issues maintaining those by hand as you scale. Get to 10,000 or more and it’s an entirely different class of problem.

This doesn’t just apply to VDI — DevOps deployments and Enterprise server farms are seeing the same kinds of challenges as they scale too.

In order to scale past a few systems, you need to start to delegate some number of tasks to someone else. Whether that be a helpdesk team of some kind, or a developer or application owner, or even potentially the end user of a VDI desktop.

However, delegation and self-service are not just a case of dumping a bunch of tech in front of folks and wishing them luck. In most cases, these folks won’t have the technical domain knowledge required to safely manage their portion of infrastructure. We need to identify the tasks that they need to be able to perform and package those up safely and succinctly.


Consider a restaurant with an all-you-can-eat buffet. One of the nice ones — we’re professionals here. Those buffets don’t have a pile of raw ingredients, knives and hotplates, yet they’re most definitely still self-service.

You’re given a selection of dishes to choose from. They’ve all been properly prepared and safely presented, so that you don’t need to worry about the food preparation yourself. There is the possibility of making some bad decisions (roast beef and custard), but you can’t really go far enough to actually do yourself any great harm.

They do this to scale. More patrons with fewer overhead costs, such as staff.

DIY Self-Service

As we deploy some kind of delegation or self-service infrastructure, we need to:

  1. Come up with a menu of tasks that we wish to allow others to perform,
  2. Work out the safety constraints around putting them in the hands of others, and
  3. Probably still having staff to pour the bottomless mimosas instead of simply a tap.

We did introduce these two things in previous series’ of articles. In particular, #1 is a case of listing and defining one or more business problems, as we saw in the automation series.. For example, users that accidentally delete or lose an important file, might need a way to retrieve files from a snapshot from a few days ago. #2 above is referring to taking and validating very limited user input. In the restore example above, we’d probably only allow the user to specify the day that contains the snapshot they’re looking for and maybe the name of their VM.

Public Cloud

Self-service and autonomy are one of the things that Public Cloud have brought to the table at a generic level. By understanding the specifics of your own Enterprise, you can not only meet, but exceed that Public Cloud agility within your own data centre. This can also be extended to seamlessly include Public Cloud for the hybrid case.

Next Steps

As with each of these series, we’re starting here with a high level overview and will follow that up with an illustrative example over the coming articles. We’ll build on what we’ve learned in those previous series and we’ll again use the common System Center suite to get some hands-on experience. As always, the concepts and workflow apply quite well to tools other than System Center too.

To summarise, delegation and self-service are essential for most organisations as they scale. When used to safely allow autonomy of other groups, it can save you and your team significantly.

[Buffet picture by Kenming Wang and used unmodified under SA2.0]



Enterprise Cloud Orchestration recap

Enterprise Cloud Orchestration recap

This brief article hopes to summarise and collect the recent set of articles published around orchestration in the Enterprise Cloud.

  1. In our first article, we gave an overview of orchestration in the context of the larger automation umbrella and looked at is as a way to simplify the safe execution of automated tasks.
  2. Part two in the series looked at orchestration workflows (runbooks in System Center speak), using System Center Orchestrator 2016 as an example.
  3. Article #3 looked at a Microsoft PowerShell template for calling complex PowerShell functionality from within a System Center Orchestrator runbook.
  4. In our next article, number four, we looked at the use-case specific code. Our example used Tintri SyncVM to perform some innovative and efficient data-copy management for our Test/Dev folks.
  5. Finally, article five in the series pulled it all together and allowed us to execute the orchestration runbook, and our PowerShell activity, and see the results.

This series extended upon our automation series to take a business problem and create an agile and automated solution suitable for safely delegating to folks outside our core infrastructure group. This could also be scheduled for regular execution within Orchestrator.

Keep your eye out for the next series, which will look at putting this in the hands of the end user through a simple self-service portal.

[La grande salle de la Philharmonie de Paris image by Jean-Pierre Dalbera and used unmodified under CC2.0]

Orchestration for Enterprise Cloud part 5

Orchestration for Enterprise Cloud part 5

We’ve spent the past four installments in this series putting together a System Center Orchestrator runbook workflow to call into PowerShell to call Tintri Automation Toolkit cmdlets to do a bunch of stuff.

The stuff that it’s doing is solving a real business need for us — we want our developers to be able to test their code against a current copy of production data. Dump and restore operations are very expensive and error prone, so we’re taking advantage of Tintri’s SyncVM functionality to handle the data synchronisation for us. As we’ll see, this is going to take less than a minute to perform!

In this article, we’ll walk through executing this runbook and show how easy it makes the task. This simplicity makes it a great candidate for a task that can be delegated to someone with less in-depth knowledge (or access to) the cloud infrastructure. This is a big step forward toward self-service.

Orchestrator Web Console

If we now point our web browser at port 82 of our Orchestrator server (for example, http://scorch-2016.vmlevel.com:82/), you should be presented with the Orchestrator Web Console and you should see our new Runbook.


Select the runbook and click the Start Runbook button.


It will prompt you for the required input — simply the name of the developer’s virtual machine. It doesn’t request any information about the VMstore that the VM is stored on, it doesn’t ask for the production VM name, it doesn’t ask which snapshot to sync from and it doesn’t ask which virtual disks to synchronise. All of that is taken care of inside the runbook. This drastically reduces the number of places we could accidentally mess something up when we’re in a hurry or if we delegate this task to someone else.

Should something go wrong with the destination VM as part of this process, the SyncVM process we’re using takes a safety snapshot automatically, so at worst, we can easily roll it back.

We’ll enter our VM name (vmlevel-devel) and kick off the runbook job.


Next we’ll click on the Jobs tab and should see a running job.


If the job doesn’t have an hour glass (indicating it’s running) or a green tick (indicating success), it’s worth checking that the Orchestrator Runbook service is started on your runbook servers (check your Services applet). I’ve noticed that at times it doesn’t start correctly by itself despite being set to be automatically started:


After a little while (it takes about 45 seconds in my lab), hit refresh and the job should have succeeded. At that point, click on View Instances and then View Details to view the details of the job.


If we click on the Activity Details tab and scroll down, we can see the parameters of the Run .Net Script activity that calls our PowerShell code. If you look closely, you’ll see the variables we have defined. This especially includes out TraceLog variable, which you can see in the above output gives us a very detailed run-down of the process executed.

Given that this has succeeded, we’ve achieved our goal. Our developer VM has our developer code and OS on it, but has a copy of the latest production data snapshot. The whole process took less than 60 seconds and the developer is now up and running with recent production data — all without costly dumps and restores.

Try it for yourself and see.

[Ovation image by Joi Ito and used unmodified under CC2.0]

Orchestration for Enterprise Cloud part 4

Orchestration for Enterprise Cloud part 4

Leading up to this point in this series, we’ve spoken a little about System Center Orchestrator and why we might want to deploy runbooks within it (or another orchestration tool). We also looked at how to create a runbook and pass parameters between runbook activities. We then looked at a Microsoft template for calling sophisticated pieces of PowerShell as part of that runbook workflow. As we covered both here and in our automation series, we’re generally doing all of this to solve a real business problem.

In this article, we’ll look at the portion of the sample code that we haven’t looked at yet. This is the code that actually calls into the Tintri Automation Toolkit for PowerShell and performs the magic that is data-copy management through SyncVM.

The Code

The use-case specific code that’s going to solve our business need is the code from line 119 to 194. This uses the Tintri Automation Toolkit for PowerShell (free download from the Support Portal) to use SyncVM to handle our zero-copy data synchronisation.

You’ll notice that each logical section of code is surrounded by a try { …. } catch { …. } block. The Tintri cmdlets will throw exceptions when an operation fails and using try and catch allows us to correctly handle those cases and collect any information needed for our trace log and to pass back to the user.

The rest is all pretty straightforward and just calls the following cmdlets to get the job done:

  1. Import-Module to import the Tintri PowerShell modules. In PowerShell 3.0 and later, this should automatically happen, but by explicitly trying to import it, it’s easy to tell when the module isn’t available. This module needs to be installed on each of our Runbook Servers.
  2. Connect-TintriServer creates a session with our Tintri VMstore. Note the use of the -UseCurrentUserCredentials option. This code is run as the Orchestrator service account (specified at install time) on the runbook servers. The -UseCurrentUserCredentials option allows the use of Kerberos Single Sign On (SSO) to authenticate against the VMstore. This means no hard-coded passwords and also means that if/when we change those service account credentials, we don’t need to track down all of the scripts that use the credentials and change those too. REST and PowerShell SSO is something that we covered in detail in a previous post.
  3. Get-TintriVM on line 148 retrieves an object representing our developer VM. We’ll use that further down.
  4. Get-TintriVM (line 161), Get-TintriVMSnapshot (line 163) and Get-TintriVDisk (line 165) get objects to represent the production VM, its most recent snapshot and the set of virtual disks within that snapshot respectively.
  5. Sync-TintriVDisk on line 178 is where the magic happens. We take the development VM object, and a subset of the vDisks attached to the latest production snapshot (data disks 1 and 2, skipping system disk 0), and performs the zero-copy data synchronisation. At the completion of this cmdlet, the development VM will have been booted with the production data from that latest snapshot.
  6. Disconnect-TintriServer on line 192 just closes our session to the Tintri VMstore. It’s always good practice to do so.


Note the types of things that we’re logging to our trace log too. In the case of the Connect-TintriServer cmdlet call, this creates a connection and authenticates us. It will fail if either of those things goes wrong. As a result, we’re logging the VMstore we’re connecting to and the username we’re connecting as. On failure, we log the exception message so that we know why it failed.

In the case of the per-VM operations, we log the VM we’re operating on and the exception message.

What we’re trying to do is to leave a very clear trail of what happened leading up to a failure.

Ready To Roll

At this point, we’re ready to execute this whole workflow. We’ll demonstrate that in the next article with a bunch of screenshots just to break things up a little.

[Clones image by HJ Media Studios and used unmodified under SA2.0]

Orchestration for Enterprise Cloud part 3

Orchestration for Enterprise Cloud part 3

In the first article in this series, we gave an overview of orchestration as an extension to our previous series around automation. We also used the second article to build an Runbook in System Center Orchestrator to call some PowerShell code. In this article and the next, we’ll look at the PowerShell code.

One of our API gurus, Rick Ehrhart, was kind enough to rent me some space in the Tintri Github repo of API examples to publish a complete worked example of the code we’ll be working with. A syntax-highlighted version can be found here for your reading pleasure, with a raw version also available if you wish to download and modify it.

This article will cover the Orchestrator plumbing part of the script, and the next article will cover the Tintri Automation Toolkit part that actually interacts with the VMs and the storage.

Adding the PowerShell code

Previously, we included a Run .NET Script activity that had some inputs and some outputs. We set the script language to PowerShell, but didn’t add any PowerShell code there. We’ll do that shortly. The text input box is very limited in size and functionality. There’s no syntax highlighting or online help or any of the other things we’ve become accustomed to. It’s also not the easiest to run code from in an iterative way.

What I suggest is to develop the code in something like PowerShell ISE and whenever it’s changed, update the copy in the runbook activity. There are a few different conventions between the two, but we’ll take that into consideration.

The Code

There are two main sections to the code example. There’s a bit in the middle that does all of the use-case specific stuff around Tintri SyncVM. This can be modified to do anything you like. The second part is all of the stuff at the top and the bottom of the script, which allows this to be run effectively under Orchestrator. That is what we’ll cover in this article and it borrows heavily from the Microsoft Best Practices example.

Standalone Mode

Line #48 sets a boolean (true/false) variable called $standalone. You’ll notice that that variable is used in a number of if statements throughout that look something like this:

if($standalone) {
} else {

By setting this variable to $true at the top of the script, this code will execute paths that are specific to running under PowerShell ISE or just PowerShell. When set to $false, it’s an indication to the rest of the script that we’re running under Orchestrator. There aren’t a lot of differences, but the way Orchestrator passes in parameters and handles script output differ. In the case of running under PowerShell ISE, we don’t have the Orchestrator Return Data activity to subscribe to our output data, so in that case, we write it to stdout to allow us to read it.

You’ll notice that when not running in standalone mode, the script sets a variable called $param to the string “{VMname from Initialize Data}“. Specific to Orchestrator, this sets this variable to the VM name that’s passed in from the Initialize Data workflow. You can set this without having to type it by right-clicking between the double quotes and clicking Subscribe:


When we copy/paste the code into Orchestrator, we need to remember to set $standalone to $false.

Script-wide variables and constants

From about line 51 through to 69, we’re setting a number of variables that control parts of our script execution and provide us with values for things like the name of the production VM we’re dealing with.

We said earlier that we wanted to limit how much control end users had over this workflow as a way to prevent accidents. Here, because we’re dealing with a single production VM on a single Tintri VMstore, we’ve just set these as constants. If we had a selection of production VMs across a number of VMstores, we might instead have another Orchestrator activity before this one, that uses whatever means is appropriate in your environment to select the correct production VM. Perhaps it looks up some tags or attributes on the named developer VM. Perhaps this is something that we can have the developer provide with some amount of validation.

We also have variables called $TraceLog, $ErrorMessage and $ResultStatus. Normal input/output mechanisms don’t apply in the Orchestrator case. So we’ll use these variables to store the final result ($ResultStatus), a user-friendly error message ($ErrorMessage) and a log of tracing information so that if anything goes wrong, we can take a look at it afterwards. These match the variable names we added as Returned Data when defining the Run .NET Script activity in the last article. These need to match.

A Script Inside A Script

System Center Orchestrator runs PowerShell scripts using a built-in PowerShell 2.0 interpreter. This means that a lot of the comforts we’ve become accustomed to with PowerShell 3.0 and later just don’t exist.

Lines 85, 96 and 211 call New-PSSession, Invoke-Command and Remove-PSSession to use PowerShell remoting to start a new PowerShell session on the same host, but using the default PowerShell instance, which will be 3.0 or later on Windows 2012r2 and later. The script that is run in this child session is the code within the -ScriptBlock { … } option.

Because this script is run as a separate process, variables outside the ScriptBlock aren’t accessible inside the ScriptBlock. As a result, we wrap up our input parameters inside an array called ArgsList and pass that into the ScriptBlock. We also wrap up the results (tracing, error message, result) in another array and return that back to the outer script.

If we wanted to pass more data in, we could add more to the $ArgsList array and use it inside the ScriptBlock. If we wanted to return more stuff from the ScriptBlock, we could add it to the $resultsArray array and use it outside the ScriptBlock.


In this brief article, we’ve looked at the plumbing needed to have some PowerShell code execute as part of a System Center Orchestrator activity. To summarise, here are the steps our code had to perform:

  1. If we’re being called from Orchestrator, collect the provided VM name as an input parameter.
  2. Set some script-wide variables and constants.
  3. Create an array of parameters and variables to pass into a child PowerShell session.
  4. Start the child PowerShell session, which takes the provided variables, does a bunch of Tintri stuff (that we’ll cover next) and returns some results in another array.
  5. Close down the child PowerShell session and place the results in a set of variables that we configured Orchestrator to pass on to the next runbook activity.

In the next article, we’ll cover the the lines of code that use Tintri’s SyncVM functionality to serve the use case we defined at the start of this series — to instantly make a copy of our production data available to our developer VMs. At that point, we have a working Orchestration runbook and can look at expanding it.

[Matryoshka image by fletcherjcm and used unmodified under SA2.0]

Orchestration for Enterprise Cloud part 2

Orchestration for Enterprise Cloud part 2

In our previous post, we set the scene for being able to use System Center Orchestrator for being able to bundle up automated functionality in a way that’s more consumable by others. We also defined the business problem we’re trying to solve and worked out which inputs we would rely on the user to provide and which things we will decide on their behalf.

In this article, we’ll start to look at the Orchestrator workflow and how to include some PowerShell automation as part of that. We’ll then follow up with some specifics around example PowerShell code, which has its share of peculiarities when being run under Orchestrator.

Orchestrator Workflows

System Center Orchestrator, like many orchestration tools, allows us to take automation modules, called Activities, and glue them together in a workflow called a Runbook. There are a bunch of pre-defined activities that ship with Orchestrator, and Microsoft provides extra activity bundles in a set of Integration Packs that are available for download. We’ll be keeping things quite simple in our example and will rely primarily on the Run .NET Script activity that comes standard.

Open Orchestrator’s Runbook Designer tool, select your Orchestrator runbook server and we’ll create a new runbook. In my example, I’ve called mine Prod-Dev Data Sync. Using the activities on the left, drag and drop the following activities into the runbook and connect them:

  • Initialize Data — this is where our runbook will start
  • Run .NET Script — this activity is where we’ll call out to PowerShell to do Tintri SyncVM magic
  • Return Data — at the end of the runbook, any returned data will be handled here and passed back to Orchestrator

It will end up looking like this:


We can far more complex runbooks than this, and indeed ours could be improved, but this will work as a starting point.

Next, right-click the Initialize Data activity, select Properties and click the Details tab. This is where we set the inputs that this runbook will take from the end user. We’ll add one string input parameter and we’ll call it VMname. It should look as follows:


We’re going to blindly pass this along to the next activities in the runbook, but it would be better to perform some kind of basic checks here to make sure that we’re operating on a developer VM and not a production VM. We’ll come back to that.

Next, right-click the Run .NET Script activity, hit Properties and again select the Details tab. This is where we’ll add our PowerShell script shortly. For now, set the language to PowerShell and leave the the script pane blank. Click on the Published Data tab. This is where we take any values returned our script and pass them to the next activity in the runbook.

We’re going to add three variables to be returned:

  1. ResultStatus as an integer
  2. ErrorMessage as a string
  3. TraceLog as a string

The names don’t matter too much, but the name you use in the Variable field will need to match the variable names we use in our script, and the Name field will be plumbed into the Return Data activity shortly. The result should look like this:


The plumbing for our runbook is almost done. We just need to take care of the data that we want to return and we’re then ready to start adding our PowerShell code.

First, right-click on the runbook’s title in the tab along the top, and click Properties, and select the Returned Data tab. This is where we’ll declare which parameters we plan to return. To begin with, these will match what we returned from the Run .NET Script activity earlier.


This defines the data we plan to return, but we still need to link that up with the previous activities in the runbook.

Right-click the Return Data activity, click Properties and select the Details tab. You’ll notice that it has already populated with the list of return parameters we defined a moment ago.

For each parameter to be returned, we need to subscribe to a piece of data published by earlier activities. In our case, we’ll right-click the text field, select Subscribe and Published Data. You should be able to find each of the variables we defined as output from the Run .NET Script activity earlier. Orchestrator will then fill in the text fields with template text that looks like ‘{ResultStatus from “Run .Net Script”}’ as seen below:


At this point, our runbook is pretty much complete. To summarise what we’ve done:

  1. Declared the VM name as an input parameter and subscribed to it
  2. Passed that VM name to a PowerShell script (still to come)
  3. Took the returned data from the script and returned that at the end of our runbook.

We’ll cover the PowerShell script in two separate articles — our next will cover the plumbing needed for PowerShell scripts in Orchestrator, and then the one after will look at the use-case specific code.

Ideas for improvement

The runbook we have will be pretty functional once we’re doing with the PowerShell component. But it is about as simple as it could be. Here are a couple of ways that we could insert some more activities into the runbook to make things more robust:

  1. Have an activity after the Initialize Data activity that checks that the virtual machine name is a developer VM, rather than production or some other application.
  2. Extend that to check that the user requesting the activity owns the VM being operated on. Most hypervisors don’t have a concept of an owner of a VM in this context, so we’d need to think of ways to map that ourselves.
  3. Check the output of the script and if the operation failed, automatically email us with the trace log data and anything else helpful.

Time permitting, we’ll try to tackle one or more of these once we have the initial project up and running.

[Orchestra image by aldern82 and used without modification under SA2.0]

Orchestration For Enterprise Cloud

Orchestration For Enterprise Cloud

In our last series, we looked at taking a business problem or need and turning into a piece of automated code. We started out by breaking the problem down into smaller pieces, we then put together a piece of sortamation to demonstrate the overall solution, and then made it more modular and added some error handling.

In this series, we’re going to extend upon this and integrate our automation into an orchestration framework. This approach will apply to any orchestration framework, but we’ll use System Center Orchestrator 2016 in our examples.

Why Orchestration?

Primarily for delegation of tasks. We may have written a script that we can run to perform some mundane task, but for us to be able to successfully scale, we need to start putting some of these tasks into the hands of others.

As a dependency for delegation, we also want to use automation and orchestration as a way to guarantee us consistent results and general damage prevention. Sure, we could just allow everyone access to SCVMM or vCenter to manage their virtual machines, but that’s a recipe for disaster. Orchestration gives us a way to safely grant controlled access to limited sets of functionality to make other groups or teams more self-sufficient.

The Process

Much like in our earlier automation series, we want to start by defining a business problem and breaking it down to smaller tasks from there. We want to extend this too to include the safe delegation of this task and this will include thinking carefully about input we’ll accept from the user, information we’ll give back to the user, and what kind of diagnostic information we’ll need to collect so that if something goes wrong, we can take a look later.

The fictitious, but relevant, business problem that we’re going to solve in this series is a common DevOps problem:

Developers want to be able to test their code against real, live production data to ensure realistic test results.

The old approach would be to have someone dump a copy of the production application database and restore the dump to each of the developers’ own database instances. This is expensive in time and capacity and can adversely impact performance. It’s also error-prone.

Instead, we’ll look at making use of Tintri’s SyncVM technology to use space-efficient clones to be able to nearly-instantly make production data available to all developers. We’ll do this with some PowerShell and a runbook in System Center Orchestrator.

We can then either schedule the runbook to be executed nightly, or we can make the runbook available to the helpdesk folks, who can safely execute the runbook from the Orchestrator Web Console. [Later, we’ll look at another series that shows us how to make this available to the developers themselves — probably through a Service Manager self-service portal — but let’s not get too far ahead of ourselves]

Core Functionality

Our production VM and developer VMs all have three virtual disks:

  1. A system drive that contains the operating system, any tools or applications that the developer needs, and the application code either in production or in development.
  2. A disk to contain database transaction logs.
  3. A disk to contain database data file logs.

In our workflow, we’ll want to use SyncVM to take vDisks #2 and #3 from a snapshot of the production VM, and attach them to the corresponding disk slots on the developer’s VM. We want this process to not touch vDisk #1, which contains the developer’s code and development tools.


For us to use SyncVM (as we saw previously), we need to pass in some information to the Tintri Automation Toolkit about what to sync. Looking at previous similar code, we probably need to know the following:

  • The Tintri VMstore to connect to.
  • The production VM and the snapshot of it that we wish to sync
  • The set of disks to sync
  • The destination developer VM

In order to limit potential accidents, we probably want to limit how much of this comes from the person we’re delegating this to. In our example, we’ll assume a single VMstore and a single known production VM. We’ve also established earlier that there is a consistent and common pattern for the sets of disks to sync (always vDisk #2 and #3). The only parameter where there should be any user input is the destination VM. The rest can all be handled consistently and reliably within our automation.


After this has been executed, we’ll need to let the user know if it succeeded or failed, along with any helpful information they’ll need. We’ll also want to track a lot of detailed information about our automation so that if an issue arises, we have some hope of resolving it.


So far, we have again defined a business need or business problem (synchronisation of production data to developer VMs), defined the set of inputs we’ll need and where those will come from, and we’ve defined the outputs.

In the next installment, we’ll start to get our hands dirty with System Center Orchestrator, followed by PowerShell and the Tintri Automation Toolkit for PowerShell.

[Orchestra image by Sean MacEntee and used unmodified under CC2.0]