Enterprise Cloud and All-You-Can-Eat Buffets

Enterprise Cloud and All-You-Can-Eat Buffets

What does Enterprise Cloud have in common with an all-you-can-eat buffet?

Delegation and Self-Service

As well as automating tasks to be run on a regular schedule to avoid manual handling, many tasks are being automated to allow the delegation of executing certain tasks to other humans.

Consider the case where you have 10 VDI desktops deployed. As common tasks come up, such as restoring files from snapshots, diagnosing performance issues or provisioning new desktops, it’s easy to jump in and take care of matters by hand. Take that number to 1000 and you’re likely going to start to see issues maintaining those by hand as you scale. Get to 10,000 or more and it’s an entirely different class of problem.

This doesn’t just apply to VDI — DevOps deployments and Enterprise server farms are seeing the same kinds of challenges as they scale too.

In order to scale past a few systems, you need to start to delegate some number of tasks to someone else. Whether that be a helpdesk team of some kind, or a developer or application owner, or even potentially the end user of a VDI desktop.

However, delegation and self-service are not just a case of dumping a bunch of tech in front of folks and wishing them luck. In most cases, these folks won’t have the technical domain knowledge required to safely manage their portion of infrastructure. We need to identify the tasks that they need to be able to perform and package those up safely and succinctly.


Consider a restaurant with an all-you-can-eat buffet. One of the nice ones — we’re professionals here. Those buffets don’t have a pile of raw ingredients, knives and hotplates, yet they’re most definitely still self-service.

You’re given a selection of dishes to choose from. They’ve all been properly prepared and safely presented, so that you don’t need to worry about the food preparation yourself. There is the possibility of making some bad decisions (roast beef and custard), but you can’t really go far enough to actually do yourself any great harm.

They do this to scale. More patrons with fewer overhead costs, such as staff.

DIY Self-Service

As we deploy some kind of delegation or self-service infrastructure, we need to:

  1. Come up with a menu of tasks that we wish to allow others to perform,
  2. Work out the safety constraints around putting them in the hands of others, and
  3. Probably still having staff to pour the bottomless mimosas instead of simply a tap.

We did introduce these two things in previous series’ of articles. In particular, #1 is a case of listing and defining one or more business problems, as we saw in the automation series.. For example, users that accidentally delete or lose an important file, might need a way to retrieve files from a snapshot from a few days ago. #2 above is referring to taking and validating very limited user input. In the restore example above, we’d probably only allow the user to specify the day that contains the snapshot they’re looking for and maybe the name of their VM.

Public Cloud

Self-service and autonomy are one of the things that Public Cloud have brought to the table at a generic level. By understanding the specifics of your own Enterprise, you can not only meet, but exceed that Public Cloud agility within your own data centre. This can also be extended to seamlessly include Public Cloud for the hybrid case.

Next Steps

As with each of these series, we’re starting here with a high level overview and will follow that up with an illustrative example over the coming articles. We’ll build on what we’ve learned in those previous series and we’ll again use the common System Center suite to get some hands-on experience. As always, the concepts and workflow apply quite well to tools other than System Center too.

To summarise, delegation and self-service are essential for most organisations as they scale. When used to safely allow autonomy of other groups, it can save you and your team significantly.

[Buffet picture by Kenming Wang and used unmodified under SA2.0]



A Very Particular Set Of Skills

A Very Particular Set Of Skills

Has anybody not heard of the recent ransomware attack known as WannaCry? No? Good. Hopefully you’re only aware of it through news articles, but for far too many folks, this is not the case.

We all keep our patches up to date and we all use various levels of protection to limit the attack surface and potential spread of these kinds of attacks.

Unfortunately, and for various reasons, these kinds of attacks can still wreak havoc.

When this does happen, it doesn’t have to ruin your year.

For an individual virtual machine affected by this, simply:

  1. Revert your affected VM back to a previous snapshot using SyncVM
  2. Start the VM disconnected from the network
  3. Apply any updates to close the exploited security hole
  4. Reconnect to the network
  5. Don’t pay the ransom

In cases where there are a very large number of affected VMs, a lot of this process can be automated.

To misappropriate and misquote the famous speech from the movie Taken, our customers have a very particular set of skills. Skills we’ve been assisting them with over a long career.

[Ransom Note image by Sheila Sund and used unmodified under CC BY 2.0]

Enterprise Cloud Orchestration recap

Enterprise Cloud Orchestration recap

This brief article hopes to summarise and collect the recent set of articles published around orchestration in the Enterprise Cloud.

  1. In our first article, we gave an overview of orchestration in the context of the larger automation umbrella and looked at is as a way to simplify the safe execution of automated tasks.
  2. Part two in the series looked at orchestration workflows (runbooks in System Center speak), using System Center Orchestrator 2016 as an example.
  3. Article #3 looked at a Microsoft PowerShell template for calling complex PowerShell functionality from within a System Center Orchestrator runbook.
  4. In our next article, number four, we looked at the use-case specific code. Our example used Tintri SyncVM to perform some innovative and efficient data-copy management for our Test/Dev folks.
  5. Finally, article five in the series pulled it all together and allowed us to execute the orchestration runbook, and our PowerShell activity, and see the results.

This series extended upon our automation series to take a business problem and create an agile and automated solution suitable for safely delegating to folks outside our core infrastructure group. This could also be scheduled for regular execution within Orchestrator.

Keep your eye out for the next series, which will look at putting this in the hands of the end user through a simple self-service portal.

[La grande salle de la Philharmonie de Paris image by Jean-Pierre Dalbera and used unmodified under CC2.0]

Orchestration for Enterprise Cloud part 5

Orchestration for Enterprise Cloud part 5

We’ve spent the past four installments in this series putting together a System Center Orchestrator runbook workflow to call into PowerShell to call Tintri Automation Toolkit cmdlets to do a bunch of stuff.

The stuff that it’s doing is solving a real business need for us — we want our developers to be able to test their code against a current copy of production data. Dump and restore operations are very expensive and error prone, so we’re taking advantage of Tintri’s SyncVM functionality to handle the data synchronisation for us. As we’ll see, this is going to take less than a minute to perform!

In this article, we’ll walk through executing this runbook and show how easy it makes the task. This simplicity makes it a great candidate for a task that can be delegated to someone with less in-depth knowledge (or access to) the cloud infrastructure. This is a big step forward toward self-service.

Orchestrator Web Console

If we now point our web browser at port 82 of our Orchestrator server (for example, http://scorch-2016.vmlevel.com:82/), you should be presented with the Orchestrator Web Console and you should see our new Runbook.


Select the runbook and click the Start Runbook button.


It will prompt you for the required input — simply the name of the developer’s virtual machine. It doesn’t request any information about the VMstore that the VM is stored on, it doesn’t ask for the production VM name, it doesn’t ask which snapshot to sync from and it doesn’t ask which virtual disks to synchronise. All of that is taken care of inside the runbook. This drastically reduces the number of places we could accidentally mess something up when we’re in a hurry or if we delegate this task to someone else.

Should something go wrong with the destination VM as part of this process, the SyncVM process we’re using takes a safety snapshot automatically, so at worst, we can easily roll it back.

We’ll enter our VM name (vmlevel-devel) and kick off the runbook job.


Next we’ll click on the Jobs tab and should see a running job.


If the job doesn’t have an hour glass (indicating it’s running) or a green tick (indicating success), it’s worth checking that the Orchestrator Runbook service is started on your runbook servers (check your Services applet). I’ve noticed that at times it doesn’t start correctly by itself despite being set to be automatically started:


After a little while (it takes about 45 seconds in my lab), hit refresh and the job should have succeeded. At that point, click on View Instances and then View Details to view the details of the job.


If we click on the Activity Details tab and scroll down, we can see the parameters of the Run .Net Script activity that calls our PowerShell code. If you look closely, you’ll see the variables we have defined. This especially includes out TraceLog variable, which you can see in the above output gives us a very detailed run-down of the process executed.

Given that this has succeeded, we’ve achieved our goal. Our developer VM has our developer code and OS on it, but has a copy of the latest production data snapshot. The whole process took less than 60 seconds and the developer is now up and running with recent production data — all without costly dumps and restores.

Try it for yourself and see.

[Ovation image by Joi Ito and used unmodified under CC2.0]

Orchestration for Enterprise Cloud part 4

Orchestration for Enterprise Cloud part 4

Leading up to this point in this series, we’ve spoken a little about System Center Orchestrator and why we might want to deploy runbooks within it (or another orchestration tool). We also looked at how to create a runbook and pass parameters between runbook activities. We then looked at a Microsoft template for calling sophisticated pieces of PowerShell as part of that runbook workflow. As we covered both here and in our automation series, we’re generally doing all of this to solve a real business problem.

In this article, we’ll look at the portion of the sample code that we haven’t looked at yet. This is the code that actually calls into the Tintri Automation Toolkit for PowerShell and performs the magic that is data-copy management through SyncVM.

The Code

The use-case specific code that’s going to solve our business need is the code from line 119 to 194. This uses the Tintri Automation Toolkit for PowerShell (free download from the Support Portal) to use SyncVM to handle our zero-copy data synchronisation.

You’ll notice that each logical section of code is surrounded by a try { …. } catch { …. } block. The Tintri cmdlets will throw exceptions when an operation fails and using try and catch allows us to correctly handle those cases and collect any information needed for our trace log and to pass back to the user.

The rest is all pretty straightforward and just calls the following cmdlets to get the job done:

  1. Import-Module to import the Tintri PowerShell modules. In PowerShell 3.0 and later, this should automatically happen, but by explicitly trying to import it, it’s easy to tell when the module isn’t available. This module needs to be installed on each of our Runbook Servers.
  2. Connect-TintriServer creates a session with our Tintri VMstore. Note the use of the -UseCurrentUserCredentials option. This code is run as the Orchestrator service account (specified at install time) on the runbook servers. The -UseCurrentUserCredentials option allows the use of Kerberos Single Sign On (SSO) to authenticate against the VMstore. This means no hard-coded passwords and also means that if/when we change those service account credentials, we don’t need to track down all of the scripts that use the credentials and change those too. REST and PowerShell SSO is something that we covered in detail in a previous post.
  3. Get-TintriVM on line 148 retrieves an object representing our developer VM. We’ll use that further down.
  4. Get-TintriVM (line 161), Get-TintriVMSnapshot (line 163) and Get-TintriVDisk (line 165) get objects to represent the production VM, its most recent snapshot and the set of virtual disks within that snapshot respectively.
  5. Sync-TintriVDisk on line 178 is where the magic happens. We take the development VM object, and a subset of the vDisks attached to the latest production snapshot (data disks 1 and 2, skipping system disk 0), and performs the zero-copy data synchronisation. At the completion of this cmdlet, the development VM will have been booted with the production data from that latest snapshot.
  6. Disconnect-TintriServer on line 192 just closes our session to the Tintri VMstore. It’s always good practice to do so.


Note the types of things that we’re logging to our trace log too. In the case of the Connect-TintriServer cmdlet call, this creates a connection and authenticates us. It will fail if either of those things goes wrong. As a result, we’re logging the VMstore we’re connecting to and the username we’re connecting as. On failure, we log the exception message so that we know why it failed.

In the case of the per-VM operations, we log the VM we’re operating on and the exception message.

What we’re trying to do is to leave a very clear trail of what happened leading up to a failure.

Ready To Roll

At this point, we’re ready to execute this whole workflow. We’ll demonstrate that in the next article with a bunch of screenshots just to break things up a little.

[Clones image by HJ Media Studios and used unmodified under SA2.0]

Automation for Private Cloud recap

Automation for Private Cloud recap

Earlier this year we looked at how to turn a business need into a piece of code to be used as part of an automation or orchestration workflow.

This (very) brief article is to briefly summarise the whole series in one easy-to-find place.

  1. Defining the high level business problem and breaking it down into smaller subtasks was the focus of the first article.
  2. The focus of the second article was to find the right tool for the job for our use case of efficient data-copy management. This gave us a few lines of PowerShell that made use of some Tintri REST APIs and Tintri’s space-efficient SyncVM feature.
  3. Our third installment looked at the concept of modularity. This allowed us to write one piece of code that could then be reused many times over without modification.
  4. We looked at separating the code from the configuration in our fourth part in the series. This allows us to start to safely delegate management of automated tasks to folks that may not need access to the code itself.
  5. Article number 5 finished up by spending time looking at the importance of error handling in our automation. We wouldn’t normally leave this to the end when developing something, but it’s a topic that deserved an article of its own.

Extending upon that, we’re moving into articles on using our automation within Orchestration frameworks for delegation and self-service. Keep an eye out for those.

[Clouds image by Tom Hall used unmodified under CC2.0]

Orchestration for Enterprise Cloud part 3

Orchestration for Enterprise Cloud part 3

In the first article in this series, we gave an overview of orchestration as an extension to our previous series around automation. We also used the second article to build an Runbook in System Center Orchestrator to call some PowerShell code. In this article and the next, we’ll look at the PowerShell code.

One of our API gurus, Rick Ehrhart, was kind enough to rent me some space in the Tintri Github repo of API examples to publish a complete worked example of the code we’ll be working with. A syntax-highlighted version can be found here for your reading pleasure, with a raw version also available if you wish to download and modify it.

This article will cover the Orchestrator plumbing part of the script, and the next article will cover the Tintri Automation Toolkit part that actually interacts with the VMs and the storage.

Adding the PowerShell code

Previously, we included a Run .NET Script activity that had some inputs and some outputs. We set the script language to PowerShell, but didn’t add any PowerShell code there. We’ll do that shortly. The text input box is very limited in size and functionality. There’s no syntax highlighting or online help or any of the other things we’ve become accustomed to. It’s also not the easiest to run code from in an iterative way.

What I suggest is to develop the code in something like PowerShell ISE and whenever it’s changed, update the copy in the runbook activity. There are a few different conventions between the two, but we’ll take that into consideration.

The Code

There are two main sections to the code example. There’s a bit in the middle that does all of the use-case specific stuff around Tintri SyncVM. This can be modified to do anything you like. The second part is all of the stuff at the top and the bottom of the script, which allows this to be run effectively under Orchestrator. That is what we’ll cover in this article and it borrows heavily from the Microsoft Best Practices example.

Standalone Mode

Line #48 sets a boolean (true/false) variable called $standalone. You’ll notice that that variable is used in a number of if statements throughout that look something like this:

if($standalone) {
} else {

By setting this variable to $true at the top of the script, this code will execute paths that are specific to running under PowerShell ISE or just PowerShell. When set to $false, it’s an indication to the rest of the script that we’re running under Orchestrator. There aren’t a lot of differences, but the way Orchestrator passes in parameters and handles script output differ. In the case of running under PowerShell ISE, we don’t have the Orchestrator Return Data activity to subscribe to our output data, so in that case, we write it to stdout to allow us to read it.

You’ll notice that when not running in standalone mode, the script sets a variable called $param to the string “{VMname from Initialize Data}“. Specific to Orchestrator, this sets this variable to the VM name that’s passed in from the Initialize Data workflow. You can set this without having to type it by right-clicking between the double quotes and clicking Subscribe:


When we copy/paste the code into Orchestrator, we need to remember to set $standalone to $false.

Script-wide variables and constants

From about line 51 through to 69, we’re setting a number of variables that control parts of our script execution and provide us with values for things like the name of the production VM we’re dealing with.

We said earlier that we wanted to limit how much control end users had over this workflow as a way to prevent accidents. Here, because we’re dealing with a single production VM on a single Tintri VMstore, we’ve just set these as constants. If we had a selection of production VMs across a number of VMstores, we might instead have another Orchestrator activity before this one, that uses whatever means is appropriate in your environment to select the correct production VM. Perhaps it looks up some tags or attributes on the named developer VM. Perhaps this is something that we can have the developer provide with some amount of validation.

We also have variables called $TraceLog, $ErrorMessage and $ResultStatus. Normal input/output mechanisms don’t apply in the Orchestrator case. So we’ll use these variables to store the final result ($ResultStatus), a user-friendly error message ($ErrorMessage) and a log of tracing information so that if anything goes wrong, we can take a look at it afterwards. These match the variable names we added as Returned Data when defining the Run .NET Script activity in the last article. These need to match.

A Script Inside A Script

System Center Orchestrator runs PowerShell scripts using a built-in PowerShell 2.0 interpreter. This means that a lot of the comforts we’ve become accustomed to with PowerShell 3.0 and later just don’t exist.

Lines 85, 96 and 211 call New-PSSession, Invoke-Command and Remove-PSSession to use PowerShell remoting to start a new PowerShell session on the same host, but using the default PowerShell instance, which will be 3.0 or later on Windows 2012r2 and later. The script that is run in this child session is the code within the -ScriptBlock { … } option.

Because this script is run as a separate process, variables outside the ScriptBlock aren’t accessible inside the ScriptBlock. As a result, we wrap up our input parameters inside an array called ArgsList and pass that into the ScriptBlock. We also wrap up the results (tracing, error message, result) in another array and return that back to the outer script.

If we wanted to pass more data in, we could add more to the $ArgsList array and use it inside the ScriptBlock. If we wanted to return more stuff from the ScriptBlock, we could add it to the $resultsArray array and use it outside the ScriptBlock.


In this brief article, we’ve looked at the plumbing needed to have some PowerShell code execute as part of a System Center Orchestrator activity. To summarise, here are the steps our code had to perform:

  1. If we’re being called from Orchestrator, collect the provided VM name as an input parameter.
  2. Set some script-wide variables and constants.
  3. Create an array of parameters and variables to pass into a child PowerShell session.
  4. Start the child PowerShell session, which takes the provided variables, does a bunch of Tintri stuff (that we’ll cover next) and returns some results in another array.
  5. Close down the child PowerShell session and place the results in a set of variables that we configured Orchestrator to pass on to the next runbook activity.

In the next article, we’ll cover the the lines of code that use Tintri’s SyncVM functionality to serve the use case we defined at the start of this series — to instantly make a copy of our production data available to our developer VMs. At that point, we have a working Orchestration runbook and can look at expanding it.

[Matryoshka image by fletcherjcm and used unmodified under SA2.0]