Orchestration for Enterprise Cloud part 4

Orchestration for Enterprise Cloud part 4

Leading up to this point in this series, we’ve spoken a little about System Center Orchestrator and why we might want to deploy runbooks within it (or another orchestration tool). We also looked at how to create a runbook and pass parameters between runbook activities. We then looked at a Microsoft template for calling sophisticated pieces of PowerShell as part of that runbook workflow. As we covered both here and in our automation series, we’re generally doing all of this to solve a real business problem.

In this article, we’ll look at the portion of the sample code that we haven’t looked at yet. This is the code that actually calls into the Tintri Automation Toolkit for PowerShell and performs the magic that is data-copy management through SyncVM.

The Code

The use-case specific code that’s going to solve our business need is the code from line 119 to 194. This uses the Tintri Automation Toolkit for PowerShell (free download from the Support Portal) to use SyncVM to handle our zero-copy data synchronisation.

You’ll notice that each logical section of code is surrounded by a try { …. } catch { …. } block. The Tintri cmdlets will throw exceptions when an operation fails and using try and catch allows us to correctly handle those cases and collect any information needed for our trace log and to pass back to the user.

The rest is all pretty straightforward and just calls the following cmdlets to get the job done:

  1. Import-Module to import the Tintri PowerShell modules. In PowerShell 3.0 and later, this should automatically happen, but by explicitly trying to import it, it’s easy to tell when the module isn’t available. This module needs to be installed on each of our Runbook Servers.
  2. Connect-TintriServer creates a session with our Tintri VMstore. Note the use of the -UseCurrentUserCredentials option. This code is run as the Orchestrator service account (specified at install time) on the runbook servers. The -UseCurrentUserCredentials option allows the use of Kerberos Single Sign On (SSO) to authenticate against the VMstore. This means no hard-coded passwords and also means that if/when we change those service account credentials, we don’t need to track down all of the scripts that use the credentials and change those too. REST and PowerShell SSO is something that we covered in detail in a previous post.
  3. Get-TintriVM on line 148 retrieves an object representing our developer VM. We’ll use that further down.
  4. Get-TintriVM (line 161), Get-TintriVMSnapshot (line 163) and Get-TintriVDisk (line 165) get objects to represent the production VM, its most recent snapshot and the set of virtual disks within that snapshot respectively.
  5. Sync-TintriVDisk on line 178 is where the magic happens. We take the development VM object, and a subset of the vDisks attached to the latest production snapshot (data disks 1 and 2, skipping system disk 0), and performs the zero-copy data synchronisation. At the completion of this cmdlet, the development VM will have been booted with the production data from that latest snapshot.
  6. Disconnect-TintriServer on line 192 just closes our session to the Tintri VMstore. It’s always good practice to do so.

Tracing

Note the types of things that we’re logging to our trace log too. In the case of the Connect-TintriServer cmdlet call, this creates a connection and authenticates us. It will fail if either of those things goes wrong. As a result, we’re logging the VMstore we’re connecting to and the username we’re connecting as. On failure, we log the exception message so that we know why it failed.

In the case of the per-VM operations, we log the VM we’re operating on and the exception message.

What we’re trying to do is to leave a very clear trail of what happened leading up to a failure.

Ready To Roll

At this point, we’re ready to execute this whole workflow. We’ll demonstrate that in the next article with a bunch of screenshots just to break things up a little.

[Clones image by HJ Media Studios and used unmodified under SA2.0]

Automation for Private Cloud recap

Automation for Private Cloud recap

Earlier this year we looked at how to turn a business need into a piece of code to be used as part of an automation or orchestration workflow.

This (very) brief article is to briefly summarise the whole series in one easy-to-find place.

  1. Defining the high level business problem and breaking it down into smaller subtasks was the focus of the first article.
  2. The focus of the second article was to find the right tool for the job for our use case of efficient data-copy management. This gave us a few lines of PowerShell that made use of some Tintri REST APIs and Tintri’s space-efficient SyncVM feature.
  3. Our third installment looked at the concept of modularity. This allowed us to write one piece of code that could then be reused many times over without modification.
  4. We looked at separating the code from the configuration in our fourth part in the series. This allows us to start to safely delegate management of automated tasks to folks that may not need access to the code itself.
  5. Article number 5 finished up by spending time looking at the importance of error handling in our automation. We wouldn’t normally leave this to the end when developing something, but it’s a topic that deserved an article of its own.

Extending upon that, we’re moving into articles on using our automation within Orchestration frameworks for delegation and self-service. Keep an eye out for those.

[Clouds image by Tom Hall used unmodified under CC2.0]

Orchestration for Enterprise Cloud part 3

Orchestration for Enterprise Cloud part 3

In the first article in this series, we gave an overview of orchestration as an extension to our previous series around automation. We also used the second article to build an Runbook in System Center Orchestrator to call some PowerShell code. In this article and the next, we’ll look at the PowerShell code.

One of our API gurus, Rick Ehrhart, was kind enough to rent me some space in the Tintri Github repo of API examples to publish a complete worked example of the code we’ll be working with. A syntax-highlighted version can be found here for your reading pleasure, with a raw version also available if you wish to download and modify it.

This article will cover the Orchestrator plumbing part of the script, and the next article will cover the Tintri Automation Toolkit part that actually interacts with the VMs and the storage.

Adding the PowerShell code

Previously, we included a Run .NET Script activity that had some inputs and some outputs. We set the script language to PowerShell, but didn’t add any PowerShell code there. We’ll do that shortly. The text input box is very limited in size and functionality. There’s no syntax highlighting or online help or any of the other things we’ve become accustomed to. It’s also not the easiest to run code from in an iterative way.

What I suggest is to develop the code in something like PowerShell ISE and whenever it’s changed, update the copy in the runbook activity. There are a few different conventions between the two, but we’ll take that into consideration.

The Code

There are two main sections to the code example. There’s a bit in the middle that does all of the use-case specific stuff around Tintri SyncVM. This can be modified to do anything you like. The second part is all of the stuff at the top and the bottom of the script, which allows this to be run effectively under Orchestrator. That is what we’ll cover in this article and it borrows heavily from the Microsoft Best Practices example.

Standalone Mode

Line #48 sets a boolean (true/false) variable called $standalone. You’ll notice that that variable is used in a number of if statements throughout that look something like this:

if($standalone) {
   ....
} else {
   ....
}

By setting this variable to $true at the top of the script, this code will execute paths that are specific to running under PowerShell ISE or just PowerShell. When set to $false, it’s an indication to the rest of the script that we’re running under Orchestrator. There aren’t a lot of differences, but the way Orchestrator passes in parameters and handles script output differ. In the case of running under PowerShell ISE, we don’t have the Orchestrator Return Data activity to subscribe to our output data, so in that case, we write it to stdout to allow us to read it.

You’ll notice that when not running in standalone mode, the script sets a variable called $param to the string “{VMname from Initialize Data}“. Specific to Orchestrator, this sets this variable to the VM name that’s passed in from the Initialize Data workflow. You can set this without having to type it by right-clicking between the double quotes and clicking Subscribe:

scorch-psvar

When we copy/paste the code into Orchestrator, we need to remember to set $standalone to $false.

Script-wide variables and constants

From about line 51 through to 69, we’re setting a number of variables that control parts of our script execution and provide us with values for things like the name of the production VM we’re dealing with.

We said earlier that we wanted to limit how much control end users had over this workflow as a way to prevent accidents. Here, because we’re dealing with a single production VM on a single Tintri VMstore, we’ve just set these as constants. If we had a selection of production VMs across a number of VMstores, we might instead have another Orchestrator activity before this one, that uses whatever means is appropriate in your environment to select the correct production VM. Perhaps it looks up some tags or attributes on the named developer VM. Perhaps this is something that we can have the developer provide with some amount of validation.

We also have variables called $TraceLog, $ErrorMessage and $ResultStatus. Normal input/output mechanisms don’t apply in the Orchestrator case. So we’ll use these variables to store the final result ($ResultStatus), a user-friendly error message ($ErrorMessage) and a log of tracing information so that if anything goes wrong, we can take a look at it afterwards. These match the variable names we added as Returned Data when defining the Run .NET Script activity in the last article. These need to match.

A Script Inside A Script

System Center Orchestrator runs PowerShell scripts using a built-in PowerShell 2.0 interpreter. This means that a lot of the comforts we’ve become accustomed to with PowerShell 3.0 and later just don’t exist.

Lines 85, 96 and 211 call New-PSSession, Invoke-Command and Remove-PSSession to use PowerShell remoting to start a new PowerShell session on the same host, but using the default PowerShell instance, which will be 3.0 or later on Windows 2012r2 and later. The script that is run in this child session is the code within the -ScriptBlock { … } option.

Because this script is run as a separate process, variables outside the ScriptBlock aren’t accessible inside the ScriptBlock. As a result, we wrap up our input parameters inside an array called ArgsList and pass that into the ScriptBlock. We also wrap up the results (tracing, error message, result) in another array and return that back to the outer script.

If we wanted to pass more data in, we could add more to the $ArgsList array and use it inside the ScriptBlock. If we wanted to return more stuff from the ScriptBlock, we could add it to the $resultsArray array and use it outside the ScriptBlock.

Summary

In this brief article, we’ve looked at the plumbing needed to have some PowerShell code execute as part of a System Center Orchestrator activity. To summarise, here are the steps our code had to perform:

  1. If we’re being called from Orchestrator, collect the provided VM name as an input parameter.
  2. Set some script-wide variables and constants.
  3. Create an array of parameters and variables to pass into a child PowerShell session.
  4. Start the child PowerShell session, which takes the provided variables, does a bunch of Tintri stuff (that we’ll cover next) and returns some results in another array.
  5. Close down the child PowerShell session and place the results in a set of variables that we configured Orchestrator to pass on to the next runbook activity.

In the next article, we’ll cover the the lines of code that use Tintri’s SyncVM functionality to serve the use case we defined at the start of this series — to instantly make a copy of our production data available to our developer VMs. At that point, we have a working Orchestration runbook and can look at expanding it.

[Matryoshka image by fletcherjcm and used unmodified under SA2.0]

Orchestration for Enterprise Cloud part 2

Orchestration for Enterprise Cloud part 2

In our previous post, we set the scene for being able to use System Center Orchestrator for being able to bundle up automated functionality in a way that’s more consumable by others. We also defined the business problem we’re trying to solve and worked out which inputs we would rely on the user to provide and which things we will decide on their behalf.

In this article, we’ll start to look at the Orchestrator workflow and how to include some PowerShell automation as part of that. We’ll then follow up with some specifics around example PowerShell code, which has its share of peculiarities when being run under Orchestrator.

Orchestrator Workflows

System Center Orchestrator, like many orchestration tools, allows us to take automation modules, called Activities, and glue them together in a workflow called a Runbook. There are a bunch of pre-defined activities that ship with Orchestrator, and Microsoft provides extra activity bundles in a set of Integration Packs that are available for download. We’ll be keeping things quite simple in our example and will rely primarily on the Run .NET Script activity that comes standard.

Open Orchestrator’s Runbook Designer tool, select your Orchestrator runbook server and we’ll create a new runbook. In my example, I’ve called mine Prod-Dev Data Sync. Using the activities on the left, drag and drop the following activities into the runbook and connect them:

  • Initialize Data — this is where our runbook will start
  • Run .NET Script — this activity is where we’ll call out to PowerShell to do Tintri SyncVM magic
  • Return Data — at the end of the runbook, any returned data will be handled here and passed back to Orchestrator

It will end up looking like this:

scorch16.png

We can far more complex runbooks than this, and indeed ours could be improved, but this will work as a starting point.

Next, right-click the Initialize Data activity, select Properties and click the Details tab. This is where we set the inputs that this runbook will take from the end user. We’ll add one string input parameter and we’ll call it VMname. It should look as follows:

scorch-init

We’re going to blindly pass this along to the next activities in the runbook, but it would be better to perform some kind of basic checks here to make sure that we’re operating on a developer VM and not a production VM. We’ll come back to that.

Next, right-click the Run .NET Script activity, hit Properties and again select the Details tab. This is where we’ll add our PowerShell script shortly. For now, set the language to PowerShell and leave the the script pane blank. Click on the Published Data tab. This is where we take any values returned our script and pass them to the next activity in the runbook.

We’re going to add three variables to be returned:

  1. ResultStatus as an integer
  2. ErrorMessage as a string
  3. TraceLog as a string

The names don’t matter too much, but the name you use in the Variable field will need to match the variable names we use in our script, and the Name field will be plumbed into the Return Data activity shortly. The result should look like this:

scorch-scriptret

The plumbing for our runbook is almost done. We just need to take care of the data that we want to return and we’re then ready to start adding our PowerShell code.

First, right-click on the runbook’s title in the tab along the top, and click Properties, and select the Returned Data tab. This is where we’ll declare which parameters we plan to return. To begin with, these will match what we returned from the Run .NET Script activity earlier.

scorch-retdata

This defines the data we plan to return, but we still need to link that up with the previous activities in the runbook.

Right-click the Return Data activity, click Properties and select the Details tab. You’ll notice that it has already populated with the list of return parameters we defined a moment ago.

For each parameter to be returned, we need to subscribe to a piece of data published by earlier activities. In our case, we’ll right-click the text field, select Subscribe and Published Data. You should be able to find each of the variables we defined as output from the Run .NET Script activity earlier. Orchestrator will then fill in the text fields with template text that looks like ‘{ResultStatus from “Run .Net Script”}’ as seen below:

scorch-retsubscribe

At this point, our runbook is pretty much complete. To summarise what we’ve done:

  1. Declared the VM name as an input parameter and subscribed to it
  2. Passed that VM name to a PowerShell script (still to come)
  3. Took the returned data from the script and returned that at the end of our runbook.

We’ll cover the PowerShell script in two separate articles — our next will cover the plumbing needed for PowerShell scripts in Orchestrator, and then the one after will look at the use-case specific code.

Ideas for improvement

The runbook we have will be pretty functional once we’re doing with the PowerShell component. But it is about as simple as it could be. Here are a couple of ways that we could insert some more activities into the runbook to make things more robust:

  1. Have an activity after the Initialize Data activity that checks that the virtual machine name is a developer VM, rather than production or some other application.
  2. Extend that to check that the user requesting the activity owns the VM being operated on. Most hypervisors don’t have a concept of an owner of a VM in this context, so we’d need to think of ways to map that ourselves.
  3. Check the output of the script and if the operation failed, automatically email us with the trace log data and anything else helpful.

Time permitting, we’ll try to tackle one or more of these once we have the initial project up and running.

[Orchestra image by aldern82 and used without modification under SA2.0]

Orchestration For Enterprise Cloud

Orchestration For Enterprise Cloud

In our last series, we looked at taking a business problem or need and turning into a piece of automated code. We started out by breaking the problem down into smaller pieces, we then put together a piece of sortamation to demonstrate the overall solution, and then made it more modular and added some error handling.

In this series, we’re going to extend upon this and integrate our automation into an orchestration framework. This approach will apply to any orchestration framework, but we’ll use System Center Orchestrator 2016 in our examples.

Why Orchestration?

Primarily for delegation of tasks. We may have written a script that we can run to perform some mundane task, but for us to be able to successfully scale, we need to start putting some of these tasks into the hands of others.

As a dependency for delegation, we also want to use automation and orchestration as a way to guarantee us consistent results and general damage prevention. Sure, we could just allow everyone access to SCVMM or vCenter to manage their virtual machines, but that’s a recipe for disaster. Orchestration gives us a way to safely grant controlled access to limited sets of functionality to make other groups or teams more self-sufficient.

The Process

Much like in our earlier automation series, we want to start by defining a business problem and breaking it down to smaller tasks from there. We want to extend this too to include the safe delegation of this task and this will include thinking carefully about input we’ll accept from the user, information we’ll give back to the user, and what kind of diagnostic information we’ll need to collect so that if something goes wrong, we can take a look later.

The fictitious, but relevant, business problem that we’re going to solve in this series is a common DevOps problem:

Developers want to be able to test their code against real, live production data to ensure realistic test results.

The old approach would be to have someone dump a copy of the production application database and restore the dump to each of the developers’ own database instances. This is expensive in time and capacity and can adversely impact performance. It’s also error-prone.

Instead, we’ll look at making use of Tintri’s SyncVM technology to use space-efficient clones to be able to nearly-instantly make production data available to all developers. We’ll do this with some PowerShell and a runbook in System Center Orchestrator.

We can then either schedule the runbook to be executed nightly, or we can make the runbook available to the helpdesk folks, who can safely execute the runbook from the Orchestrator Web Console. [Later, we’ll look at another series that shows us how to make this available to the developers themselves — probably through a Service Manager self-service portal — but let’s not get too far ahead of ourselves]

Core Functionality

Our production VM and developer VMs all have three virtual disks:

  1. A system drive that contains the operating system, any tools or applications that the developer needs, and the application code either in production or in development.
  2. A disk to contain database transaction logs.
  3. A disk to contain database data file logs.

In our workflow, we’ll want to use SyncVM to take vDisks #2 and #3 from a snapshot of the production VM, and attach them to the corresponding disk slots on the developer’s VM. We want this process to not touch vDisk #1, which contains the developer’s code and development tools.

Inputs

For us to use SyncVM (as we saw previously), we need to pass in some information to the Tintri Automation Toolkit about what to sync. Looking at previous similar code, we probably need to know the following:

  • The Tintri VMstore to connect to.
  • The production VM and the snapshot of it that we wish to sync
  • The set of disks to sync
  • The destination developer VM

In order to limit potential accidents, we probably want to limit how much of this comes from the person we’re delegating this to. In our example, we’ll assume a single VMstore and a single known production VM. We’ve also established earlier that there is a consistent and common pattern for the sets of disks to sync (always vDisk #2 and #3). The only parameter where there should be any user input is the destination VM. The rest can all be handled consistently and reliably within our automation.

Output

After this has been executed, we’ll need to let the user know if it succeeded or failed, along with any helpful information they’ll need. We’ll also want to track a lot of detailed information about our automation so that if an issue arises, we have some hope of resolving it.

Summary

So far, we have again defined a business need or business problem (synchronisation of production data to developer VMs), defined the set of inputs we’ll need and where those will come from, and we’ve defined the outputs.

In the next installment, we’ll start to get our hands dirty with System Center Orchestrator, followed by PowerShell and the Tintri Automation Toolkit for PowerShell.

[Orchestra image by Sean MacEntee and used unmodified under CC2.0]

Could The AWS Outage Happen To You?

Could The AWS Outage Happen To You?

No doubt we’ve all been affected by the recent Amazon Web Services outage in one form or probably many. It’s had coverage the Internet over… now that the Internet is again running. We’re also starting to get a picture of what went wrong.

This article isn’t a big I told you so about public cloud, nor is it a jab at Amazon or the poor Engineer who must have had a rough couple of days this week.

I’m hoping that this article serves as a bit of a public service announcement for the way that we all operate within our Enterprise and Private clouds so that the same doesn’t happen to those of us running the infrastructure within our organisations.

What Happened?

Amazon has posted a summary of what went down. I want to take a look at a specific excerpt:

At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

The details are thin, but our immediate reaction might be to jump on that poor S3 team member. I hope his or her week is looking up already and I’m not sure that they’re individually to blame as much as the automation.

The industry is talking about automation and orchestration being a necessity due to the scale and complexity of the in-house systems we have. It’s unreasonable to expect that the complexity of one of the biggest pieces of technical infrastructure can be maintained in the heads of the engineers running it — especially being as fluid as it is.

Any automation or orchestration needs to put bounds around what the end user can do. Whether it be a strong, technical engineer, an end user, or someone in between.

What Can We Learn From This?

We’re all talking a lot about orchestration and automation. We’ve got self-service portals for end users and we’re putting new tools in front of helpdesk staff. We’re having to automate big chunks of what used to be standard IT stuff because the scale is too big to do these things manually anymore.

Whether the automation or orchestration is designed for the most technical of folks, or whether it’s for someone without the technical chops, it’s important to make sure that the tooling only allows stuff that should be allowed and prevents everything else. It sounds simpler than it really is, but it’s worth considering and limiting the potential use cases to avoid tragedy in-house.

A Simple, Contrived Example

To illustrate the point, let’s assume that we have some orchestration to allow an operator to control RAID arrays in servers in our data centre. One of their responsibilities is to monitor the number of write cycles to the SSDs in the RAID array. When a drive crosses a threshold, the operator marks it failed and organises a replacement drive.

[I told you that this illustrative example would be contrived…]

These fictional arrays are made up of RAID 6 stripe sets, which as we know supports two drive failures.

Now let’s say that the operator sees three drives cross the threshold at the same time and pushes the big red button on each of them. We now have an incomplete RAID set and an outage.

Sure, in the real world, we wouldn’t remove non-failed disks from an array before a replacement was there and would only ever do one at a time. However, if we expand this type of cluster-like redundancy to hosts, clusters and geographic regions, we’re starting to see how quickly we’re going to lose visibility of all of the moving parts.

The point is that if you have a piece of automation that accepts input from someone (including yourself late, late at night), try to preempt the valid use cases and and restrict or limit the rest.

[Image by William Warby and used unmodified under CC 2.0]

Automation and Private Cloud part V

Automation and Private Cloud part V

We’ve come a long way in this series of articles. Thanks for coming along for the ride.

Whilst we haven’t worked our way through this in a fashion typical of a development project (today’s topic should never come last), hopefully it’s allowed us to tackle parts of automation projects independently.

At the end of our last article, we had some fairly modular automation code to solve a particular business problem. There is one critical piece missing though. Our code only covers the happy case. If anything unexpected happens, it all falls apart. In this article, we’ll look at some ways to handle that.

This would normally never be the last step in a project and would be something we add as we go. For educational purposes (and clarity of previous examples), I’ve left it to the end as its own article.

What Could Possibly Go Wrong?

Our code currently does the following:

  1. Reads and parses a JSON file of VM mappings
  2. Loops through each mapping pair and calls our Sync-ProdRep function, which:
    1. Connects to a Tintri VMstore using SSO
    2. Retrieves an object representing the production and the reporting VMs
    3. Takes a Tintri snapshot of the production VM and retrieves the snapshot object
    4. Calls Sync-TintriVDisk to attach the production snapshot database vDisks to the reporting VM

[Note: the above language representation of the code at a high level is a good way to draft things out before firing up your favourite automation text editor]

This is all great. Except for the times that it isn’t. What happens in any of these cases:

  • The JSON file has a syntax error or typo in it?
  • The VMstore is unavailable due to network outage?
  • The current user’s credentials have expired or similar (including clock skew in the Kerberos case)?
  • The user typed the VM names incorrectly?
  • The snapshot was unable to be taken for some reason?
  • The SyncVM operation was unable to complete for some reason?

Things will break. The first you’ll probably hear about it is first thing in the morning when the angry masses are amassed outside your cube with pitchforks and torches. You’re going to need two things:

  • An answer as to what went wrong, and
  • The least amount of impact

The exact nature of how you handle issues is going to very from project to project and operation to operation and requires some careful consideration.

There is a lot of documentation on the syntax of various error handling constructs for PowerShell and other languages and tools. Instead of rehashing that here, we’ll concentrate more on the impact to various parts of our automation example.

Exceptions

PowerShell cmdlets in the Tintri PowerShell Toolkit, like with many PowerShell cmdlets, indicate failure by throwing something called an exception. Whilst there are other ways to determine success or failure of an operation we’ll use exceptions here to illustrate our point.

We can tell PowerShell to try a particular Tintri operation and if it fails, allow us to catch an exception object so that we can determine what to do next.

If we take our code as an example, each of the cmdlets could potentially fail for a variety of reasons. The Connect-TintriServer cmdlet could fail due to network issues or authentication issue or a range of other issues. The Get-TintriVM cmdlet might fail if the VM name was mistyped or the the VM had been migrated to another storage appliance.

Let’s look at the Connect-TintriServer cmdlet specifically:

Try {
    Connect-TintriServer $vmstore -UseCurrentUserCredentials -ErrorAction Stop
} Catch {
    Write-Output "$(Get-Date -Format o): $($_.Exception)"
    return $false
}

We’ve added the -ErrorAction Stop option to the cmdlet to tell it to halt and raise an exception and we’ve wrapped it in this Try/Catch block. If Connect-TintriServer fails, an exception is raised and the code inside the catch block is executed.  In this case, we show a message including a datestamp and the exception message. None of the rest of the cmdlets following this will be of any use without a successful Tintri Server session, so we use the return statement to stop our function and indicate failure. Our Foreach-Object loop will continue on with the next VM pair.

Ideally, we’d take a look at each of the rest of the calls and come up with some form of action to take on error. It may just be a case of failing in the same way that we have for Connect-TintriServer, but perhaps there’s cases where instead of failing, we could try again or perform some other type of recovery action.

It’s also worth us explicitly checking the returned value when we call it from our Foreach-Object loop. Consider this example:

$mappings.mappings | `
   Foreach-Object {
       $result = Sync-ProdRep -Prod $_.prod -Report $_.report
       if($result -eq $false) {
           Write-Output "$(Get-Date -Format o): $_.prod succeeded"
       } else {
           Write-Output "$(Get-Date -Format o): $_.prod failed"
       }
 }

In this example, we just display a message of success on success or indicating failure otherwise.

We probably also want to add a final line to our function to explicitly return $true if all of the operations succeed.

Here’s a complete version of our script with all of the modifications:

function Sync-ProdRep {
    [CmdletBinding()]
    param(
      [parameter(Mandatory=$true)][String]$prodname,
      [parameter(Mandatory=$true)][String]$reportname
    )
    # Connect to our VMstore
    $vmstore = "vmstore01.vmlevel.com"
    Try {
      Connect-TintriServer -UseCurrentUserCredentials $vmstore -ErrorAction Stop
    } Catch {
      Write-Output "$(Get-Date -Format o): $($_.Exception)"
      return $false
    }

    # Retrieve a VM object for both VMs by name
    Try {
        $report = Get-TintriVM -Name $reportname -ErrorAction Stop
    } Catch {
        Write-Output "$(Get-Date -Format o): $reportname $($_.Exception)"
        return $false
    }
    Try {
        $prod = Get-TintriVM -Name $prodname -ErrorAction Stop
    } Catch {
        Write-Output "$(Get-Date -Format o): $prodname $($_.Exception)"
        return $false
    }
    # Take a snapshot of our production VM and using the
    # Returned snapshot ID, retrieve the snapshot object
    Try {
      $snapshotIdNew-TintriVMSnapshot `
         -SnapshotDescription "Reporting snapshot" `
         -VM $prod -ErrorAction Stop `
         -SnapshotConsistency CRASH_CONSISTENT
      $snapshot = Get-TintriVMSnapshot -VM $prod `
         -SnapshotId $snapshotId -ErrorAction Stop
    } Catch {
       Write-Output "$(Get-Date -Format o): snap $prodname $($_.Exception)"
       return $false
    }
    # Use SyncVM's vDisk sync to synchronise the data and
    # log vDisks from the prod snapshot to the reporting VM
    Try {
     $result = Sync-TintriVDisk -VM $report `
        -SourceSnapshot $snapshot `
        -AllButFirstVDisk -ErrorAction Stop
    } Catch {
       Write-Output "$(Get-Date -Format o): sync $prodname $($_.Exception)"
       return $false
    }
}

$mappings = ConvertFrom-Json "$(Get-Content 'host-mappings.json')"
$mappings.mappings | `
   Foreach-Object {
       $result = Sync-ProdRep -Prod $_.prod -Report $_.report
       if($result -eq $false) {
           Write-Output "$(Get-Date -Format o): $_.prod succeeded"
       } else {
           Write-Output "$(Get-Date -Format o): $_.prod failed"
       }
 }

So we’ve seen here how to indicate success or failure within our own functions and modules, and we’ve also seen how to catch and deal with exceptions from running PowerShell cmdlets we’ve called.

There are many ways that we could continue to improve the code above, but as it stands now, it’s a little more robust than how it was at the end of the last article.

[Image by Hernán Piñera and used unmodified under CC 2.0]