How to Copy Files Using BITS and PowerShell

BITS PowerShel

Using BITS with PowerShell is a great way to copy files because you get the best of both worlds. BITS doesn’t care if the network connection drops or if the machine restarts, it will just wait patiently until it can connect again and start from where it left off. So you can leave the job to do it’s thing and know that the file will get there in the end. This is especially useful if you have to regularly upload logs from a client/laptop because the job will just keep going until the file is transferred.

By leveraging BITS with PowerShell, you’re able to seamlessly integrate BITS transfers within your PowerShell scripts which makes for a pretty powerful combination!

Creating a BITS job with PowerShell

To create a BITS job with PowerShell you just need to use the Start-BITSTransfer cmdlet and give it the source and destination. This will create the job and start copying the file right away.

The -Asynchronous switch makes it run in the background so you don’t need to watch the progress bar. You can leave this off if you want to watch it run, but it’s perfect for when you are scheduling a copy job etc.

If you are copying lots of files it’s worth thinking about the -DisplayName switch so that you can spot the job in the list.

Finding BITS jobs with PowerShell

To find BITS jobs, just use the Get-BitsTransfer PowerShell cmdlet. Notice the DisplayName of the job is there because we specified it with the switch.

And if we pipe this into Select * it gives you all the attributes for the job.

Notice this shows the status of the BITS job and the BytesTotal/BytesTransferred. In this case, the BITS job has finished, but the job still exists in the list. This means that we can still see all the details of the jobs and we can even add more files to it and it will start transferring those as well. We do this by piping the job into the Add-BitsFile PowerShell cmdlet, and specifying the files we want to add.

Pausing and restarting a BITS job with PowerShell

If we need to pause the job halfway through and start it up later we can use the Suspend-BitsTransfer and Resume-BitsTransfer cmdlets.

Removing BITS jobs with PowerShell

If we want to close down the jobs once they are done with, we can use the Complete-BitsTransfer cmdlet and they will disappear from the list once they are complete. Either pipe the job to it or the list of jobs.

If you are often uploading large log files or client logs, using PowerShell and BITS can be a great way to do this!

How to Create/Restore an Azure Virtual Machine Snapshot with PowerShell

Azure Virtual Machine Snapshot

I’ve learned that creating an Azure virtual machine snapshot isn’t quite as easy as creating checkpoints in my on-prem Hyper-V lab. Frustrated that I had to build my own tool for this, I set out to do it anyway.

As you may have found out already creating an Azure virtual machine snapshot isn’t quite as right-click and click as you’d expect. Sure, Azure has Azure Backup now but it seems overkill for the constant snapshots and reverts I’d like to do in my development environment. Instead, Azure has a Snapshots feature that appears to have been neglected by Microsoft. There’s little to no documentation on creating and restoring Azure virtual machine snapshots.

So I dug in and automated it!

I wanted two PowerShell functions so I can easily create and restore snapshots from OS disk attached to Azure virtual machines called New-AzureRmVmSnapshot and Restore-AzureRmVmSnapshot so I got to work.

Creating Azure Snapshots

First, I need to figure out how to create an Azure virtual machine snapshot in the first place. To do that, I first needed to shut down the VM that I wanted to create a snapshot for.

Stopping the Azure VM

Creating the snapshot

Once the VM was stopped, I then learned how I could create a snapshot using the New-AzureRmSnapshot cmdlet but it, unfortunately, wasn’t that easy. To gather up all of the information I needed required first finding the OS disk that was attached to the VM, creating a config and then I could create the snapshot.

Create a Disk from the Snapshot

Once the snapshot was created, you actually have to create a new Azure disk from the snapshot which requires creating another disk config and creating the disk with the New-AzureRmDisk cmdlet.

Yay! We have created our first Azure virtual machine snapshot!

Restoring an Azure virtual machine Snapshot

Now let’s say you’ve been messing around in the VM and want to restore the snapshot. This is the step that was a challenge because I could find no great step-by-step tutorial to do this. I eventually found out how to detach the old Azure OS disk from the VM and attach the new one.

Detach the Old OS Disk and Reattach new one

At this point, the snapshot should be restored but you’ve still got that old OS disk hanging around. Go ahead and clean that up.

Delete the Old OS Disk

Does all of this sound like a major pain? It did me too! Luckily, my pain is your gain! Here are two functions to take a snapshot of an OS disk attached to a VM and one to restore it. They are nothing fancy and could be improved greatly

Functions

 

Using Set-DnsServerForwarder and Others to Manage DNS Forwarders

Using Set-DnsServerForwarder

Windows DNS Forwarders and Conditional Forwarders are an important part of your DNS infrastructure. You will find that on occasion you need to add or manage these forwarder addresses and that some of these changes need to be made across multiple DNS servers in your enterprise. Thankfully, using commands like PowerShell’s Set-DnsServerForwarder cmdlet and others allow you to easily manage both of these DNS services with ease.

Using Set-DnsServerForwarder to Replace Forwarders

DNS Forwarders are used by the DNS server to lookup queries for addresses that aren’t contained in any zones that the server is authoritative for. This provides your DNS servers with an efficient means for resolving names. Without the forwarders in place, your DNS server would have to query the root hint servers in order to start resolving unknown addresses. While these forwarder addresses are configured separately on each DNS server, using PowerShell makes managing them a lot easier by allowing us to use the Set-DnsServerForward command among others.

So, let’s begin by viewing the currently configured forwarders for the local DNS server by using the Get-DnsServerForwarder cmdlet. We’ll use the Set-DnsServerForwarder in a minute to add one.

As seen below, there are two forwarders configured, as listed beside “IPAddress”

Using Get-DnsServerForwader

Next, we want to add an additional forwarder, possibly a new DNS server that we have configured in our DMZ, or perhaps using a forwarding address provided by our ISP. In this case, we’ll use the Set-DnsServerForwarder cmdlet to set the new address and then use Get-DnsServerForwarder to confirm that the address was set correctly.

The results of using Set-DnsServerForwarder

Unfortunately, this did not have the desired outcome. As you can see here, using the Set-DnsServerForwarder cmdlet actually replaces the list of forwarders rather than adding to it. In order to add the address to the list, rather than replacing the entire list, you need to use Add-DnsServerForwarder. To correct this, what we’ll do is replace the list with the original two forwarders, add the new address, then check to see if we are successful.

Get-DnsServerForwarder

There, that looks much better. We now have all three forwarders added.

Now, if you want to remove a forwarder address, you would simply use the Remove-DnsServerForwarder cmdlet as shown, then check to see if the address has been removed. If Set-DnsServerForwarder replaces the DNS forwarder, Remove-DnsServerForwarder removes it completely.

The results of Remove-DnsServerForwarder

Sometimes, you will need to be able to add or remove a forwarder address on multiple DNS servers. In this instance, Set-DnsServerForwarder will not work. Thankfully PowerShell makes scaling this task to multiple DNS servers relatively easy. If we use Invoke-Command, include a list of all of our DNS servers, then put Add-DnsServerForwarder into the ScriptBlock, we can modify all of the DNS servers with a single command. Then using a similar command, view the results of our changes.

Running Get-DnsServerForwarder on multiple servers

That brings us to the end of the section on DNS Forwarders.

Conditional Forwarders

A special type of forwarder, called a conditional forwarder, cannot be modified with theSet-DnsServerForwarder cmdlet but can be used when you have been provided with the IP address(es) of the DNS server(s) for a known DNS domain name. Conditional forwarders are used by the DNS server before using the server forwarders listed earlier in this article. For example, if you have a conditional forwarder configured for tailspintoys.com, your DNS server will, after checking that it isn’t a domain it is authoritative for,  check the conditional forwarders and find that an entry exists. At this point, your DNS server queries the DNS server listed for the desired address in the tailspintoys.com domain.

One nice feature of conditional forwarders is that they can be replicated to other DNS servers in the same way that any Active Directory Integrated DNS Zone can be.

Let’s start by checking to see if we have a conditional forwarder configured by using the Get-DnsServerZones cmdlet.

Get-DnsServerZone PowerShell

Conditional forwarders show up in this list as ZoneType: Forwarder. In this case, we don’t seem to have one configured. So, we will use Add-DnsServerConditionalForwarderZone to create the conditional forwarder, set it to replicate to the entire Active Directory forest, and then confirm it has been created.

The results of Add-DnsServerConditionalForwarderZone

The output shows that we have our conditional forwarder configured, and it is ready to go. PowerShell really does make managing DNS forwarders a snap.

Additional Resources

Companion video: “How To Manage DNS Forwarders With PowerShell

OpenSSH on Windows: How to Get it Set Up

The future is now. SSH isn’t just for Linux anymore. With the latest Win32 port of OpenSSH, we can now run OpenSSH on Windows with little effort. This post is a text form of my TechSnips video. If you’d rather watch a video walkthrough, I highly encourage you to check that out.

The code I will be using in this demo is available on GitHub in the TechSnips SnipScripts repository.

Installing OpenSSH on Windows

First, make sure to download the latest release of Win32-OpenSSH. Now that that is downloaded, let’s get started with the demo.

I’m going to set some variables for the location of the OpenSSH on Windows zip file and the location where I want to install the OpenSSH files. I’m going to use c:\Program Files\OpenSSH.

Next, we need to extract the OpenSSH-Win64.zip file to the install path.

Once that is complete, we are ready to install OpenSSH on Windows. To do so, we need to run the install-sshd.ps1 script from the install directory.

Run the OpenSSH on Windows install script

This script will create 2 SSH services which we will use in a minute (ssh-agent and sshd)

Configuring OpenSSH on Windows

But first, let’s go ahead and add an exception to the Windows Firewall for port 22 so that SSH traffic will be allowed through.

Now we need to configure the OpenSSH on Windows services. As you can see when we run Get-Service, The services are currently stopped.

View the OpenSSH on Windows services that were installed

So, let’s set the StartupType of both services to Automatic then start the services.

Running the netstat command we can see that the machine is now listening on Port 22.

Netstat shows that OpenSSH on Windows is listening on port 22

Connecting to Our OpenSSH on Windows Server

Next, I’m going to open up my SSH client, PuTTY and connect to the SSH service running on my localhost.

PuTTY connection to local Windows host over SSH

Accept host key message

SSH connection to Windows with OpenSSH

When prompted to accept the host key, choose Yes. This will open an OpenSSH on Windows session in a Command Prompt. If you would like to have SSH sessions open up in a PowerShell prompt, you will need to add an entry in the registry to define the Defaultshell and specify the location of the PowerShell executable.

Now that we have defined the default shell, when we open a new SSH session, you will notice that the Prompt has changed to a “PS” prompt.

PowerShell running through SSH connection with syntax highlighting and tab completion

You will also notice that tab completion and syntax highlighting both work over SSH as well.

Once you’ve set up OpenSSH on Windows and you’re using PowerShell, I encourage you to check out my other post on How to Configure OpenSSH on Windows for PowerShell Remoting Over SSH.

How to Managed Distributed File System (DFS) Links with PowerShell

Distributed File System (DFS) Links reduce the complexity of working with network file shares. DFS Links allow users and applications to access a “virtual path name” to connect to shared folders. We can create DFS links with PowerShell.

This “virtual namespace” enables administrators to present shared folders located on different servers, or even change a shared folder’s location, completely transparent to that folder’s consumers.

Users will not need to update bookmarks, and applications will not be required to be updated with new paths when File Servers change.

Benefit to Users

For the users, access to network share folders is simplified down to a “namespace\FolderName” format, a reduction in the complexity associated with folders stored on remote servers.

Benefit to Applications

For applications, hard-coded paths to resources on the network do not have to be changed due to a change in the network path. A simple update to the DFS Link and the application will continue to access the resources at their new location.

Prerequisites for the DFS Server Role

  • Active Directory
  • File and Storage Services role installed on a Windows Server:
  • Windows Server (Semi-Annual Channel)
  • Windows Server 2016
  • Windows Server 2012 R2
  • Windows Server 2012
  • Windows Server 2008 R2 Datacenter/Enterprise

Prerequisites for PowerShell cmdlets:

  • An administrator account with the proper permissions
  • RSAT Tools with the ‘File Services Tools – DFS Management Tools’ installed

Enabling DFS Tools with PowerShell

Download and install RSAT
Next, you need to Enable the Distributed File System Tools
* Open PowerShell as an Administrator
* Run the Install-WindowsFeature command:

How to Set Up DFS with PowerShell

We first need to install all of the necessary Windows features. This will install DFS Management GUI, the DFS Namespaces module for Windows PowerShell to create DFS links with PowerShell and manage them, and command-line tools, but it does not install any DFS services on the server.

Administration of DFS Links

You can administer namespaces by using DFS Management GUI, the DFS Namespace (DFSN) Cmdlets in Windows PowerShell, the DfsUtil commands, or scripts that call WMI.

Common PowerShell commands

Get-DfsnRoot – Discover all DFS Namespaces in the current domain
– Commonly used to check for available namespaces in the current domain

New-DfsnFolder – Create a new DFS Folder Name
– Commonly used to create a new DFS Folder in a NameSpace

New-DfsnFolderTarget – Assign path(s) to a DFS Folder Name
– Commonly used to assign one or more network folder paths to a DFS Folder

Remove-DfsnFolderTarget – Removes a path from a DFS Folder but does not remove the DFS Folder.
– Commonly used to remove one or more network folder paths from a DFS Folder

Remove-DfsnFolder – Removes a folder and all its paths
– Commonly used to remove a DFS Folder from a NameSpace

Finding DFS namespaces with PowerShell

We’ll start out by getting an idea of all the online and available NameSpaces in the current Domain using the cmdlet Get-DfsnRoot.

This shows that there are two NameSpaces that are Online in this domain.

Finding DFS folders with PowerShell

Using the Get-DfsnFolder command we can list the DFS Folders in each NameSpace.

From this output, we can see that in the AppRoot NameSpace this is not a DFS Folder named PowerShell.

Creating DFS Link Folders with PowerShell

In this example, we have a replicated folder named PowerShell on our 3 File Services servers
* FileServer01
* FileServer02
* Datacenter

The goal is to share this replicated folder with our Admins using a single path.

To do this, we’ll create a new DFS Link folder in the AppRoot NameSpace called PowerShell using New-DfsnFolder and point it to the Datacenter server’s FileShare, set the DFS Folder State to Online and set the TargetPath state to Online.

In the DFS Management GUI, we can see that the “PowerShell” DFS Folder does not exist.

Our Folder Target UNC Path

Run PowerShell and check for the DFS FolderTargetPath

\\Tech.io\AppRoot\PowerShell – if it doesnt exist it will write the output ‘Path not found. Clear to proceed’ in green text in the terminal window.

From the output we see that:
* Path has been created
* Referral Priority Class is set to Global-High
* State is Online

The GUI confirms what PowerShell told us.

and our path is now available in Windows Explorer

Creating DFS Folder Targets with PowerShell

Now that we successfully created the “Powershell” DFS Folder in our NameSpace, we’re going to add an additional Folder Target Path to it and set that path as Online using New-DfsnFolderTarget.

Up to this point, we have two of our three Server Paths added, and online. For our last Folder Path, we want to add the path but not make it available to users. So let’s add a Folder Target Path to our “PowerShell” DFS Folder and this time set the DFS Folder Path State to Offline we will again use New-DfsnFolderTarget

FileServer01 and Datacenter’s path is currently Online
FileServer02‘s state has been set to Offline

Setting DFS Folders Targets to Offline or Online with PowerShell

We can change which servers are the Online and Offline hosts, and even which will be our server will be the primary host of the file path using Set-DfsnFolderTarget.

As you can see below in the ScreenCap FileServer01‘s path has changed to Offline
Datacenter server’s ReferralPriorityClass has switched to sitecost-normal from global-high
Also, notice that FileServer02‘s path has changed its state to Online and its ReferralPriorityClass has switched to global-high

Removing DFS folder target paths with PowerShell

I try to vaccinate my code against the Fat Finger Flu as much as possible. Here we will try to install a “Safety Net” before removing one of the FolderTargetPaths by making sure that it is at least Offline before deleting it.

So long path FileServer01

For those who prefer to “cowboy” it up and forego the “safety net” option, we can accommodate you, brave souls, also.

We’ve bid adieu to the path to FileServer02

Removing DFS Folders with PowerShell

It’s has been a long and winding path, but the time for our DFS Link has come to an end. We can Remove the PowerShell folder, DFS Link, using Remove-DfsnFolder.

A quick double-check of the DFS Management GUI shows our DFS Link is no more.

One final note…These commands are only removing the DFS Links, no folders were harmed in the production of this demo.

For More Information about DFS Links

How to manage Distributed File System links with PowerShell

How to manage Distributed File System links with the GUI

Other Helpful TechSnips

Josh King’s Snip on Splatting at TechSnips.io

Matt McElreath’s Snip on Using Visual Studio Code to do Splatting at TechSnips.io

Microsoft – DFS Overview

How to “Rename” Amazon S3 “Folder” Objects with Python

 

To rename a folder on a traditional file system is a piece of cake but what if that file system wasn’t really a file system at all? In that case, it gets a little trickier! Amazon’s S3 service consists of objects with key values. There are no folders or files to speak of but we still need to perform typical filesystem-like actions like renaming folders.

Renaming S3 “folders” isn’t possible; not even in the S3 management console but we can perform a workaround. We can create a new “folder” in S3 and then move all of the files from that “folder” to the new “folder”. Once all of the files are moved, we can then remove the source “folder”.

To do this, I’ll be using Python and the boto3 module. If you’re working with S3 and Python and not using the boto3 module, you’re missing out. It makes things much easier to work with.

Prerequisites

For the demonstration I’ll be showing you to work, you’ll need to meet a few prereqs ahead of time:

  • MacOS/Linux
  • Python 3+
  • The boto3 module (pip install boto3 to get it)
  • An Amazon S3 Bucket
  • An AWS IAM user access key and secret access key with access to S3
  • An existing “folder” with “files” inside in your S3 bucket

Renaming an Amazon S3 Key

To rename our S3 folder, we’ll need to import the boto3 module and I’ve chosen to assign some of the values I’ll be working with as variables.

Once I’ve done that, I’ll need to authenticate to S3 by providing my access key ID and secret key for the IAM user I’ll be using. In this case, I’ve chosen to use a boto3 session. I’ll be using a boto3 resource to work with S3.

Once I’ve done that, I then need to find all of the files matching my key prefix. You can see below that I’m using a Python for loop to read all of the objects in my S3 bucket. I’m using the optional filter action and filtering all of the S3 objects in the bucket down to only the key prefix for the folder I want to rename.

Once I’ve started the for loop iterating over the “folder” key and all of the “file” keys inside of it, I’ll then need to exclude the “folder” key itself since I won’t be copying that. I just need the file keys. I’m excluding that by an if statement that matches all key values that don’t end with a forward slash.

After I’m in the block that will only contain file key values, I’m now assigning the file name and destination key names to make it easier to reference.

Once I have all of that setup, I then finally do the actual copy using the copy_from action. You can see below that I’m creating an S3 object using the bucket name and destination file key. I’m then passing the source key to the copy_from action.

Once the loop has finished and all of the files have been copied to the new key, I’ll then need to use the delete action to clean all of the files including the “folder” key since it is not inside of the if condition.

At this point, we’re done! You should now see all of the files that were previously in the source key under the destination key with no sign of the source key!

How to Write to a Text File with PowerShell

A common need amongst IT professionals is to create and append to text files. The ubiquitous nature of a plain-text file lends itself to lots of uses. In a PowerShell script, we can quickly create and append to plain-text files using a couple of built-in cmdlets called Add-Content and Set-Content. In this article, let’s go over how we can use both of these cmdlets to handle all of our text file needs.

Both the Add-Content and Set-Content cmdlet both perform similar functions; creating and writing to text files but they behave a little differently. The biggest difference is one always appends text to a file while the other always overwrites it. This is important to know when you’re writing to an important log file!

Continue reading “How to Write to a Text File with PowerShell”

How to Create a PSCredential Object Without Using Get-Credential in PowerShell

PSCredential objects are commonplace in PowerShell. They are a creative way to store and pass credentials to various services securely. Many built-in and third-party cmdlets require PSCredential objects on many different commands.

Typically, to create a PSCredential object, we’d use the Get-Credential cmdlet. The Get-Credential cmdlet is the most common way that PowerShell receives input to create the PSCredential object like the username and password.

Continue reading “How to Create a PSCredential Object Without Using Get-Credential in PowerShell”

How to Reference an Object Property in a String in PowerShell

Strings by themselves are a simple data type in PowerShell. Merely surrounding a set of characters with single or double quotes creates a string. But whenever you need to insert an expression inside of that string is when things get a little more complicated. This process is typically known as string interpolation.
By default, PowerShell interprets a string as a literal. For example, just typing 'This is a string' in PowerShell creates a string just fine. However, when you need to insert some expression like a variable or an object property in our case, you have to use double quotes.
String Interpolation
In this case, we used a single variable. To get an expression to expand inside of a string, we just need to insert the variable inside of double quotes but happens when we try to expand an object property?

Continue reading “How to Reference an Object Property in a String in PowerShell”

The “I’m Not Good Enough” Excuse has Officially Been Eradicated

All throughout my time in the tech communities, the common theme I keep hearing from people is “I’m not good enough” or “I could never do that!”. It’s frustrating to me because I know that every good IT guy or gal has knowledge in their head that’s people would love to learn about.

What’s stopping you?

TechSnips’ mission is to provide a platform for technology professionals to showcase their work via video screencasts. However, those that aren’t used to presenting and required more coaching than others tended to get frustrated when they were forced to re-record their screencasts due to quality concerns.

Now they don’t have to!

Continue reading “The “I’m Not Good Enough” Excuse has Officially Been Eradicated”