Making all Objects in an AWS S3 Bucket Always Public

At TechSnips, we use Amazon S3 to store all of the stuff required to operate. One ability we need is to provide a publicly accessible repository of files. Luckily, S3 has this ability to set objects to public read-access.

To set an object to public read-access, you can click on Make Public via right-clicking on the object inside of the S3 Management Console.

S3 Make Public

This is all well and good but if you’ve got tons of files constantly being uploaded to S3, I’m not about to manually make all of my objects public like this!

After a bit of digging, I was able to figure out how to make all objects be public the moment they are added to a particular bucket. To do this requires creating a bucket policy. This bucket policy applies to all GetObject actions. You can see the bucket policy I used below for our techsnips-public S3 bucket.

This bucket policy can be assigned to the bucket via the Management Console

Once you have the bucket policy set, you’ll then need to also assign Public Access to the Everyone group as well via the Access Control List.

How to Create/Restore an Azure Virtual Machine Snapshot with PowerShell

Azure Virtual Machine Snapshot

I’ve learned that creating an Azure virtual machine snapshot isn’t quite as easy as creating checkpoints in my on-prem Hyper-V lab. Frustrated that I had to build my own tool for this, I set out to do it anyway.

As you may have found out already creating an Azure virtual machine snapshot isn’t quite as right-click and click as you’d expect. Sure, Azure has Azure Backup now but it seems overkill for the constant snapshots and reverts I’d like to do in my development environment. Instead, Azure has a Snapshots feature that appears to have been neglected by Microsoft. There’s little to no documentation on creating and restoring Azure virtual machine snapshots.

So I dug in and automated it!

I wanted two PowerShell functions so I can easily create and restore snapshots from OS disk attached to Azure virtual machines called New-AzureRmVmSnapshot and Restore-AzureRmVmSnapshot so I got to work.

Creating Azure Snapshots

First, I need to figure out how to create an Azure virtual machine snapshot in the first place. To do that, I first needed to shut down the VM that I wanted to create a snapshot for.

Stopping the Azure VM

Creating the snapshot

Once the VM was stopped, I then learned how I could create a snapshot using the New-AzureRmSnapshot cmdlet but it, unfortunately, wasn’t that easy. To gather up all of the information I needed required first finding the OS disk that was attached to the VM, creating a config and then I could create the snapshot.

Create a Disk from the Snapshot

Once the snapshot was created, you actually have to create a new Azure disk from the snapshot which requires creating another disk config and creating the disk with the New-AzureRmDisk cmdlet.

Yay! We have created our first Azure virtual machine snapshot!

Restoring an Azure virtual machine Snapshot

Now let’s say you’ve been messing around in the VM and want to restore the snapshot. This is the step that was a challenge because I could find no great step-by-step tutorial to do this. I eventually found out how to detach the old Azure OS disk from the VM and attach the new one.

Detach the Old OS Disk and Reattach new one

At this point, the snapshot should be restored but you’ve still got that old OS disk hanging around. Go ahead and clean that up.

Delete the Old OS Disk

Does all of this sound like a major pain? It did me too! Luckily, my pain is your gain! Here are two functions to take a snapshot of an OS disk attached to a VM and one to restore it. They are nothing fancy and could be improved greatly

Functions

 

How to “Rename” Amazon S3 “Folder” Objects with Python

 

To rename a folder on a traditional file system is a piece of cake but what if that file system wasn’t really a file system at all? In that case, it gets a little trickier! Amazon’s S3 service consists of objects with key values. There are no folders or files to speak of but we still need to perform typical filesystem-like actions like renaming folders.

Renaming S3 “folders” isn’t possible; not even in the S3 management console but we can perform a workaround. We can create a new “folder” in S3 and then move all of the files from that “folder” to the new “folder”. Once all of the files are moved, we can then remove the source “folder”.

To do this, I’ll be using Python and the boto3 module. If you’re working with S3 and Python and not using the boto3 module, you’re missing out. It makes things much easier to work with.

Prerequisites

For the demonstration I’ll be showing you to work, you’ll need to meet a few prereqs ahead of time:

  • MacOS/Linux
  • Python 3+
  • The boto3 module (pip install boto3 to get it)
  • An Amazon S3 Bucket
  • An AWS IAM user access key and secret access key with access to S3
  • An existing “folder” with “files” inside in your S3 bucket

Renaming an Amazon S3 Key

To rename our S3 folder, we’ll need to import the boto3 module and I’ve chosen to assign some of the values I’ll be working with as variables.

Once I’ve done that, I’ll need to authenticate to S3 by providing my access key ID and secret key for the IAM user I’ll be using. In this case, I’ve chosen to use a boto3 session. I’ll be using a boto3 resource to work with S3.

Once I’ve done that, I then need to find all of the files matching my key prefix. You can see below that I’m using a Python for loop to read all of the objects in my S3 bucket. I’m using the optional filter action and filtering all of the S3 objects in the bucket down to only the key prefix for the folder I want to rename.

Once I’ve started the for loop iterating over the “folder” key and all of the “file” keys inside of it, I’ll then need to exclude the “folder” key itself since I won’t be copying that. I just need the file keys. I’m excluding that by an if statement that matches all key values that don’t end with a forward slash.

After I’m in the block that will only contain file key values, I’m now assigning the file name and destination key names to make it easier to reference.

Once I have all of that setup, I then finally do the actual copy using the copy_from action. You can see below that I’m creating an S3 object using the bucket name and destination file key. I’m then passing the source key to the copy_from action.

Once the loop has finished and all of the files have been copied to the new key, I’ll then need to use the delete action to clean all of the files including the “folder” key since it is not inside of the if condition.

At this point, we’re done! You should now see all of the files that were previously in the source key under the destination key with no sign of the source key!

How to Write to a Text File with PowerShell

A common need amongst IT professionals is to create and append to text files. The ubiquitous nature of a plain-text file lends itself to lots of uses. In a PowerShell script, we can quickly create and append to plain-text files using a couple of built-in cmdlets called Add-Content and Set-Content. In this article, let’s go over how we can use both of these cmdlets to handle all of our text file needs.

Both the Add-Content and Set-Content cmdlet both perform similar functions; creating and writing to text files but they behave a little differently. The biggest difference is one always appends text to a file while the other always overwrites it. This is important to know when you’re writing to an important log file!

Continue reading “How to Write to a Text File with PowerShell”

How to Create a PSCredential Object Without Using Get-Credential in PowerShell

PSCredential objects are commonplace in PowerShell. They are a creative way to store and pass credentials to various services securely. Many built-in and third-party cmdlets require PSCredential objects on many different commands.

Typically, to create a PSCredential object, we’d use the Get-Credential cmdlet. The Get-Credential cmdlet is the most common way that PowerShell receives input to create the PSCredential object like the username and password.

Continue reading “How to Create a PSCredential Object Without Using Get-Credential in PowerShell”

How to Reference an Object Property in a String in PowerShell

Strings by themselves are a simple data type in PowerShell. Merely surrounding a set of characters with single or double quotes creates a string. But whenever you need to insert an expression inside of that string is when things get a little more complicated. This process is typically known as string interpolation.
By default, PowerShell interprets a string as a literal. For example, just typing 'This is a string' in PowerShell creates a string just fine. However, when you need to insert some expression like a variable or an object property in our case, you have to use double quotes.
String Interpolation
In this case, we used a single variable. To get an expression to expand inside of a string, we just need to insert the variable inside of double quotes but happens when we try to expand an object property?

Continue reading “How to Reference an Object Property in a String in PowerShell”

The “I’m Not Good Enough” Excuse has Officially Been Eradicated

All throughout my time in the tech communities, the common theme I keep hearing from people is “I’m not good enough” or “I could never do that!”. It’s frustrating to me because I know that every good IT guy or gal has knowledge in their head that’s people would love to learn about.

What’s stopping you?

TechSnips’ mission is to provide a platform for technology professionals to showcase their work via video screencasts. However, those that aren’t used to presenting and required more coaching than others tended to get frustrated when they were forced to re-record their screencasts due to quality concerns.

Now they don’t have to!

Continue reading “The “I’m Not Good Enough” Excuse has Officially Been Eradicated”

How to Run PowerShell Code by Invoking a Webhook in Azure Automation

Back in my day, we could only run PowerShell scripts on local Windows computers. Nowadays though. it’s amazing all of the different ways to execute PowerShell scripts. We not only have PowerShell on Linux but Azure Cloud Shell too! Azure Cloud Shell is an awesome way to quickly bring up a PowerShell environment to execute code in the cloud. Speaking of Azure, we’ve also got Azure Automation runbooks. Using PowerShell Azure Automation runbooks are another great way to get stuff done with PowerShell.

Whenever I build a PowerShell Azure Automation runbook, the trigger for the runbook has always been on a schedule or invoked via another runbook. However recently, I needed the ability to invoke it outside of Azure completely via a service we use here at TechSnips called Zapier.

Continue reading “How to Run PowerShell Code by Invoking a Webhook in Azure Automation”

A PowerShell Tool to Create IIS Websites

During one of my latest snips on How to Manage IIS Websites with PowerShell, I decided to create a PowerShell function to make it easier. This function wraps up everything I spoke about in that snip with some extra goodness as well. It’s a great function to use if you find yourself constantly creating IIS sites on a remote computer.

Continue reading “A PowerShell Tool to Create IIS Websites”

Introducing the TechSnips Paid Blog Post Program

TechSnips started as a video-only platform and continues to grow every week. However, we realize that not everyone prefers to learn new material via screencasts. This is why we are introducing the paid blog post program. The paid blog post program is a feature of TechSnips that allows approved contributors to supplement their screencast submissions with a blog post and get paid for it!

Continue reading “Introducing the TechSnips Paid Blog Post Program”