How to “Rename” Amazon S3 “Folder” Objects with Python

 

To rename a folder on a traditional file system is a piece of cake but what if that file system wasn’t really a file system at all? In that case, it gets a little trickier! Amazon’s S3 service consists of objects with key values. There are no folders or files to speak of but we still need to perform typical filesystem-like actions like renaming folders.

Renaming S3 “folders” isn’t possible; not even in the S3 management console but we can perform a workaround. We can create a new “folder” in S3 and then move all of the files from that “folder” to the new “folder”. Once all of the files are moved, we can then remove the source “folder”.

To do this, I’ll be using Python and the boto3 module. If you’re working with S3 and Python and not using the boto3 module, you’re missing out. It makes things much easier to work with.

Prerequisites

For the demonstration I’ll be showing you to work, you’ll need to meet a few prereqs ahead of time:

  • MacOS/Linux
  • Python 3+
  • The boto3 module (pip install boto3 to get it)
  • An Amazon S3 Bucket
  • An AWS IAM user access key and secret access key with access to S3
  • An existing “folder” with “files” inside in your S3 bucket

Renaming an Amazon S3 Key

To rename our S3 folder, we’ll need to import the boto3 module and I’ve chosen to assign some of the values I’ll be working with as variables.

Once I’ve done that, I’ll need to authenticate to S3 by providing my access key ID and secret key for the IAM user I’ll be using. In this case, I’ve chosen to use a boto3 session. I’ll be using a boto3 resource to work with S3.

Once I’ve done that, I then need to find all of the files matching my key prefix. You can see below that I’m using a Python for loop to read all of the objects in my S3 bucket. I’m using the optional filter action and filtering all of the S3 objects in the bucket down to only the key prefix for the folder I want to rename.

Once I’ve started the for loop iterating over the “folder” key and all of the “file” keys inside of it, I’ll then need to exclude the “folder” key itself since I won’t be copying that. I just need the file keys. I’m excluding that by an if statement that matches all key values that don’t end with a forward slash.

After I’m in the block that will only contain file key values, I’m now assigning the file name and destination key names to make it easier to reference.

Once I have all of that setup, I then finally do the actual copy using the copy_from action. You can see below that I’m creating an S3 object using the bucket name and destination file key. I’m then passing the source key to the copy_from action.

Once the loop has finished and all of the files have been copied to the new key, I’ll then need to use the delete action to clean all of the files including the “folder” key since it is not inside of the if condition.

At this point, we’re done! You should now see all of the files that were previously in the source key under the destination key with no sign of the source key!

How to Write to a Text File with PowerShell

A common need amongst IT professionals is to create and append to text files. The ubiquitous nature of a plain-text file lends itself to lots of uses. In a PowerShell script, we can quickly create and append to plain-text files using a couple of built-in cmdlets called Add-Content and Set-Content. In this article, let’s go over how we can use both of these cmdlets to handle all of our text file needs.

Both the Add-Content and Set-Content cmdlet both perform similar functions; creating and writing to text files but they behave a little differently. The biggest difference is one always appends text to a file while the other always overwrites it. This is important to know when you’re writing to an important log file!

Continue reading “How to Write to a Text File with PowerShell”

How to Create a PSCredential Object Without Using Get-Credential in PowerShell

PSCredential objects are commonplace in PowerShell. They are a creative way to store and pass credentials to various services securely. Many built-in and third-party cmdlets require PSCredential objects on many different commands.

Typically, to create a PSCredential object, we’d use the Get-Credential cmdlet. The Get-Credential cmdlet is the most common way that PowerShell receives input to create the PSCredential object like the username and password.

Continue reading “How to Create a PSCredential Object Without Using Get-Credential in PowerShell”

How to Reference an Object Property in a String in PowerShell

Strings by themselves are a simple data type in PowerShell. Merely surrounding a set of characters with single or double quotes creates a string. But whenever you need to insert an expression inside of that string is when things get a little more complicated. This process is typically known as string interpolation.
By default, PowerShell interprets a string as a literal. For example, just typing 'This is a string' in PowerShell creates a string just fine. However, when you need to insert some expression like a variable or an object property in our case, you have to use double quotes.
String Interpolation
In this case, we used a single variable. To get an expression to expand inside of a string, we just need to insert the variable inside of double quotes but happens when we try to expand an object property?

Continue reading “How to Reference an Object Property in a String in PowerShell”

The “I’m Not Good Enough” Excuse has Officially Been Eradicated

All throughout my time in the tech communities, the common theme I keep hearing from people is “I’m not good enough” or “I could never do that!”. It’s frustrating to me because I know that every good IT guy or gal has knowledge in their head that’s people would love to learn about.

What’s stopping you?

TechSnips’ mission is to provide a platform for technology professionals to showcase their work via video screencasts. However, those that aren’t used to presenting and required more coaching than others tended to get frustrated when they were forced to re-record their screencasts due to quality concerns.

Now they don’t have to!

Continue reading “The “I’m Not Good Enough” Excuse has Officially Been Eradicated”

How to Run PowerShell Code by Invoking a Webhook in Azure Automation

Back in my day, we could only run PowerShell scripts on local Windows computers. Nowadays though. it’s amazing all of the different ways to execute PowerShell scripts. We not only have PowerShell on Linux but Azure Cloud Shell too! Azure Cloud Shell is an awesome way to quickly bring up a PowerShell environment to execute code in the cloud. Speaking of Azure, we’ve also got Azure Automation runbooks. Using PowerShell Azure Automation runbooks are another great way to get stuff done with PowerShell.

Whenever I build a PowerShell Azure Automation runbook, the trigger for the runbook has always been on a schedule or invoked via another runbook. However recently, I needed the ability to invoke it outside of Azure completely via a service we use here at TechSnips called Zapier.

Continue reading “How to Run PowerShell Code by Invoking a Webhook in Azure Automation”

A PowerShell Tool to Create IIS Websites

During one of my latest snips on How to Manage IIS Websites with PowerShell, I decided to create a PowerShell function to make it easier. This function wraps up everything I spoke about in that snip with some extra goodness as well. It’s a great function to use if you find yourself constantly creating IIS sites on a remote computer.

Continue reading “A PowerShell Tool to Create IIS Websites”

Introducing the TechSnips Paid Blog Post Program

TechSnips started as a video-only platform and continues to grow every week. However, we realize that not everyone prefers to learn new material via screencasts. This is why we are introducing the paid blog post program. The paid blog post program is a feature of TechSnips that allows approved contributors to supplement their screencast submissions with a blog post and get paid for it!

Continue reading “Introducing the TechSnips Paid Blog Post Program”

Introducing our VP of Content

Building a startup is a ton of work. When I first started, I thought I could do everything myself. Boy, was I wrong! Building a business from the ground up is like changing a light bulb. There is so much to do that I never even considered.
In this business, content is king. At TechSnips, we have to publish as many videos as possible as often as possible. Content is the most important part of our business. This is why I’ve decided to bring on someone to help.

Continue reading “Introducing our VP of Content”