What is Boto3?
Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python. It enables Python developers to create, configure, and manage AWS services and resources, including Amazon S3, Amazon EC2, Amazon DynamoDB, and Amazon CloudFormation, among others.
Why do we use it?
With Boto3, Python developers can build applications that integrate with AWS services using just a few lines of code. This makes it easier to build applications that can scale with the needs of the business without having to worry about the underlying infrastructure.
Key Features of Boto3?
Easy-to-use API: Boto3 provides an easy-to-use API that allows developers to interact with AWS services and resources using Python.
Comprehensive documentation: Boto3 has comprehensive documentation that makes it easy for developers to get started with the SDK.
High-level abstractions: Boto3 provides high-level abstractions for AWS services and resources, which makes it easy for developers to write code without worrying about low-level details.
Customizable: Boto3 is highly customizable, which allows developers to tailor the SDK to their specific needs.
Scalable: Boto3 is designed to scale with the needs of the business, making it easy to build applications that can grow and adapt over time.
How to set up Boto3?
To install and set up Boto3, you can follow these steps:
Install Python: Boto3 requires Python 2.7.x or Python 3.4+ to run. If you don’t have Python installed, you can download and install it from the official Python website: https://www.python.org/downloads/
Install pip: pip is a package manager for Python that makes it easy to install and manage Python packages, including Boto3. If you don’t already have pip installed, you can install it by following the instructions here: https://pip.pypa.io/en/stable/installing/
# Run the following command in your terminal or command prompt:
# Install Boto3: Once you have pip installed
# install Boto3 by
pip install boto3
#Set up AWS credentials: Before you can use Boto3 to interact with AWS services
# Mention your aws access key and aws secret access key and region for aws cli using command below:-
$ aws configure
AWS Access Key ID [None]: "your access key"
AWS Secret Access Key [None]: "your secret key"
Default region name [None]: "give region name"
Default output format [None]: "leave default"
Few examples of using boto3 for s3:
import logging
import boto3
#creating client so that we can connect to s3.
# in place of s3 client we can use any services that is supported by boto3
def client():
s3 = boto3.client('s3')
return s3
#########################################
###### For creating bucket ###############
##########################################
def create_bucket_s3(bucket_name, region):
location = {"LocationConstraint": region}
client().create_bucket(
Bucket=bucket_name, CreateBucketConfiguration=location)
create_bucket("bucket-name", "us-east-1")
#########################################
###### For creating bucket Policy ###############
##########################################
def create_bucket_policy(Bucket_name):
bucket_policy = {
'Version': '2012-10-17',
'Statement': [{
'Sid': 'AddPerm',
'Effect': 'Allow',
'Principal': '*',
'Action': ['s3:GetObject'],
'Resource': f'arn:aws:s3:::{Bucket_name}/*'
}]
}
policy_string = json.dumps(bucket_policy)
return client().put_bucket_policy(Bucket=Bucket_name, Policy=policy_string)
create_bucket_policy("nayanshivhare1234")
#########################################
###### For Listing bucket ###############
##########################################
bucket_list = []
def list_bucket():
list_of_bucket = client().list_buckets()
for bucket in list_of_bucket["Buckets"]:
bucket_list.append(bucket["Name"])
return bucket_list
list_bucket()
print(bucket_list)
#########################################
###### For Listing Object########################
##########################################
def list_object(Bucket_name):
object_in_bucket = client().list_objects_v2(Bucket=Bucket_name)
for object in object_in_bucket["Contents"]:
print(object["Key"])
list_object("nsdfffivhare1234")
#########################################
###### For Uploading Object########################
##########################################
def upload_object(source_file, bucket_name, dest_obj_name):
client().upload_file(source_file, bucket_name, dest_obj_name)
print("file-uploaded")
upload_object("./file.txt", "mybucketname", "file1.txt")
#########################################
###### For Multi-part upload########################
##########################################
def multi_part_upload(sourcefile, bucket, destfile):
# covert byte to gb
GB = 1024 ** 3
config = TransferConfig(multipart_threshold=5*GB)
client().upload_file(sourcefile, bucket, destfile, Config=config)
print("In progress")
multi_part_upload("./Nayan/list.txt", "nayanshivhare1234", "file1.txt")
print("successfull")
#########################################
###### For Downloading objects###############
##########################################
def download_files(Bucket_name, Object_name, Downloaded_file_name):
file = client().download_file(Bucket_name, Object_name, Downloaded_file_name)
print("file downloaded######################")
download_files("xyzbucketname", "naya1n.pdf", "file1.pdf")
#########################################
###### For creating pre-signed ###############
##########################################
def pre_signed_url(Bucket_name, File_name, Expire):
url = client().generate_presigned_url('get_object',
Params={
"Bucket": Bucket_name,
"Key": File_name},
ExpiresIn=Expire)
return url
print(pre_signed_url("nayanshivhare1234", "naya1n.pdf", 120))
#########################################################
###### Copy file from source bucket to destination Bucket###############
#########################################################
def copy_to_dest(Destination_bucket, source_bucket, source_filename, saved_filename):
copy = client().copy_object(
Bucket=Destination_bucket,
CopySource=f'/{source_bucket}/{source_filename}',
Key=saved_filename
)
for key, value in copy["ResponseMetadata"].items():
if key == "HTTPStatusCode":
if value == 200:
print("SuccessFully- copied to Destination ")
else:
print("error occured")
copy_to_dest("elasticbeanstalk", "nayanshivhare1234", "naya1n.pdf", "nmo.pdf")
Boto3 can also be used with data processing tools such as Panda and numpy:
pip install boto3 pandas
import boto3
import pandas as pd
# example code snippet that loads a CSV file from an S3 bucket into a pandas DataFrame:
s3 = boto3.resource('s3')
bucket = s3.Bucket('your-bucket-name')
obj = bucket.Object('your-file-name.csv')
body = obj.get()['Body']
df = pd.read_csv(body)
# Once you have loaded the data into a pandas DataFrame, you can process it using pandas functions. For example, you can use the groupby function to group the data by a particular column and calculate the mean of another column:
grouped = df.groupby('category')['sales'].mean()
print(grouped)
Conclusion:
Boto3 is a powerful SDK for Python developers who want to build applications that integrate with AWS services and resources. Its easy-to-use API, comprehensive documentation, high-level abstractions, and scalability make it a popular choice for developers who want to automate AWS infrastructure or process and analyze data stored in AWS service.
Reference:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html