Copying files to S3 with aws-cli¶
There would be times when you want to upload multiple files from your laptop to S3. This document describes the process for it.
Installing aws-cli on mac¶
brew install awscli
Configure AWS Credentials¶
- Enter your awsAccessKeyId - Entery your awsSecretAccessKey
View S3 Buckets¶
- aws s3 ls
View S3 Directory¶
- aws s3 ls s3://bucket_name/dir1/
Copy files to S3¶
Copy all files from local_direcory to s3://bucket-name/dir1:
aws s3 cp local_directory s3://bucket-name/dir1 --recursive
Delete All Files in Directory¶
- aws s3 rm s3://bucket_name/dir1/ –recursive
Setting Roles and Policies for EMR¶
In order to be able to access S3 files from the EMR cluster, attach the AmazonS3FullAccess Policy to the EMRDefaultRole.
Now the EMR cluster would have access to the S3 buckets.
REFERENCE : Creating Access Key & Secret Key¶
- You’ll need create a user with programmatic access by following the steps here (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html).
- Next, you’ll create an IAM policy that defines what this user has access to in your AWS account. It’s important to only grant this user minimal access within your account. See this documentation for how to create IAM policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html).
- Finally, you’ll create an access key and secret key for this user (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey).
Note It’s important to regularly rotate your access and secret keys. See this documentation for more information (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#rotating_access_keys_console)