Tuesday, June 20, 2023

Uploading to S3 with Presigned upload URLs

By Michoel Samuels • 🍿 6 min. read

How S3 presigned upload URLS work

AWS S3 is my preferred way to store arbitrary content on the web. Storage is affordable, robust, and practically infinite in quantity.

There are a number of ways S3 can be used to store arbitrary user content.

Like many cloud services, configuring this properly can be error-prone and immensely frustrating.

This article walks through the precise setup I use to generate signed upload URLs that actually work.

Technical overview

This is my understanding of how presigned urls work:

A /Pre-signed URL/ is used to allow untrusted users to temporary access to private S3 resources. (Example: upload an image to a private S3 bucket).

The URL is a string consists of: A. AWS URL of the object to be accessed, and B. a signed base64 string describing the HTTP request to be permitted (HTTP verb (i.e. PUT), the S3 object key, and the S3 operation (putObject, etc.), and your Access Key ID).

The string is signed by your AWS Secret Access Private Key.

When the URL is used, S3 takes the action that is being attempted (PUT mybucket.s3.amazonaws.com/nazgul.png) , extracts the Access Key ID and calculates what the signature should look like. If any of the components don't exactly match, (Say, using POSTinstead of PUT, using the wrong object key, etc), the action will fail with SignatureDoesNotMatch. S3 cannot tell you what part of the request is being used incorrectly, all it knows is that the signature does not match

If the signature passes, S3 will temporarily sign in as the IAM user who signed the URL and will execute with their permissions. If the signing IAM user does not have permission to perform the requested action, the request will fail with AccessDenied. Permission can be granted by adding a rule to the S3 bucket's Permissions > Bucket Policy , or by adding an IAM Security Policy granting access to the bucket to the IAM user.

If the IAM user permission check passes, the action will be performed. The signed URL will be immediately expired to prevent reuse.

The IAM user that generates the URL must have the S3 permissions to access the bucket (The S3 permissions getObject, putObject, listBucket, etc.)

Setup S3

Create the S3 bucket.
Add the CORS policy from the Appendix in Permissions > CORS Configuration. This will allow JavaScript in the browser to talk to S3.

Lambda

Create a new Node.js Lambda function
Paste in the code from the Appendix, customizing it with your bucket name and anything else you may want to add.
Expose it to the public web using API Gateway (In the Lambda console of your function, click + Add Trigger and select API Gateway)
Locate the Execution Role. Click to visit it in the IAM console.
In the IAM console of the execution role, create a new Policy by clicking Attach Policy > Create Policy. In the wizard, grant access to the S3 bucket we've just created.

✅ Lambda is ready to start signing URLs

Appendix 1: Lambda function

'use strict';
console.log('Loading generate presigned URL function');
var AWS = require('aws-sdk');
var s3 = new AWS.S3({
	signatureVersion: 'v4',
});
exports.handler = (event, context, callback) => {
		
		const url = s3.getSignedUrl('putObject', {
				Bucket: 'gemach-app-images', // Your bucket name here
				Key: event.queryStringParameters.filekey,
				ContentType: event.queryStringParameters.contentType,
				ACL: 'public-read',
				Expires: 600, //expiry time in sec
			});
		var name ;
		var responseCode = 200;
				console.log("request: " + JSON.stringify(event));
				
				if (event.queryStringParameters !== null && event.queryStringParameters !== undefined) {
				if (event.queryStringParameters.filekey !== undefined && event.queryStringParameters.filekey !== null && event.queryStringParameters.filekey !== "") {
						console.log("Received name: " + event.queryStringParameters.name);
						name = event.queryStringParameters.name;
				}
			 
				if (event.queryStringParameters.httpStatus !== undefined && event.queryStringParameters.httpStatus !== null && event.queryStringParameters.httpStatus !== "") {
						console.log("Received http status: " + event.queryStringParameters.httpStatus);
						responseCode = event.queryStringParameters.httpStatus;
				}
		}
 
		var responseBody = {
				signed_url: url,
		};
		var response = {
				statusCode: responseCode,
				headers: {
						"Access-Control-Allow-Origin": "*"
				},
				body: JSON.stringify(responseBody)
		};
		console.log("response: " + JSON.stringify(response))
	
		callback(null, response);
	
};

Appendix 2; Bucket Policy

This policy make the bucket contents public

{
		"Version": "2012-10-17",
		"Id": "Policy1564527997207",
		"Statement": [
				{
						"Sid": "Stmt1564527321604",
						"Effect": "Allow",
						"Principal": "*",
						"Action": "s3:GetObject",
						"Resource": "arn:aws:s3:::gemach-app-images/*"
				}
		]
}

Appendix 3: CORS Configuration

Allows

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
		<AllowedOrigin>http://127.0.0.1:8080</AllowedOrigin>
		<AllowedMethod>GET</AllowedMethod>
		<AllowedMethod>PUT</AllowedMethod>
		<AllowedMethod>POST</AllowedMethod>
		<AllowedMethod>DELETE</AllowedMethod>
		<MaxAgeSeconds>3000</MaxAgeSeconds>
		<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>