It's crazy just how WRONG the first several search results for things like "how to upload a file with s3 in node" and "node s3 upload" are.

I don't think I can really make a dent in the universe on this one, but for those that want to see the correct way to upload to s3 with node, here it is:

TL;DR The Right Way

Don't use node, use Go. The end. (just kidding... kinda...)

Here's how you can upload very large (or small, or anything in-between) files, images, documents, videos, etc to S3:

'use strict';

var AWS = require('aws-sdk');

// You don't have to use environment variables.
// You can store your AWS secrets in a config file just fine.
// This works great for local development.
var config = require('./config.js');

// For Docker and cloud deployments, you probably want to use ENVs.
// In that case you can fall back to that when a local config isn't available.
// (you could also use the `dotenv` package for local development)
var AWS_ACCESS_KEY = config.awsAccessKey || process.env.AWS_ACCESS_KEY;
var AWS_SECRET_ACCESS_KEY = config.awsSecretAccessKey || process.env.AWS_SECRET_ACCESS_KEY;

var s3 = new AWS.S3({
    accessKeyId: AWS_ACCESS_KEY,
    secretAccessKey: AWS_SECRET_ACCESS_KEY
});

var fs = require('fs');

function uploadToS3(bucketName, keyPrefix, filePath) {
    // ex: /path/to/my-picture.png becomes my-picture.png
    var fileName = path.basename(filePath);
    var fileStream = fs.createReadStream(filePath);

    // If you want to save to "my-bucket/{prefix}/{filename}"
    //                    ex: "my-bucket/my-pictures-folder/my-picture.png"
    var keyName = path.join(keyPrefix, fileName);

    return new Promise(function(resolve, reject) {
        fileStream.once('error', reject);
        s3.upload(
            {
                Bucket: bucketName,
                Key: keyName,
                Body: fileStream
            },
            function(err, result) {
                if (err) {
                    reject(err);
                    return;
                }

                resolve(result);
            }
        );
    });
}

Usage Example:

var filePath = '/path/to/selfie.jpg';

uploadToS3('my-bucket-name', 'my-picture-folder', filePath).then(function(result) {
    console.info('Success! Uploaded ' + filePath + ' to ' + result.Location);
});

Explainer: Buckets & Keys

AWS S3 has a concept of a "bucket" and a "key".

There are no folders!

Buckets

If you're familiar with Windows, this is basically like a "Drive", such as _C:_, and a "File".

If you're familiar with Mac, this is basically like a "Volume", such as when you plug in an SD card and it shows up on your Desktop and in /Volumes/.

If you're familiar with Linux, it's like a "Mount", such as /mnt/my_drive.

Keys

We're pretty much all familiar with files - y'know _selfie.jpeg_ and such. Normally you put files in a folder, on a drive.

You might have C:\Users\Jon\My Pictures\selfie.jpg, for example.

AWS DOES NOT HAVE FOLDERS.

Instead, you're allowed to have / as part of the name of a file - which they call the "key" of an "object" - and you're allowed to limit a view of files in a bucket to only the files that match a particular prefix.

In the web interface this kinda looks like folders, but it's not.

Explainer: Everyone Else is WRONG

I'm completely baffled, but when I searched for this myself I was expecting a quick and easy snippet, but instead what I found on the first page of Google results was an absolute dumpster fire.

  • DO NOT use fs.readFile, use fs.createReadStream
  • DO NOT base64 encode your files... unless you intend to on purpose... which you don't
  • DO NOT JSON.toString() a node buffer or string of text... that doesn't make sense
  • DO NOT throw s3err; in the AWS callback... you can't catch it, it'll crash your app
  • DO NOT convert node buffers to binary strings, just use the buffer

I dunno how so many people got their examples so wrong... but they did.

That said, it gave me enough info to do the right thing.

Even more baffling: Why are so many people linking to examples that obviously don't work?


By AJ ONeal

If you loved this and want more like it, sign up!


Did I make your day?
Buy me a coffeeBuy me a coffee  

(you can learn about the bigger picture I'm working towards on my patreon page )



Published

2019-9-13



Buy me a coffeeBuy me a coffee


  73% of Goal Reached


Want more like this?