It's crazy just how WRONG the first several search results for things like "how to upload a file with s3 in node" and "node s3 upload" are.

I don't think I can really make a dent in the universe on this one, but for those that want to see the correct way to upload to s3 with node, here it is:

Don't use node, use Go.
The end.

(just kidding... kinda...)

...

TL;DR The Right Way

Here's how you can upload very large (or small, or anything in-between) files, images, documents, videos, etc to S3:

'use strict';

var AWS = require('aws-sdk');

// You don't have to use environment variables.
// You can store your AWS secrets in a config file just fine.
// This works great for local development.
var config = require('./config.js');

// For Docker and cloud deployments, you probably want to use ENVs.
// In that case you can fall back to that when a local config isn't available.
// (you could also use the `dotenv` package for local development)
var AWS_ACCESS_KEY_ID = config.awsAccessKeyId || process.env.AWS_ACCESS_KEY_ID;
var AWS_SECRET_ACCESS_KEY = config.awsSecretAccessKey || process.env.AWS_SECRET_ACCESS_KEY;

var s3 = new AWS.S3({
    accessKeyId: AWS_ACCESS_KEY_ID,
    secretAccessKey: AWS_SECRET_ACCESS_KEY
});

var fs = require('fs');
var path = require('path');

function uploadToS3(bucketName, keyPrefix, filePath) {
    // ex: /path/to/my-picture.png becomes my-picture.png
    var fileName = path.basename(filePath);
    var fileStream = fs.createReadStream(filePath);

    // If you want to save to "my-bucket/{prefix}/{filename}"
    //                    ex: "my-bucket/my-pictures-folder/my-picture.png"
    var keyName = path.join(keyPrefix, fileName);

    return new Promise(function(resolve, reject) {
        fileStream.once('error', reject);
        s3.upload({
            Bucket: bucketName,
            Key: keyName,
            Body: fileStream
        })
            .promise()
            .then(resolve, reject);
    });
}

Usage Example:

var filePath = '/path/to/selfie.jpg';

uploadToS3('my-bucket-name', 'my-picture-folder', filePath).then(function(result) {
    console.info('Success! Uploaded ' + filePath + ' to ' + result.Location);
});

Explainer: Buckets & Keys

AWS S3 has a concept of a "bucket" and a "key".

There are no folders!

Buckets

If you're familiar with Windows, this is basically like a "Drive", such as _C:_, and a "File".

If you're familiar with Mac, this is basically like a "Volume", such as when you plug in an SD card and it shows up on your Desktop and in /Volumes/.

If you're familiar with Linux, it's like a "Mount", such as /mnt/my_drive.

Keys

We're pretty much all familiar with files - y'know selfie.jpeg and such. Normally you put files in a folder, on a drive.

You might have C:\Users\Jon\My Pictures\selfie.jpg, for example.

AWS DOES NOT HAVE FOLDERS.

Instead, you're allowed to have / as part of the name of a file - which they call the "key" of an "object" - and you're allowed to limit a view of files in a bucket to only the files that match a particular prefix.

In the web interface this kinda looks like folders, but it's not.

Explainer: Everyone Else is WRONG

I'm completely baffled, but when I searched for this myself I was expecting a quick and easy snippet, but instead what I found on the first page of Google results was an absolute dumpster fire.

• DO NOT use fs.readFile, use fs.createReadStream
• DO NOT base64 encode your files... unless you intend to on purpose... which you don't
• DO NOT JSON.toString() a node buffer or string of text... that doesn't make sense
• DO NOT throw s3err; in the AWS callback... you can't catch it, it'll crash your app
• DO NOT convert node buffers to binary strings, just use the buffer

I dunno how so many people got their examples so wrong... but they did.

That said, it gave me enough info to do the right thing.

Even more baffling: Why are so many people linking to examples that obviously don't work?

By AJ ONeal


Did I make your day?
Buy me a coffee  

(you can learn about the bigger picture I'm working towards on my patreon page )

---

---