Unicode String to a UTF-8 TypedArray Buffer in JavaScript

See also

• Unicode, UTF-8, TypeArray, Uint8Array, JavaScript, and You
• Direct TypedArray to Base64 JavaScript conversion

This code was inspired by Unibabel JS, which in turn was inspired by MDN's The Unicode Problem.

This is truly "vanilla" JavaScript that can easily be adapted to work even in weird environments (such as Windows Scripting Host, TI's ARM build system, Rhino, node.js, MSIE6).

Note: Working in weird environments is not a design goal of this code. Hence, I don't do anything "wonky" to acheive that. To run this in MSIE6 you would simply need to replace Uint8Array() with Array() and provide a a polyfill for 'map' and 'forEach'.

from String to Uint8Array

'use strict';

// string to uint array
function unicodeStringToTypedArray(s) {
    var escstr = encodeURIComponent(s);
    var binstr = escstr.replace(/%([0-9A-F]{2})/g, function(match, p1) {
        return String.fromCharCode('0x' + p1);
    });
    var ua = new Uint8Array(binstr.length);
    Array.prototype.forEach.call(binstr, function (ch, i) {
        ua[i] = ch.charCodeAt(0);
    });
    return ua;
}

Demo

var unicode = "I ½ ♥ 💩";
var buf = unicodeStringToTypedArray(data);
var arr = Array.prototype.slice.call(buf);
// [73, 32, 194, 189, 32, 226, 153, 165, 32, 240, 159, 146, 169];

from Uint8Array to String

'use strict';

// uint array to string
function typedArrayToUnicodeString(ua) {
    var binstr = Array.prototype.map.call(ua, function (ch) {
        return String.fromCharCode(ch);
    }).join('');
    var escstr = binstr.replace(/(.)/g, function (m, p) {
        var code = p.charCodeAt(p).toString(16).toUpperCase();
        if (code.length < 2) {
            code = '0' + code;
        }
        return '%' + code;
    });
    return decodeURIComponent(escstr);
}

Demo

var arr = [73, 32, 194, 189, 32, 226, 153, 165, 32, 240, 159, 146, 169];
var data = new Uint8Array(arr);
var unicode = typedArrayToUnicodeString(data); // "I ½ ♥ 💩";

This needs to be standardized!

This is the pull request I made to Netflix today.

And this is my rant-tweet to @ts36

Please contact your local JavaScript standardization committee to recommend that a feature be standardized in JavaScript.

By AJ ONeal


Did I make your day?
Buy me a coffee  

(you can learn about the bigger picture I'm working towards on my patreon page )

---

---