CSR, My Old FriendPublished 2018-11-11
This is the story of how I figured out a stupid-simple solution to generating (and potentially parsing) ASN.1 and X.509 (for certificate requests in my case).
This is not a quick story. The, ultimately simple, discovery came after much (read: years of) toil, which was riddled with emotional turmoil and frustration. Hence, I'm not diving straight into the solution, but rather recount the "joy" of the journey.
(that said, feel free to skip ahead to the goodies - I won't know, so my feelings won't get hurt)
how did I come to be one of the few and unfortunate people that can understand and explain ASN.1?
If you want more of the details of what I actually learned, that's available now too:
(a.k.a How I got this emotional baggage)
I'm working on an in-home personal cloud, which I've always wanted to be a consumer device. Obviously I want it to be safe for the non-technical populace that I envision will one day be using it, so automating all of the security and presenting it in a non-technical way has always been a top priority.
At the time I started I had only a faint idea of how I might solve the problem of automating https. I was moving forward with development despite a seemingly insurmountable task up ahead... and then Mozilla + EFF announced Let's Encrypt. The timing literally could not have been more perfect.
The IGG had actually prototyped some of the ACME protocol in node (for which I've been a long-time developer and was already using for this project) before moving the client to python and the server to Go.
Someone had gotten a rudemantary demo working, which I forked and morphed and rebuilt as letsencrypt.js, which later became greenlock.js (now complete with Let's Encrypt v2 support) - and I even ported it to the browser (w00t!).
The Usual Suspects
Of course, none of them are complete enough to do all that needs to be done. They've all been built at different times, so they have different, incompatible ways of handling binary data, requiring you to write special one-off converters as you pass data between them to get a complete flow.
You have to use a different set in node than in the browser. And each browser has a varying level of support for the necessary features.
Fortunately, in recent months it has become possible to do pretty much everything in native node crypto and native webcrypto (in almost all browsers), except for one thing...
CSR, My old friend
Certificate Signing Requests (or CSRs) are a doozy. They're kind of like JWK+JWT of the dark ages.
The underlying format that is written in is called ASN.1 which is like the OG* XML (which is like the JSON of the dark ages, for those unfamiliar with XML). But layered on top of that is a standard called X.509, which is pretty much the OG XLST (think json-schema.org of the dark ages).
* that's Original Gangster for those not in the know
Summarized in a different way (and yet still step further down the rabbit hole):
- PEM is simply a base64 encoded DER file (with a plain-text header and footer)
- DER is the binary format of ASN.1, more or less a dark-ages BSON
- (you colud say DER is to UTF-8 as ASN.1 is to Unicode)
- ASN.1 would then be a sort of dark-ages object notation for anything
- X.509, then, is a dark-ages schema that uses ASN.1 to describe a specefic, concrete thing
So you see, there is no new thing under the sun. What's old is inevitably new again.
The General Problem... It's complicated
As programmers we like to make this overly complicated - but our hearts are in the right place. We try to do it for a good reason.
See, when we do things the "stupid" way we look... stupid. If we do it the dumb way it's easy for someone to point out "hey, that's dumb!" and, inevitably, excuse ourselves with "oh, this is just spaghetti code because... legacy/tech debt/learning" to to explain it away with less guilt.
To compensate we get clever. Too clever. We over-engineer. We design things to be so uber modular as if to solve all possible problems rather than just the problem at hand. (xkcd#947, anyone?)
Imagine that the JSON spec itself include everything on json-schema.org - that's near the level of entanglement that ASN.1 seems to have with its schemas, with every thing defined by some strange level of section, subsection, and paragraph header naming scheme that looks something like the dewey decimal system for object notation.
I mean, how the heck does a type end up with `1.2.840.1135220.127.116.11' as its identifier? How many types were there supposed to be in this system? Oh, all of them.
Arriving at an easier approach
Complicated standards - especially complicated binary standards - especially complicated binary standards that only really ever had a single use case (and were then abondened), as you might imagine, require complicated code to parse and generate. (these are "solved" problems, you see, that we needn't bother ourselves with any longer, generally speaking)
For so long I've wanted to find a less capable, stupider, but cleaner solution but everything seemed so complicated I thought that I was simply subject to the powers that be.
And then, finally, I was shown the light. Someone commented on my Let's Encrypt v2 Step by Step guide with a cobbled together, half-baked solution so stupid that I immediately believed that it was not just a solution that could work, but probably the way that anyone who has ever done this probably should have done it to begin with - instead of actually understanding ASN.1, just make a template to copy the bits that stay the same and paste in the values that are different.
Many thanks to the man who showed me how easy it could be to work backwards.
There's an old wise saying of our dark arts:
Remember, a few hours of trial and error can save you several minutes of looking at the README - @iamdeveloper
But equally true we must also concede to the counter point:
Generally a few hours of trial and error are necessary before the README makes any sense! - @frogfather
In my search to decompose the mystical CSR I quickly came across the golden nugget, https://lapo.it/asn1js/. However, I couldn't yet make sense of it. I needed to dive a little further.
Rather than approaching the problem from the point of the "general solution", I figured that if I could just generate a known-good CSR for just one key type and just one profile - say either 2048-bit RSA or a P-256 ECDSA - I could make several variations changing only exactly one thing at a time and learn how a CSR was constructed:
- which bits always stayed the same
- which bits represented the domain names
- which bits indicated length
- which bits indicated the number of domains
With a little help from @mathwhiz1212 I was able to create the necessary CSR permutations to do just that.
The basic script to generate a keypair and CSR is as follows:
# generate keypair openssl genrsa -out ./privkey-rsa-2048.pem 2048 openssl rsa -in ./privkey-rsa-2048.pem -pubout -out ./pubkey-rsa-2048.pem # generate csr (with inline config file) openssl req -key ./privkey-rsa-2048.pem -new -nodes \ -config <(printf "[req] prompt = no req_extensions = req_ext distinguished_name = dn [ dn ] CN = example.com [ req_ext ] subjectAltName = @alt_names [ alt_names ] DNS.1 = example.com DNS.2 = www.example.com") \ -out example.com-www-rsa-2048.csr node convert-to-der.js example.com-www-rsa-2048.csr
These were my target variations:
- Trial 1 - Control Group / Base Case
- Trial 2 - Modify a single letter by one bit (boundary checking)
- Trial 3 - Add a single letter (bits identifying length)
- Trial 4 - Add another domain (how arrays are handled)
- Trial 5 - Try a different key size (meta data location)
- Trial 6 - Try a different key type (format metadata)
And I was able to use Hex Fiend (try DHEX on linux) to diff the files to see what was changing. After a few rounds of opening two files individually and selecting "Compare" from the File menu to actually start the diff, I was able to "get my bearings", if you will, and the ASN.1 Debugger went from looking like gibberish to making perfect sense. Yay!
Voila! I created my own CSR Game Genie*!
* this is exactly how the Game Genie worked (and how all reverse engineering works): targeting and tracking changes
Up Next: ASN.1 and X.509 Explained
Update: The conclusion is here:
Here we are at the meat and potatoes, but I'm tired and want to head to bed.
But the conclusion is coming, I promise!
Update (Tuesday Nov 13): There are just a few kinks I need to work out before I do the next post.
Update (Thursday Nov 15): I'm going to need this next weekend to finish this. I ran into some stuff that needs further exploration.
Update (Tuesday Nov 20): Sooooo.... close
Update (Sunday Nov 25): Finally working on the article today.
The result of my learning became these great (in my opinion) solutions:
- ECDSA-CSR.js - Generate a CSR with EC keys
- Eckles.js (w/ CLI) - Generate EC keys, convert between JWK, PEM, and SSH, and generate CSRs
- RSA-CSR.js - Generate a CSR with RSA keys
- Rasha.js (w/ CLI) - A lightweight VanillaJS library to generate RSA keys, convert between JWK, PEM, and SSH, and generate CSRs
By AJ ONeal
Did I make your day?
Buy me a coffee