A big oops with Greenlock
Due to a mistake that I made, many greenlock users received several strange emails last night that looked like spam.
As soon as I found out about the problem I corrected it and (thought) I sent out an apology to all of those affected. However, it hadn't actually sent so I wrote an edited follow up and sent again.
However, as this was the first message sent out from my mail server, it was already marked as spam for many people.
For that reason, I'm also publishing my full-length apology and linking to it so that others can find it.
My Apology
I want to outline what went wrong and what I'm going to do to fix it now and in the future:
Here's what went wrong:
The "good" news is that the idiot mistake was caused directly by me, not a security breach of any kind in greenlock or related projects.
The bad news is that a large number of people got about 10 emails that looked like spam and contained an email address (which is one email address too many).
The importance of privacy and trust cannot be overstated. I truly regret this.
Here's how it went wrong:
communityMember: true
When the option communityMember: true is selected, we wanted to make it so that
one of the 4 of us that is working on greenlock or telebit (or one of our other
community-oriented projects) get an email so that we can reach out and ask you
"Did it work for you? Is there anything we can do to help?"
The email that is supposed to go out isn't so generic as the one that you received. It does show the email address (otherwise we can't do the reach out that was asked), but it keeps the domain anonymous.
Last night I made a change to the mail server so that we would actually start getting the emails and be able to reach out to people. I made an unfortunate mistake in which the fall-through case (people who ARE NOT supposed to get an email) was handled in almost the worst way possible - sending out a very generic email that looks like spam to hundreds of people. The desired behavior was that only the catchall "rob the robot" (which forwards to me) would get the generic email, and I would know to debug that there was an unhandled case.
When tried a few test messages, I didn't test to see what happened when an email wasn't supposed to send. Big mistake.
securityUpdates
When I added the securityUpdates option to greenlock.js, I forgot to also update the greenlock-express.js documentation with the option as well. I thought that I had, but now it's obvious that I haven't.
This is for only the most important (pure transactional, non-community updates) messages. I believe I've used this only once - to send a message to everyone about a breaking change in the Let's Encrypt v2 API (after I had deployed it, but before it was "stable") that would have caused unexpected service disruption.
This needs to be made clear in the greenlock-express.js documentation (and perhaps more clear in the greenlock.js documentation).
Here's how I'm going to fix it for the future:
I've already fixed the bug in the mail settings. Emails that should not go out are not going out (and as before, ones that should go out are only going out to those that should receive them).
That's the biggest issue.
I'm also going to update the greenlock documentation (and documentation in the projects that use it directly) and create a proper privacy policy.
These things happen, but that doesn't make them okay.
As many of you will recall, there was a similar mistake a few months ago related to the babel project - one of the largest projects on npm - in which when npm had to send out an email to the entire userbase about the hacked module which was running arbitrary code on people's machines that was harvesting email addresses and passwords.
Obviously the affect of an email going out is not anywhere near as dangerous as that, however, whereas the babel project has a huge level of trust with its users and was able to regain that quickly, greenlock is not a huge project and my mistake violated trust.
Again, I apologize.
AJ ONeal
(385) xxx-xxxx
https://greenlock.domains
On Wed, Nov 14, 2018 at 7:56 AM AJ @ Greenlock aj@greenlock.domains wrote: Hello everyone, I want to sincerely, sincerely apologize about the several emails that went out last night.
That SHOULD NOT have happened and I feel VERY VERY bad about it.
A while back I added the
communityMemberandsecurityUpdatesoption to greenlock so that people who are interested in important updates can opt in and that those who don't want to hear even about security updates can opt-out.Last night I made a change to the mail settings and there were about a dozen messages that went out to everyone.
I cannot apologize enough for this. I understand how gravely serious this problem is. I have fixed the problem. I bear much guilt over this. It weighs on me greatly and I want to do anything that I can to make it right.
I know that my apology cannot be enough for the loss of making a mistake that seriously breaks trust. Please contact me if you have any further questions - or if you just need to yell at me to get some raw emotions off your chest - I deserve it.
For many of you this is the first contact from me and I am SO sorry that it had to be like this.
Sincerely and regretfully,
AJ ONeal
(801) xxx-xxxx
https://greenlock.domains
By AJ ONeal
Did I make your day?
Buy me a coffee
(you can learn about the bigger picture I'm working towards on my patreon page )
Published
Tweet
Buy me a coffee
73% of Goal Reached