While recently working on some server side web request in a Rails application, I ran into a rather interesting problem. We’ve been working on some OAuth authentication and Restful services with various social networks. (Facebook, Google+, LinkedIn, Twitter) All was going well in development and production was even working with the first few services. That’s when we went a service to far, and spent a week learning about the Certificate Authority and certificate verfication.
Our non-working service was Twitter. Nothing to crazy here, “It worked on my development machine”. First thing to check was the production “Consumer Key” and “Consumer Secret”. All looked well. Reviewing the application error log didn’t reveal much, only the following error generated by the Ruby OAuth library:
OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed):
What the *#-$@. Okay, so it doesn’t like something about our https request, but its working with everything else?
Some quick googling turned up a link to the following Stack Overflow posting:
We experimented with the Net::HTTP class with a little help from the following link:
Wow, Twitter works if we turn SSL verification off! I know, 80% of developers would stop here. It works!
Do we really want to turn off SSL verification in our nice new Social Network application? Probably not! Here’s a good review on not turning it off:
Okay, so how do we make it work? We verified that the CA cert file was where it was supposed to be and that Ruby was utilizing it. All looked good.
A little more googling turned up the following Twitter post:
This brought on the Aha moment, the Certificate Authority for Twitter, Verisign, had revoked its root certificate that existed in the /etc/ssl/cert/ca-certificates.crt on our production Ubuntu server. Now, everyone should pay attention to that last statement! We take for granted that our OS vendors, Microsoft, Apple, Ubuntu, etc, package and update the Root CA keys for us. It should also be noted that Verisign revoked this Root CA as mitigation to compromise of its CA!
Sounds like a simple problem to fix, we’ll just run an OS update on the production server, have new Root CA keys, and be up and running.
LOL:( It just so happens that our Ubuntu server is a bit behind the times running 9.10. It’s been retired by Ubuntu and the package servers have been turned off! No updating the production server, without a major Upgrade to Ubtuntu 10.04LTS at a minimum.
How do we update the Root CA’s ourselve? A cry for help to the local LUG (COLUG) and Ruby user groups (CRB) list server solicited a limited response. (A single response from each)
It was the weekend by then and we spent some time digging in to the question with great results.
Debian flavors of Linux derive their Root CA certificates from /usr/share/ca-certificates as defined in the /etc/ca-certificates.conf. This contains a sub directory for each Root CA, with their respective directories containing various keys. A shell script, update-ca-certificates, extracts these keys from the various directories into the /etc/ssl/ca-certificates.crt file.
So, to update the Root CA keys on Debian/Ubuntu systems with the new Verisign Root Certificate:
1. Download the Versign Certificates - http://www.verisign.com/support/roots.html
2. Create /usr/share/ca-certificates/verisign.com
3. Copy the Verisign PEM files into /usr/share/ca-certificates/verisign.com, renaming them from *.pem to *.crt
4. Edit the /usr/share/ca-certificates/verisign.com/*.pem files for the CR/LF’s format that the update-ca-certifcates script desires. Hint: Look at one of the existing CA’s .pem files /usr/share/ca-certificates/mozilla/*.crt.
5. Edit the /etc/ca-certificates.conf to remove the old Verisign certificate directories and utilize the new Verisign certificates.
6. Run “update-ca-certificates -v -f” to construct the /etc/ssl/cert/ca-sertificates.crt.
Your friend in this endeavor – man update-ca-certificates