Certificate hashes using Base64, DER, ASN.1, PkiPath

Daniel_Danopia · October 19, 2009, 11:00pm

I’m trying to write a server for a communications system
(coughgoogle wavecough) but I’m having some problems with signing
the data payloads. After a while I managed to find this code to sign
the data:

require ‘base64’
require ‘openssl’
include OpenSSL
include PKey
include Digest

private_key = RSA.new(File.open("/Users/andy/tmp/private.pem").read)
signature = private_key.sign(OpenSSL::Digest::SHA1.new, “Some text to sign”)
puts Base64.encode64(signature)

A few tweaks (I hate includes) and I can sign data. Now, however, the
protocol wants me to include a hash of the certificate. After a solid
hour of googling it looks like it wants a Base64 encoding (I got that
part) of a PkiPath of a DER-encoded cert chain. I can’t figure out how
to generate the last two in Ruby. I do have reference code in Java but
it just calls some methods in the framework, so it doesn’t help much.

  CertificateFactory certFactory = CertificateFactory.getInstance

(X509);
CertPath path = certFactory.generateCertPath(certs);
byte[] encodedCertPath = path.getEncoded(PKI_PATH_ENCODING);
MessageDigest digest = MessageDigest.getInstance(
AlgorithmUtil.getJceName(getHashAlgorithm()));
return digest.digest(encodedCertPath);

Any tips for generating this in Ruby?

Daniel_Danopia · October 20, 2009, 9:59am

Daniel Danopia wrote:

it wants a Base64 encoding (I got that
part) of a PkiPath of a DER-encoded cert chain. I can’t figure out how
to generate the last two in Ruby.

Do you have a test case you can share - i.e. an actual certificate in
PEM format, together with the corresponding hash?

Ruby provides a very thin layer on top of OpenSSL’s API, so if you can
answer the question for openssl you can probably do it in Ruby.

Googling for “openssl pkipath” I found this, which at least defines the
pkipath:
http://marc.info/?l=openssl-users&m=109655589300276&w=2

Encoding an ASN1 ‘Sequence’ is fairly straightforward, and so is
converting it to DER.

OpenSSL::ASN1::Sequence.new([OpenSSL::ASN1::IA5String.new(“hello”), OpenSSL::ASN1::IA5String.new(“world”)]).to_der
=> “0\016\026\005hello\026\005world”

So my first attempt would be something along the lines of:

c1 = OpenSSL::X509::Certificate.new(File.read(“/etc/ssl/certs/Verisign_Class_1_Public_Primary_Certification_Authority.pem”))
c2 = OpenSSL::X509::Certificate.new(File.read(“/etc/ssl/certs/Verisign_Class_1_Public_Primary_Certification_Authority_-_G3.pem”))
OpenSSL::ASN1::Sequence.new([c1,c2])

If you have the source code to the Java libraries referred to in the
example code, that may help too.

Daniel_Danopia · October 21, 2009, 12:20am

It seems like the hardest part is that it gives a hash. I think I’m
going to add some debug code into the Java and print out a Base64 pre-
hashing to see what is going on.

Daniel_Danopia · October 21, 2009, 9:43am

Daniel Danopia wrote:

It seems like the hardest part is that it gives a hash.

That’s the easiest part to implement, although it does hide the
structure of what you’re hashing

data = “hello”
require ‘digest/sha1’
puts [Digest::SHA1.digest(data)].pack(“m”)

I think I’m
going to add some debug code into the Java and print out a Base64 pre-
hashing to see what is going on.

That’s a good idea. You can then see if it’s really an ASN1 Sequence of
one or more certificates.

Full specs of ASN1 and DER/BER are in these PDFs:

HTH,

Brian.

Daniel_Danopia · October 21, 2009, 11:45pm

I know how to hash, I said it’s the hard part because I don’t know how
far off I am until I am right

Here’s the Base64 of my .cert file (I have a .key and .cert):
MIIDqDCCAxGgAwIBAgIJAMIj9IivtNDGMA0GCSqGSIb3DQEBBQUAMIGVMQswCQYDVQQGEwJVUzETMBEGA1UECBMKTmV3IEplcnNleTEUMBIGA1UEBxMLTGFuZGlzdmlsbGUxEDAOBgNVBAoTB0Rhbm9waWExFDASBgNVBAsTC0dvb2dsZSBXYXZlMRQwEgYDVQQDEwtkYW5vcGlhLm5ldDEdMBsGCSqGSIb3DQEJARYObWVAZGFub3BpYS5uZXQwHhcNMDkxMDE3MjE0MjUxWhcNMTAxMDE3MjE0MjUxWjCBlTELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0xhbmRpc3ZpbGxlMRAwDgYDVQQKEwdEYW5vcGlhMRQwEgYDVQQLEwtHb29nbGUgV2F2ZTEUMBIGA1UEAxMLZGFub3BpYS5uZXQxHTAbBgkqhkiG9w0BCQEWDm1lQGRhbm9waWEubmV0MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDHnWFfWfm6DNWrSarAfklo1EiJlDUEuik3U
+F7b+Xuw66WwGLS9a55KwSc3QKLbf/HjNf9EDRRAqEjIRe3/KV+x19/bVvrc0EVYRocx1cN
+4NoDor150IbffqBZEnjUGseGVe4B8yntex1P30y4MkCmWY5hKaWxHCdz861E1MDqQIDAQABo4H9MIH6MB0GA1UdDgQWBBTO0Ypdux4LGZAbemROJtfmKZZQrzCBygYDVR0jBIHCMIG/
gBTO0Ypdux4LGZAbemROJtfmKZZQr6GBm6SBmDCBlTELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0xhbmRpc3ZpbGxlMRAwDgYDVQQKEwdEYW5vcGlhMRQwEgYDVQQLEwtHb29nbGUgV2F2ZTEUMBIGA1UEAxMLZGFub3BpYS5uZXQxHTAbBgkqhkiG9w0BCQEWDm1lQGRhbm9waWEubmV0ggkAwiP0iK
+00MYwDAYDVR0TBAUwAwEB/
zANBgkqhkiG9w0BAQUFAAOBgQCA5My3QFJE8Svsw0eQN7MZ2/
OPvvDkVfqt3GX79y23pfy0b/y7DkyCAeQ7ilsFBlkiRaY8Y7g/pWg
+w1jycN1bVbw8L2b04mYs
+lALagtmKSZaui74QzN6wbaAm2YmbDGs7UdOpOODM4+jqdAUH4ZxAIx2YOYqJF1lJyPc3e7ZaA==

(I sure hope Groups word-wraps.)

And here is the Base64 of what it hashes.
MIIDrDCCA6gwggMRoAMCAQICCQDCI/
SIr7TQxjANBgkqhkiG9w0BAQUFADCBlTELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0xhbmRpc3ZpbGxlMRAwDgYDVQQKEwdEYW5vcGlhMRQwEgYDVQQLEwtHb29nbGUgV2F2ZTEUMBIGA1UEAxMLZGFub3BpYS5uZXQxHTAbBgkqhkiG9w0BCQEWDm1lQGRhbm9waWEubmV0MB4XDTA5MTAxNzIxNDI1MVoXDTEwMTAxNzIxNDI1MVowgZUxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtMYW5kaXN2aWxsZTEQMA4GA1UEChMHRGFub3BpYTEUMBIGA1UECxMLR29vZ2xlIFdhdmUxFDASBgNVBAMTC2Rhbm9waWEubmV0MR0wGwYJKoZIhvcNAQkBFg5tZUBkYW5vcGlhLm5ldDCBnzNBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAx51hX1n5ugzVq0mqwH5JaNRIiZQ1BLopN1Phe2/
l7sOulsBi0vWueSsEnN0Ci23/x4zX/RA0UQKhIyEXt/
ylfsdff21b63NBFWEaHMdXDfuDaA6K9edCG336gWRJ41BrHhlXuAfMp7XsdT99MuDJAplmOYSmlsRwnc/
OtRNTA6kCAwEAAaOB/TCB
+jAdBgNVHQ4EFgQUztGKXbseCxmQG3pkTibX5imWUK8wgcoGA1UdIwSBwjCBv4AUztGKXbseCxmQG3pkTibX5imWUK
+hgZukgZgwgZUxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtMYW5kaXN2aWxsZTEQMA4GA1UEChMHRGFub3BpYTEUMBIGA1UECxMLR29vZ2xlIFdhdmUxFDASBgNVBAMTC2Rhbm9waWEubmV0MR0wGwYJKoZIhvcNAQkBFg5tZUBkYW5vclhLm5ldIIJAMIj9IivtNDGMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAgOTMt0BSRPEr7MNHkDezGdvzj77w5FX6rdxl
+/ctt6X8tG/
8uw5MggHkO4pbBQZZIkWmPGO4P6VoPsNY8nDdW1W8PC9m9OJmLPpQC2oLZikmWrou
+EMzesG2gJtmJmwxrO1HTqTjgzOPo6nQFB+GcQCMdmDmKiRdZScj3N3u2Wg=

Here are my attempts at encoding my .cert into what it wants:

irb(main):522:0> cert=Base64.decode64(cert)
=> …
irb(main):525:0> prehash.size
=> 942
irb(main):526:0> cert.size
=> 940
irb(main):527:0> cert2=“0\202\003\254#{cert}”
=> …
irb(main):528:0> i=0;i+=1 while cert2[i]==prehash[i];i
=> 382

irb(main):531:0> c = OpenSSL::X509::Certificate.new(File.read
(“danopia.net.cert”))
=> #<OpenSSL::X509::Certificate subject=/C=US/ST=New Jersey/
L=Landisville/O=Danopia/OU=Google Wave/CN=danopia.net/
[email protected], issuer=/C=US/ST=New Jersey/L=Landisville/
O=Danopia/OU=Google Wave/CN=danopia.net/[email protected],
serial=13989293735443484870, not_before=Sat Oct 17 21:42:51 UTC 2009,
not_after=Sun Oct 17 21:42:51 UTC 2010>
irb(main):532:0> seq=OpenSSL::ASN1::Sequence.new([c])
=> #<OpenSSL::ASN1::Sequence:0xb78fe6a4 @tag_class=:UNIVERSAL,
@tagging=nil, @tag=16, @value=
[#<OpenSSL::X509::Certificate …>]>
irb(main):539:0> seq.to_der
=> "0\202\003\2540\202\003\2500\202\003\021\240\003\002\001\002\002\t
\000\302#\364\210\257\264\320\3060\r\006\t*\206H\206\367\r
.…
irb(main):540:0> prehash
=> "0\202\003\2540\202\003\2500\202\003\021\240\003\002\001\002\002\t
\000\302#\364\210\257\264\320\3060\r\006\t*\206H\206\367\r
.…
irb(main):541:0> i=0;i+=1 while seq.to_der[i]==prehash[i];i
=> 382

So it looks like the first part is right, but about a third of the way
through, it suddenly changes.
Here’s the “good” version when it starts changing, followed by mine:
(The first two chars are the same.)

=> “\201\2373A\202J\241\222!\275\303@@@A@\000\340c@\f b@\240@1\347XW \326~n\2035j\322j\260\037\222Z5\022\"e\rA.\212M\324\370^\333\371 {\260\353\245\260\030\264\275k\236J\301'7@\242\333\177\361\3435\377D\r \024@\250H\310E\355\377)_\261\327\337\333V\372\334\320EXF\2071\325\303~ \340\332\003\242\275y\320\206\337~\240Y\022x\324\032\307\206U \356\001\363)\355{\035O\337L\2702@\246Y\216a)\245\261\034's\363\255D \324\300\352@\200\300@\000h\340\177L ~\214\aA\200\325GC \201\005\201\0053\264b\227n\307\202\306d \006\336\231\023\211\265\371\212e\224+\314 r\201\200\325GH\301 p\214 o \340\0053\264b\227n\307\202\306d\006\336\231\023\211\265\371\212e\224+ \350f\351 f\f eLB\314\002A\200\325A\001\204\300\225T\314D\314\004A
\200\325A\002\004\302\223\231]\310\022\231\\234\331^LE\f
\004\201\200\325A\001\304\302\323\030[\231\032\\335\232[\e\031LD\f
\003\201\200\325A\002\204\301\321\030[\233\334\032XLE\f
\004\201\200\325A\002\304\302\321\333\333\331\333\031H\025\330]\231LE\f
\004\201\200\325A\000\304\302\331\030[\233\334\032XK\233\231]\fGL
\006\301\202J\241\222!\275\303@B@E\203\233YP\031\030
[\233\334\226\022\346\346WH \220\f"?H\212\373M\fc\000\3005Q\3210@S \0000\020\037\363\000\320\222\250d\210op\320\020\020PP\0008\030\020\b
\016L\313t\005$O\022\276\3144y\003{1\235\2778\373\357\016E_
\252\335\306_\277r\333z_\313F\377\313\260\344\310 \036C\270\245\260Pe
\222$Zc\306;\203\372V\203\3545\217’\r\325\265[\303\302\366oN&b
\317\245\000\266\240\266b\222e\253\242\357\20437\254\eh\t\266bf
\303\032\316\324t\352N838\372:\235\001A\370g\020\b\307f\016b\242E
\326Rr=\315\336\355\226”

=> “\201\2370\r\006\t*\206H\206\367\r
\001\001\001\005\000\003\201\215\0000\201\211\002\201\201\000\307\235a_Y
\371\272\f\325\253I\252\300~Ih\324H\211\2245\004\272)7S\341{o
\345\356\303\256\226\300b\322\365\256y+\004\234\335\002\213m
\377\307\214\327\375\0204Q\002\241#!\027\267\374\245~\307_\177m[\353sA
\025a\032\034\307W\r\373\203h\016\212\365\347B\e}\372\201dI\343Pk
\036\031W\270\a\314\247\265\354u?}2\340\311\002\231f9\204\246\226\304p
\235\317\316\265\023S
\003\251\002\003\001\000\001\243\201\3750\201\3720\035\006\003U
\035\016\004\026\004\024\316\321\212]\273\036\v\031\220\ezdN&
\327\346)\226P\2570\201\312\006\003U\035#
\004\201\3020\201\277\200\024\316\321\212]\273\036\v\031\220\ezdN&
\327\346)\226P\257\241\201\233\244\201\2300\201\2251\v0\t\006\003U
\004\006\023\002US1\0230\021\006\003U\004\b\023\nNew
Jersey1\0240\022\006\003U\004\a\023\vLandisville1\0200\016\006\003U
\004\n\023\aDanopia1\0240\022\006\003U\004\v\023\vGoogle
Wave1\0240\022\006\003U\004\003\023\vdanopia.net1\0350\e\006\t*\206H
\206\367\r\001\t\001\026\[email protected]\202\t\000\302#
\364\210\257\264\320\3060\f\006\003U
\035\023\004\0050\003\001\001\3770\r\006\t*\206H\206\367\r
\001\001\005\005\000\003\201\201\000\200\344\314\267@RD\361+\354\303G
\2207\263\031\333\363\217\276\360\344U\372\255\334e\373\367-
\267\245\374\264o\374\273\016L\202\001\344;\212[\005\006Y"E\246<c\270?
\245h>\303X\362p\335[U\274</f\364\342f,\372P\vj\vf)&Z\272.\370C3z \301\266\200\233f&l1\254\355GN \244\343\2033\217\243\251\320\024\037\206q\000\214v`\346*$]e’#
\334\335\356\331h”

Daniel_Danopia · October 22, 2009, 9:48am

Daniel Danopia wrote:

Here’s the Base64 of my .cert file (I have a .key and .cert):

Hmm, a PEM would have been much safer. Anyway I’ve attempted to
reconstruct the base64. When I do and then decode it to binary I get 940
bytes, with SHA1
cf0d16a0dde94fec3814d21a7c6daed6c17fd9cd. It does seem to parse:

I presume “prehash” is what the Java had before hashing, and if I’ve
decoded your base64 correctly, that has 942 bytes, with SHA1
50251d690f2a9e15e029080bc3324ba16dd40ce6

So I would attempt to build pkipath like this:

c = OpenSSL::X509::Certificate.new(File.read(“tst.cert”))
=> #<OpenSSL::X509::Certificate subject=/C=US/ST=New
Jersey/L=Landisville/O=Danopia/OU=Google
Wave/CN=danopia.net/[email protected], issuer=/C=US/ST=New
Jersey/L=Landisville/O=Danopia/OU=Google
Wave/CN=danopia.net/[email protected],
serial=13989293735443484870, not_before=Sat Oct 17 21:42:51 UTC 2009,
not_after=Sun Oct 17 21:42:51 UTC 2010>
pkipath = OpenSSL::ASN1::Sequence.new([c]).to_der

This gives me 944 bytes.

Now I can suggest a couple of tools to debug this. A low-level one is
hexdump -C tst.cert >tst.cert.hd
which shows what you found before: differences starting at pos 0x17e
(382)

$ diff -u prehash.hd cert2.hd
— prehash.hd 2009-10-22 08:28:29.000000000 +0100
+++ cert2.hd 2009-10-22 08:28:34.000000000 +0100
@@ -21,40 +21,40 @@
00000140 65 20 57 61 76 65 31 14 30 12 06 03 55 04 03 13 |e
Wave1.0…U…|
00000150 0b 64 61 6e 6f 70 69 61 2e 6e 65 74 31 1d 30 1b
|.danopia.net1.0.|
00000160 06 09 2a 86 48 86 f7 0d 01 09 01 16 0e 6d 65 40
|….H…me@|
-00000170 64 61 6e 6f 70 69 61 2e 6e 65 74 30 81 9f 33 41
|danopia.net0…3A|
-00000180 82 4a a1 92 21 bd c3 40 40 40 41 40 00 e0 63 40
|.J…!..@@@A@…c@|
…
+00000170 64 61 6e 6f 70 69 61 2e 6e 65 74 30 81 9f 30 0d
|danopia.net0…0.|
+00000180 06 09 2a 86 48 86 f7 0d 01 01 01 05 00 03 81 8d
|….H…|

Notice also that in cert2.hd at offset 0270…0300 you can see the
details of the signing certificate, whilst this appears to be garbage in
prehash.hd

A more useful tool is:
openssl asn1parse -inform der -in tst.cert >tst.cert.txt

This decodes your original certificate and my cert2, but it fails
entirely on the ‘prehash’:

$ openssl asn1parse -inform der -in prehash >prehash.txt
10767:error:0D07209B:asn1 encoding routines:ASN1_get_object:too
long:asn1_lib.c:142:

So whatever is in your ‘prehash’, it’s not valid ASN.1

This suggests to me that perhaps the file was corrupted on its way out
from Java (a text/binary problem??) Perhaps you could print the prehash
size from within the Java?

Also, I’ve just remembered that the final output of the Java is some
sort of hash of this string. Can you replicate this in Ruby to confirm
that the extracted prehash is accurate? Can you post the hash and the
algorithm used, just for sanity checking?

Regards,

Brian.

Daniel_Danopia · October 22, 2009, 10:49am

I think I can even tell you which bytes are missing in your base64

If I start with your original certificate, convert it to a Sequence,
base64 encode it and then output one character per line:

require ‘openssl’
cert = OpenSSL::X509::Certificate.new(File.read(“tst.cert”))
cert2 = OpenSSL::ASN1::Sequence.new([cert])
b64 = [cert2.to_der].pack(“m”).gsub(/\s/,’’)
File.open(“cert2.b64l”,“wb”) { |f| b64.each_byte { |c| f.puts c.chr } }

and compare this to your posted prehash base64, also split into one
character per line:

— prehash.b64l 2009-10-22 09:41:50.000000000 +0100
+++ cert2.b64l 2009-10-22 09:39:09.000000000 +0100
@@ -508,6 +508,7 @@
B
n
z
+A
N
B
g
@@ -1018,6 +1019,7 @@
5
v
c
+G
l
h
L

So there are the two missing characters from prehash base64, and their
positions - put them back and it’ll all match exactly.

The moral of the story: when dealing with large amounts of base64, break
it into short fixed-length lines. RFC 2045 says “The encoded output
stream must be represented in lines of no more than 76 characters each”

Cheers,

Brian.

Daniel_Danopia · October 22, 2009, 10:22am

Daniel Danopia wrote:

And here is the Base64 of what it hashes.
…

Almost certainly this is corrupt. If I paste all those characters into a
file and remove all the newlines, I get 1258 characters. However, a
base64 encoding should always be a multiple of 4 characters (see RFC
4648 section 4, inc the notes on final padding)

This explains why the data in the DER appears to switch to garbage.