UTF-8, AES encryption, the works - help!

Rails
1

Hi,

I would like to store some sensitive data into my DB using Rails, it has
to be encrypted. I’ve tried a lot of methods of encryption, they all
work. Fine by now, yet when I try to store the results into my DB
(tables are UTF-8, SET NAMES UTF8; is there, server UTF-8, DB Connection
UTF-8) – I can cmd (yeah, Windows…) and enter a UTF-8 record into the
table, so that’s not where the problem is.

So after I use the encryption method (AES, Blowfish yada - all returning
UTF-8, they all work - encrypt then decrypt, strings match), I would
like to store the result in the DB -> Just using an insert with the
encrypted string fails, as it inserts only regular, Latin1-chars, the
other characters are being discarded. I have googled and tried all
options,

ic = Iconv.new(‘UTF-8//IGNORE’, ‘UTF-8’)
valid_string = ic.iconv(enc + ’ ')[0…-2]

raises InvalidCharacter or InvalidSequence on some strings so not
viable… Should I convert the result of AES and so on into Latin1 ?? I’m
quite lost as you can see, spent too much on it already.

So am I beating around the bush too much (wasted a lot of time on this
already), or should I search for a non-Unicode encryption solution
(Salted base64 ?) ? Message digests like SHA1 MD5 and so on are not an
option, need to be able to recreate the message (username or email, for
instance).

Thanks a whole lot ! I can provide extra details if required, no
problem.

2

On Jun 2, 3:46 am, Abe C. [email protected] wrote:

So after I use the encryption method (AES, Blowfish yada - all returning
UTF-8, they all work - encrypt then decrypt, strings match), I would
like to store the result in the DB → Just using an insert with the
encrypted string fails, as it inserts only regular, Latin1-chars, the
other characters are being discarded. I have googled and tried all
options,

I doubt that the encrypted data is legal utf8 - half the point of
encryption is that it’s going to look pretty damn close to random
garbage, and a lot of databases will ignore or truncate invalid utf8
if you try to insert it into a column with charset utf8. Given that
you have binary data you should be storing it in an appropriate column
type (eg blob). If you really didn’t want to do that you could base64
the result of the encryption.

Fred

3

Frederick C. wrote:

I doubt that the encrypted data is legal utf8 - half the point of
encryption is that it’s going to look pretty damn close to random
garbage, and a lot of databases will ignore or truncate invalid utf8
if you try to insert it into a column with charset utf8. Given that
you have binary data you should be storing it in an appropriate column
type (eg blob). If you really didn’t want to do that you could base64
the result of the encryption.

Fred

Works like a charm Fred, I was sure that AES would make them UTF-8
compatible (what Advanced Encryption Standard would it be then !? :slight_smile: )
and not encrypted junk ! Base64’d the AES encrypted string, for decoding
DeBase64, AES decrypt.

Thank you a mil !

4

On Jun 2, 1:06 pm, Abe C. [email protected] wrote:

Frederick C. wrote:

Works like a charm Fred, I was sure that AES would make them UTF-8
compatible (what Advanced Encryption Standard would it be then !? :slight_smile: )
and not encrypted junk !

AES is a general purpose encryption algorithm - it knows nothing about
UTF8 (or any other text encoding for that matter).

Fred