IO and an encoded email attachment

Ruby
1

Hi all,

I have been trying to set up a small Ruby script as a spam filter. It
would be used as a content_filter from Postfix, and then pass the
message to SpamAssassin. I didn’t use SA as a direct content filter
because I wanted the flexibility to be able to do things like pick and
choose which email addresses get filtered.

Anyway, I set this up and it worked fine. Then towards the end of last
week people were having problems sending emails with attachments to us.
Postfix was bouncing them complaining that the content_filter was taking
too long to run.

I have spent a long time attempting to debug the issue. It basically
appears to come down to Ruby treating the base64 encoded attachment
really weirdly when doing IO stuff.

I have this small script (you can ignore the gsub it doesn’t make a
difference):

text = $stdin.read.gsub(/([`$\“])/, ‘\\\1’)
command = ‘echo "’ + text + '”’
output = `#{command}`

Then I run:

$ cat message.txt | ruby test.rb

Theoretically it shouldn’t output anything. But it does. It turns out
the output is STDERR, so:

$ cat message.txt | ruby test.rb 2>&1 | head
test.rb:3: command not found: echo
"691737r3Xvfuvde9+691737r3UWs/wCA7/7D/ex7917oLdwfof8A2P8AvQ9+690Hu2117urU+urY
O71t/wAGyG0x/wAT7chNJoj/AEh/h6bmzDKP6J/wdOX8P/2j/eP+NexB4vz6IfD+XVeHzY7Z7A6y
7P8AihtDau+98bC2t2pvLfeG3/V9bdd4XsvfFbjMLtzHZHGJt3buR627TyVRWQVspLLRYmokaJ3L
qVXUhZb3Tz81W+1y3Eq2B2y7mIjTUxkiXVGcI7BdVA/BQhZmKga1MHt0j5c3DcEhja8TcLKJS7aV
EczSrKMuiliqgpUli4VVDFtLAl1n88s5sD48025u68hjN7dj5X5Mbt6F2mN3122emMhQ0lJmZ3x2
Z+QcUOGjoOnJdv4WKaTIKMK1THTilY0jNNJP7UQbg9xY8jI628e77jtrTTnxaWyPbuFuNMgM9X0v
EREjSapX0KyghVansBBuHOL1mO02UyeAPCLTypPCHtgkZEZZZisjCZ/DQKsvnGqupW/mW42XalHk
sL1Rht47qX5N4P425PF7J7fxe4Nk1tTuakyNdt7fOxOyRs6kx+7sHl6WhHihqKLEujsyyvGFDM7Y
zy7lPy1Bawsr7ibyM+LWPwZrR4lZWoHLxMs8cnjR6qqWMSTDSXbubeOzi5klnmVk26zt7r9Pv8SO
4Vjp7imiZGjkR0NVDKKuKkK+1/8AMYwG3di7gyW8eu8RtjsbB/JnKfF1tsV3Z9HT9dR7rw4gqchv

$ echo “hi” | ruby test.rb
#-> No output

So I think… right, something in that file has to be triggering the
problem. I cut the file in half to keep narrowing down the problem. I
take the top off and run it… no output. I cut the bottom off and run
it… no output. But with top and bottom… I get this weird STDERR
output.

Also, I’d note that I’ve tried other methods… Kernel#system works
okay, but I need the output. %x[] doesn’t (that’s Kernel#` anyway).
IO.popen doesn’t work. IO.popen then writing to the process doesn’t work
(which is what I was doing originally). Stranger still, if I write to a
temporary file it doesn’t all write correctly. It’s stops writing at a
random point in the string. Is it just me or does this behaviour totally
not make any sense?

I would be really really seriously grateful if anyone can help me out :slight_smile:

Thanks very much,
Jon

The files:
http://clients.dunwoodydev.co.uk/jon/message.txt
http://clients.dunwoodydev.co.uk/jon/test.rb

2

Jon L. wrote:

Hi all,

I have been trying to set up a small Ruby script as a spam filter. It
would be used as a content_filter from Postfix, and then pass the
message to SpamAssassin. I didn’t use SA as a direct content filter
because I wanted the flexibility to be able to do things like pick and
choose which email addresses get filtered.

If you use procmail for your MDA you can set up recipes so that you can
pass incoming message that match these rules to spamd. This is a much
faster solution than implementing in Ruby. Just something to consider.

3

Cliff C. wrote:

Jon L. wrote:

Hi all,

I have been trying to set up a small Ruby script as a spam filter. It
would be used as a content_filter from Postfix, and then pass the
message to SpamAssassin. I didn’t use SA as a direct content filter
because I wanted the flexibility to be able to do things like pick and
choose which email addresses get filtered.

If you use procmail for your MDA you can set up recipes so that you can
pass incoming message that match these rules to spamd. This is a much
faster solution than implementing in Ruby. Just something to consider.

Thanks for the suggestion. I considered procmail but didn’t really want
to use it because I already had set up a working system using Postfix
and virtual mailboxes. I’m sure that’s possible in procmail too but just
adds another technology to the chain and something else to learn.
However, if we can’t find a reasonable solution to this problem I may
reconsider it.

4

On Thu, 31 Aug 2006 11:26:37 -0400, Jon L.
[email protected]
wrote:

appears to come down to Ruby treating the base64 encoded attachment
really weirdly when doing IO stuff.
I have this small script (you can ignore the gsub it doesn’t make a
difference):
text = $stdin.read.gsub(/([`$\“])/, ‘\\\1’)
command = ‘echo "’ + text + '”’
output = `#{command}`

This is probably because the echo command is a command that is built
into
the shell. Try finding an echo program, like this:
which echo
Then use that with the full path in the command= line.
It worked for me after I did that.

Regards,
JJ

5

John J. wrote:

On Thu, 31 Aug 2006 11:26:37 -0400, Jon L.
[email protected]
wrote:

appears to come down to Ruby treating the base64 encoded attachment
really weirdly when doing IO stuff.
I have this small script (you can ignore the gsub it doesn’t make a
difference):
text = $stdin.read.gsub(/([`$\“])/, ‘\\\1’)
command = ‘echo "’ + text + '”’
output = `#{command}`

This is probably because the echo command is a command that is built
into
the shell. Try finding an echo program, like this:
which echo
Then use that with the full path in the command= line.
It worked for me after I did that.

This didn’t work for me. Does this sound like a bug in Ruby to anyone
else? Or perhaps the shell? Aside from finding a different solution
(which I am considering), how should I proceed? I have no idea really
how I can debug this…

6

On Fri, 1 Sep 2006, Jon L. wrote:

Hi all,

I have been trying to set up a small Ruby script as a spam filter. It
would be used as a content_filter from Postfix, and then pass the
message to SpamAssassin. I didn’t use SA as a direct content filter
because I wanted the flexibility to be able to do things like pick and
choose which email addresses get filtered.

You might take a look at gurgitate-mail:

http://www.dagbrown.com/software/gurgitate-mail/

It makes doing complex things with email, using a Ruby DSL, easy. I
have
been using it for…I don’t even know for sure. I think for close to as
long as it has existed. 3 years?

I use it for a variety of things, and like it a lot because of the ease
and flexibility.

Kirk H.

7

Hello all,

Apologies for bringing this thread up from it’s grave (last post was 12 years ago).

I am doing a study on email messages (encoding, headers, MTA etc) and landed on this page with similar subject. I do understand basics like what are the different encoding applicable to email (considering 7 bit nature of SMTP) etc. However I have hard time understanding how to decode an email message without headers.

OP has mentioned a message file on which he wants to apply filters (in Postfix). This particular message file he is using is not having headers. If you base64 decode it, it doesn’t decode to anything that gives indication of type of file. I don’t understand if it is a full message file that can be opened by an MUA or attachment or anything else. Obviously, I am not interested in this particular message, but I want to understand the structure of data in such emails. Is there a way to identify/decode such a message file?

I apologise for posting a question not relating to ruby directly. But after spending a LOT of time trying to dig into this subject, this is the only post that has a matching context.

Thanks in advance