Gateway is malfunctioning

Ruby
1

Someone recently mentioned missing posts. I just did a comparsion of
comp.lang.ruby and ruby-talk-google and indeed the usenet is missing
posts.

James you want to have a look at the gateway?

T.

2

Trans wrote:

Someone recently mentioned missing posts. I just did a comparsion of
comp.lang.ruby and ruby-talk-google and indeed the usenet is missing
posts.

James you want to have a look at the gateway?

T.

I think someone mentioned earlier that the missing posts were mostly
those with HTML in them. Why the HTML posts are filtered out I do not
know.

3

Michael W. Ryder wrote:

those with HTML in them. Why the HTML posts are filtered out I do not
know.
Not true. I had some missing posts (a couple misdated and a couple not)
and I only send text.
Cheers,
V.-

4

On Fri, 1 Dec 2006, Trans wrote:

Someone recently mentioned missing posts. I just did a comparsion of
comp.lang.ruby and ruby-talk-google and indeed the usenet is missing posts.

James you want to have a look at the gateway?

as far as i can tell it always is - or at least has been for weeks.

can we write a script to compare on a regular basis? a monitor?

are you in charge of this james?

cheers.

-a

5

On Nov 30, 2006, at 9:22 PM, Trans wrote:

Someone recently mentioned missing posts. I just did a comparsion of
comp.lang.ruby and ruby-talk-google and indeed the usenet is missing
posts.

Here’s the information I have on this currently:

  • I rewrote the entire Gateway this last week, so anything before
    that is history. The news-to-email side is already deployed. My
    email-to-news code is complete, but not deployed. (Our SysAdmin will
    get to it soon now.) If you raise issues, please make sure they are
    recent.
  • I now have very detailed logs on everything the Gateway does, so
    please be specific. For example, tell me which messages are missing,
    not just that they are.
  • Our Usenet host does not allow us to post HTML emails. This is not
    changing. (I’m not bothered by this because I do not believe anyone
    should be sending HTML email to Ruby T…)
  • Our Gateway is an NNTP <-> email Gateway. There has been at least
    one instance of a Usenet post using ancient header formatting
    predating NNTP. These messages are not supported.

James you want to have a look at the gateway?

I will address individual issues, as long as they are aware of all
four points above.

James Edward G. II

6

On Dec 1, 2006, at 4:06 AM, Damphyr wrote:

Not true. I had some missing posts (a couple misdated and a couple
not) and I only send text.

Your posts appeared on one side of the Gateway (the Ruby T. mailing
list or the Usenet group comp.lang.ruby), but did not reach the other
side? If so, please lead me to the posts on the side where they did
appear.

James Edward G. II

7

On Sat, 2 Dec 2006, James Edward G. II wrote:

  • I rewrote the entire Gateway this last week, so anything before that is
    history. The news-to-email side is already deployed. My email-to-news code
    is complete, but not deployed. (Our SysAdmin will get to it soon now.) If
    you raise issues, please make sure they are recent.

mine aren’t - so ignore for now.

  • I now have very detailed logs on everything the Gateway does, so please be
    specific. For example, tell me which messages are missing, not just that
    they are.

can’t we automate this? monitor/scraper?

  • Our Usenet host does not allow us to post HTML emails. This is not
    changing. (I’m not bothered by this because I do not believe anyone should
    be sending HTML email to Ruby T…)

harp:~ > echo ‘html_2_text?’ |
links -dump -stdin
html_2_text?

??

  • Our Gateway is an NNTP <-> email Gateway. There has been at least one
    instance of a Usenet post using ancient header formatting predating NNTP.
    These messages are not supported.

check.

I will address individual issues, as long as they are aware of all four
points above.

so… i have this rash… :wink:

-a

8

On Dec 1, 2006, at 8:29 AM, [email protected] wrote:

are you in charge of this james?

I rewrote and now run the Ruby Gateway, yes.

James Edward G. II

9

On Dec 1, 2006, at 9:39 AM, [email protected] wrote:

  • I now have very detailed logs on everything the Gateway does, so
    please be specific. For example, tell me which messages are
    missing, not just that they are.

can’t we automate this? monitor/scraper?

Probably not too meaningfully. Every Usenet host uses a different
system for numbering posts, so the numbers wouldn’t be much help to
the outside world.

Besides, now that I understand the Gateway code intimately, I’m
hoping to get it robust enough to make problems rare.

  • Our Usenet host does not allow us to post HTML emails. This is
    not changing. (I’m not bothered by this because I do not believe
    anyone should be sending HTML email to Ruby T…)

harp:~ > echo ‘html_2_text?’
| links -dump -stdin
html_2_text?

??

I would much rather Ruby T. rejected HTML emails. :wink:

I think they are pure evil, so it will be hard to convince me to
spend much effort on them.

I will address individual issues, as long as they are aware of all
four points above.

so… i have this rash… :wink:

OK… Gateway issues!

James Edward G. II

10

James Edward G. II wrote:

email-to-news code is complete, but not deployed. (Our SysAdmin will
predating NNTP. These messages are not supported.
Have a look at recent topic: Wisdom of including Rakefile in releases.
The 2nd post from Jason R. appears to be missing.

T.

11

James G. wrote:

I would much rather Ruby T. rejected HTML emails. :wink:

+1

12

On 01.12.2006 17:05, Trans wrote:

email-to-news code is complete, but not deployed. (Our SysAdmin will
predating NNTP. These messages are not supported.

Have a look at recent topic: Wisdom of including Rakefile in releases.
The 2nd post from Jason R. appears to be missing.

I do not see any posting of Jason on the news side and only a single
posting on the email side:

Rakefiles allow users to run the tests for themselves, to understand how a library is put together, etc. The only thing that can go wrong is that the stuff doesn't work (unless you're grabbing very untrustworthy packages, but assumptions have to be made somewhere). Not to mention, having the Rakefile there allows other people to properly make changes to a library if / when they need to.

I don’t see anything bad with including said Rakefiles. Do you have any
specific worries?

Cheers

robert
13

On 01.12.2006 18:30, James Edward G. II wrote:

the Gateway logs show it was forwarded.
[email protected] - Re: Wisdom of including Rakefile in releases
11/30/06-15:21:43: mail2news | nntp.post() result = 441 437 HTML post

Translation: Our NNTP host rejected the HTML post.

I cannot see that it is a HTML posting - in fact, I do not see any HTML
in that posting. Is it possible that the gateway rejects multipart
messages? James, if you need the source (or what reached my gmail box)
let me know and I forward it to you.

Kind regards

robert
14

On Dec 1, 2006, at 4:20 PM, Robert K. wrote:

post
Translation: Our NNTP host rejected the HTML post.

I cannot see that it is a HTML posting - in fact, I do not see any
HTML in that posting. Is it possible that the gateway rejects
multipart messages? James, if you need the source (or what reached
my gmail box) let me know and I forward it to you.

Fred, can you comment on this? The link above shows pretty much how
the rejected message was formatted.

James Edward G. II

15

Le 1 décembre 2006 à 23:30, James Edward G. II a écrit :

On Dec 1, 2006, at 4:20 PM, Robert K. wrote:

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/227459

I cannot see that it is a HTML posting - in fact, I do not see any
HTML in that posting. Is it possible that the gateway rejects
multipart messages? James, if you need the source (or what reached
my gmail box) let me know and I forward it to you.

I’d like to see the whole raw message, headers included (my email is
valid, please attach the whole message, (g)zipped, or it will be bounced
by my personal filters).

Fred, can you comment on this? The link above shows pretty much how
the rejected message was formatted.

Are you sure the mailing-list archive software doesn’t filter out HTML ?
A multipart message with only one part sounds quite unlikely to me.

James, maybe the gateway should save the rejected messages somewhere, if
only temporarily, don’t you think ?

Anyway, I’m using cleanfeed on the server, an omnipresent filtering
package. It’s usually quire reliable, even if it has to use some
complicated heuristics from time to time (since MIME isn’t widely used
on Usenet). It’s a quite standard version, with a bit of tweaking, but
not in that domain.

Why am I telling you all of this, you ask ? Well, even if it is a false
positive, if I tune my filters to let the message pass, it will anyway
have a really bad distribution, being rejected by most of the hosts that
run the same soft.

James Edward G. II

Fred

16

Le 2 décembre 2006 à 11:16, F. Senault a écrit :

I’d like to see the whole raw message, headers included (my email is
valid, please attach the whole message, (g)zipped, or it will be bounced
by my personal filters).

Ok, Robert sent me what seemed to be the full message, and it’s in fact
a message declared with a MIME content-type of multipart/alternative but
only one part (text/plain). This is quite absurd, and I’d suspect that
something, somewhere, decided to strip away the HTML part but leave the
MIME declaration intact.

And, of course, cleanfeed declared the message as HTML on the basis of
the headers…

Fred
(Not sure of my English on this one…)

17

On Dec 2, 2006, at 5:50 AM, F. Senault wrote:

multipart messages? James, if you need the source (or what reached
alternative but
only one part (text/plain). This is quite absurd, and I’d suspect
that
something, somewhere, decided to strip away the HTML part but leave
the
MIME declaration intact.

Forgive my weak knowledge of email types here, but if someone sent a
short message to the list with an attached Ruby script, would it be a
multipart message? If so, that would be rejected by the NNTP host?

James

18

maybe.

19

On Sun, 2006-12-03 at 01:10 +0900, James Edward G. II wrote:

I cannot see that it is a HTML posting - in fact, I do not see any
fact
multipart message? If so, that would be rejected by the NNTP host?
I have a feeling this is what is, or at least was, happening - I wasn’t
going to say anything (since it was prior to December first) but I
noticed that posts with attachments never made it across.

ruby-talk 226884, for example, doesn’t appear to have made it to the
newsgroup.

20

On Dec 1, 2006, at 10:35 AM, Robert K. wrote:

On 01.12.2006 17:05, Trans wrote:

Have a look at recent topic: Wisdom of including Rakefile in
releases.
The 2nd post from Jason R. appears to be missing.

I have not yet received this message from Trans. I do find it in the
mailing list archives though and I see it on comp.lang.ruby. I’ll
assume my host blocked it, ruby talk is hiccuping, or something
because the Gateway logs show it was forwarded.

I do not see any posting of Jason on the news side and only a
single posting on the email side:

Right. I’ll assume we are talking about this message:

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/227459

The log says:

[email protected] - Re: Wisdom of including Rakefile in releases
11/30/06-15:21:43: mail2news | nntp.post() result = 441 437 HTML post

Translation: Our NNTP host rejected the HTML post.

James Edward G. II