403 errors with standard browser, pages work with GUI browser

Hi,

I’ve run across several websites where I get a 403 error when
I connect to them with my standard webbrowser (lynx), yet the
very same URIs work with the GUI browser fallback (Firef*x).

Example link:

http://blog.streamingmedia.com/2015/07/new-patent-pool-wants-share-of-revenue-from-content-owners.html

Output from webbrowser:

←←←
403 Forbidden
403 Forbidden
______________________________________________________________________________________________

                                              nginx

All of these sites is common that they have an nginx error
message, so this must be connected to your software somehow.

To add insult to injury, this appears to be dependent on
the User-Agent HTTP header: “Mozilla/5.0 (compatible)”
works, “Lynx/2.8.9dev.6” works, “Lynx/2.8.9dev.6 libwww-FM/2.14”
doesn’t, nor “Mozilla/5.0 (compatible) libwww-FM/2.14”.

It appears that there is some blacklist on some component
of my webbrowser. Please kindly remove it.

Thanks,
//mirabilos

Why don’t you use JavaScript? I also don’t like enabling JavaScript in
Because I use lynx as browser.
+1
– Octavio Alvarez, me and ⡍⠁⠗⠊⠕ (Mario Lang) on debian-devel

On Wed, Jul 29, 2015 at 05:00:40PM +0200, Thorsten Glaser wrote:

Hi there,

I’ve run across several websites where I get a 403 error when
I connect to them with my standard webbrowser (lynx), yet the
very same URIs work with the GUI browser fallback (Firef*x).

All of these sites is common that they have an nginx error
message, so this must be connected to your software somehow.

That suggests that they run nginx as their web server.

There is unlikely to be any other connection.

To add insult to injury, this appears to be dependent on
the User-Agent HTTP header: “Mozilla/5.0 (compatible)”
works, “Lynx/2.8.9dev.6” works, “Lynx/2.8.9dev.6 libwww-FM/2.14”
doesn’t, nor “Mozilla/5.0 (compatible) libwww-FM/2.14”.

It appears that there is some blacklist on some component
of my webbrowser. Please kindly remove it.

I suspect that they have configured their instance of their web server
to deny access to user agents which include the string “libwww”.

I suspect that your best option for accessing the content is to change
your web browser user agent string to omit “libwww” (and whatever other
parts the server owner has chosen to block); at least until you contact
the owner and convince them to change their configuration.

Good luck with it,

f

Francis D. [email protected]