Squid questions and solutions blog (optimising web delivery)

Howto debug network errors in squid with wireshark

noreply@blogger.com (Unknown) — Thu, 21 Jun 2007 06:43:00 +0000

If you see in the squid logs errors such as:


> 2007/06/12 11:15:45| parseHttpRequest: Unsupported method '^C'
> 2007/06/12 11:15:45| clientReadRequest: FD 145 (x.x.x.x:62332) Invalid
> Request
> 2007/06/12 11:15:48| parseHttpRequest: Requestheader contains NULL
> characters
> 2007/06/12 11:15:48| parseHttpRequest: Unsupported method '^C'
> 2007/06/12 11:15:48| clientReadRequest: FD 1611 (x.x.x.x:60853) Invalid
> Request
> 2007/06/12 11:15:49| parseHttpRequest: Requestheader contains NULL
> characters
> 2007/06/12 11:15:49| parseHttpRequest: Unsupported method '^C'

You could do some debug with utils such as ethereal/wireshark:

PROCEDURE:

1. Start wireshart, and start a new packet capture.

2. When seeing the error in cache.log, enter the filter
ip.address == YYYYYY && tcp.port == XXXXX

where YYYYY is the IP and XXXXX is the port number from the Invalid
Request log line.

Then select the first packet shown (should be a SYN), and choose Analyze
-> Follow TCP Stream. This opens a new window with the TCP stream
decoded. In this area you'll have all important data about the problem.

Differences between an HTTP/FTP/SSL proxy and a SOCKS proxy

noreply@blogger.com (Unknown) — Wed, 30 May 2007 06:26:00 +0000

Squid is a HTTP proxy. All communication to Squid is HTTP. But HTTP proxies can resolve a number of different URL-schemes for their HTTP clients.

The difference between an HTTP proxy and SOCKS is that the HTTP proxy is
fully aware of the protocol being proxied, and as result have well defined behavior. The main difference is the ability to cache results.

SOCKS on the other hand is protocol neutral and SHOULD NOT perform caching or other heavily protocol dependent actions.

With a web browser you can support both kind of proxys in order to surf the web.

Transparent SSL proxy

noreply@blogger.com (Unknown) — Mon, 02 Apr 2007 08:00:00 +0000

Does squid support this feature actually?. Yet NO (squid-2.6-Stable12)

Several people ask on the squid mailing list they are working with a transparent proxy but they need transparently "proxy" the 443 port "HTTPS" they mean that their transparent proxy could work with https urls.

But there are some misconceptions between HTTPS/SSL and proxy/reverse proxys.

Brief description:

Working as a normal proxy, squid can tunnel SSL requests when are requested by a HTTP user-agent (Netscape Documentation) vía HTTP proxies.
This involved a HTTP method (CONNECT) for establishing the tunnel.

But in a interception proxy know as transparent proxy as well, the proxy becomes the server for the client and becomes the client for the web server. The connection between the two parts who starts the connection is broken and the identity of each is hidden (SSL), so in this special case the transparent proxy doesn't know how to handle the SSL requests because is not operating as a normal proxy.

Some ideas to implement:

Listen on a different port than the current port used for the transparent proxy (usually 80)
Accept the SSL connection.
Do the acl lookups sourc/destination IP, source MAC, , time srcdomain...
Convert it to a HTTP CONNECT request suitable for the http proxy.

Another small project to work in with squid.

A fun project with squid (soc?)

noreply@blogger.com (Unknown) — Wed, 14 Mar 2007 15:18:00 +0000

Right now, google is starting the soc, in order to get more open source projects and helping to the software students into participating in open source development.

Some ideas to help with squid:

In the squid list we are seeing several sites which are not working with squid due to a broken sites, this sites could not work due to a multiple factors, a fun project should be to make a software in order to test this broken sites.

Some issues with these sites are:

ECN
Windows Scaling
Forgetting Vary
Mixing up ETag (same ETag on multiple incompatible entities)
Various malformed responses

Double content lenght
Malformed headers
Repeated single-value headers

And you can help a lot of with squid too, only take a look to bugzilla.

Any more squid project related?

Squid accelerator tips to serving content when a backend server is down

noreply@blogger.com (Unknown) — Wed, 07 Mar 2007 08:02:00 +0000

We can have squid to serve stale objects although the backend server is down.

Make sure negative_ttl is set to 0 seconds to disable the caching of errors.
If the web server become unreachable, set the connect timeout sufficiently short. There are three different connect timeouts depending on your config and requirements:

- config_timeout: for requests going DIRECT
- peer_config_timeout: for requests going to a cache_peer
- cache_peer ... timeout=XXX: specific timeout for this cache_peer, this one override peer_config timeout.

Default values are 1 minute for requests going direct and 30 seconds for requests sent to a cache_peer.

As a general statement in accelerator setups, you want the backend connect timeout quite short, a few seconds.

New look in the squid web.

noreply@blogger.com (Unknown) — Tue, 27 Feb 2007 16:18:00 +0000

Adrian Chadd is an active squid developer who has helped with the new look on the web page.

The squid team is remarking the new "How to Help out" section in order to catch more people helping to improve the squid software.

It seems in a near future We'll have a new merchandise section where the people from the list could buy squid related things like t-shirts, cup of tea or similar in order to help the project.

Personally I feel the squid list is a good source of information so We all would have to feel more close to the squid and to try help to the squid team by some way . Because YOU can help in a differentd ways.

SSL support, Squid 2.6 branch and RedHat

noreply@blogger.com (Unknown) — Thu, 15 Feb 2007 09:40:00 +0000

Testing the last squid version (squid-2.6Stable9) with ssl support in order to operate as a reverse proxy, I get this errors in the compilation process:

---cut---
ssl_support.h:49: syntax error before '*' token
ssl_support.h:49: warning: type defaults to `int' in declaration of `sslCreateServerContext'
ssl_support.h:49: warning: data definition has no type or storage class
ssl_support.h:50: syntax error before '*' token
ssl_support.h:50: warning: type defaults to `int' in declaration of `sslCreateClientContext'
ssl_support.h:50: warning: data definition has no type or storage class
ssl_support.h:54: syntax error before "SSL"
ssl_support.h:56: syntax error before '*' token
ssl_support.h:57: syntax error before '*' token
ssl_support.h:58: syntax error before '*' token
ssl_support.h:59: syntax error before '*' token
ssl_support.h:60: syntax error before '*' token
---cut--

What's happening?

The problem here is with RedHat, They have built OpenSSL with Kerberos support

#[root@proxy squid-2.6.STABLE9]# rpm -qR openssl-devel-0.9.7a-33.12
krb5-devel

but Kerberos is not in the standard library and include path. This makes it impossible to build OpenSSL applications without manually including /usr/kerberos in the include and library paths.

More info.

Reverse proxy configurations

noreply@blogger.com (Unknown) — Tue, 06 Feb 2007 10:14:00 +0000

The configuration of a reverse proxy, it depends on what functionality you want to achive.
There is three ways of using this depending on what your functionality
requirements are:

a) With Squid acting as an accelerator/reverse proxy for a defined list
of sites, upgrading these sites to https. You then use the ssl option to
cache_peer to wrap the request in SSL.

b) By using a HTTP client sending https:// URLs to Squid. Squid will
then maintain the SSL on behalf of the client.

Here, the client has to send the https:// request using HTTP to the
proxy, just as it does for http://. This is:

GET https://www.example.com/path/to/file HTTP/1.1
[headers]

It does not work for clients using the CONNECT method asking for a SSL
tunnel over the proxy.

At this case, the clients are knowing they should not run the SSL themselves and
delegating this task to the proxy. They don't have any SSL capabilities and instead
rely on the proxy to perform the SSL encryption.

c) Using a url rewriter helper to rewrite selected http:// URLs into
https:// per your own specifications, making Squid process the request
as a https:// request even if the client requested http://

At this case, the clients are emulating this by rewriting http:// URLs into https://
at the proxy.

It's also possible to extend Squid with the capability to decrypt
CONNECT SSL proxy requests allowing inspection of https traffic.
For more information on this way you can contact with Henrik Nordstrom.
Contactos de squid.

Squid running out of free ports.

noreply@blogger.com (Unknown) — Mon, 05 Feb 2007 10:09:00 +0000

Symptons from a busy squid with high traffic:

commBind: Cannot bind socket FD 98 to *:0: (98) Address already in use

Solution:

You have run out of free ports, all available ports occupied by
TIME_WAIT sockets.

Things to look into

1. Make sure you internally use persistent connections between Squid and
the web servers. This cuts down on the number of initiated connections/s
considerably.

2. Configure the unassigned port range as big as possible in your OS. On
Linux this is set in /proc/sys/net/ipv4/ip_local_port_range. The biggest
possible range is 1024-65535 and can sustain up to at least 500
connections/s continuous load squid->webservers.

What does Squid do or act like when its out of file descriptors?

noreply@blogger.com (Unknown) — Mon, 05 Feb 2007 09:59:00 +0000

When Squid sees it's short of filedescriptors it stops accepting new
requests, focusing on finishing what it has already accepted.

And long before there is a shortage it disables the use of persistent
connections to limit the pressure on concurrent filedescriptors.

What does it to do in such case?

Once Squid has detected a filedescriptor limitation it won't go
above the number of filedescriptor it used at that time, and you need to
restart Squid to recover after fixing the cause to the system wide
filedescriptor shortage.

do squid recover or do it need to be restarted?

depends on the reason to the filedescriptor shortage.

If the shortage is due to Squid using very many filedescriptors then no
action need to be taken (except perhaps increase the amount of
filedescriptors available to Squid to avoid the problem in future).
Squid automatically adjusts to the per process limit and hitting the
system wide limit if it's lower than the per-process limit.

If the shortage is due to some other process causing the systems as a
whole to temporarily run short of filedescriptors or related resources
then you need to restart Squid after fixing the problem as Squid has got
fooled in this situation into thinking that your system can not support
a reasonable amount of active connections.

Different log for each domain

noreply@blogger.com (Unknown) — Thu, 26 Oct 2006 06:58:00 +0000

With squid 2.6 in a reverse proxy configuration with several domains, it's possible to have log messagees to separate files per cache_peer_domain.

Example with two peer domains.
www.abc.com
www.xyz.com

---squid.conf----
acl abc dstdomain www.abc.com
acl xyz dstdomain www.xyz.com

access_log /path/to/xyz.log squid xyz
access_log none xyz
access_log /path/to/abc.log squid abc
access_log none abc
access_log /path/to/access.log squid
---squid.conf end---

HTTP1.0 / HTTP1.1

noreply@blogger.com (Unknown) — Mon, 23 Oct 2006 07:30:00 +0000

Squid is a HTTP/1.0 proxy. But all HTTP/1.1 clients and servers MUST interoperate with HTTP/1.0 to be compliant.

The main difference this brings to the protocol (apart from version number) is that aplications or servers can't use transfer-encoding. The rest of HTTP/1.1 is supported fine over HTTP/1.0.

Transfer-Encoding tells the receiver what encoding has been performed on the message in order for it to be safely transported.
All transfer-encoding values are case-insensitive. HTTP/1.1 uses transfer-encoding values in the TE header field and in the Transfer-Encoding header field. The latest HTTP specification defines only one transfer encoding, chunked encoding.

Relay server for SSL connections

noreply@blogger.com (Unknown) — Fri, 13 Oct 2006 13:25:00 +0000

If you have overladed web servers you can save the SSL load put in front of their a relay proxy for ssl connections. This means that this proxy will accept HTTPS connections and will translate them into HTTP requests on another server.

This can be achieve with:
https_port directive.
and setting up squid as an https accelerator for the http server.

Example configuration;
https_port 443 cert=/path/to/sslcert.pem defaultsite=your.site.name
cache_peer ip.of.apache 80 0 no-query originserver

Redirector in a reverse proxy scenario

noreply@blogger.com (Unknown) — Fri, 13 Oct 2006 13:14:00 +0000

A common technique using in a reverse proxy scenario where virtual hostnames are distributed to several internal servers is using a redirector. But with the new 2.6 squid there is a better way to achieve this feature.
You can forward the requests with:
cache_peer + cache_peer_acess + never_direct

This is the recommend method if the other server is supposed to act exactly like the real web site, including it's name.

2Gb access.log

noreply@blogger.com (Unknown) — Fri, 13 Oct 2006 12:54:00 +0000

"FATAL: logfileWrite: /var/log/squid/access.log: (11) Resource
> temporarily unavailable"

A quit common cause for this message is the access.log reaching the magic 2GB barrier of 32-bit applications.

A common solution for this is compile squid with large cache files support.

From the configure script.

--with-large-files Enable support for large files (logs etc).
--enable-large-cache-files
Enable support for large cache files (>2GB).
WARNING: on-disk cache format is changed by this option

Use the no_cache directive correctly

noreply@blogger.com (Unknown) — Mon, 02 Oct 2006 06:51:00 +0000

If you don't want cache some object, the best directive to achieve it is the
no_cache directive

acl QUERY urlpath_regex cgi-bin \? \.asp$ \.php$
no_cache deny QUERY

The refresh_pattern directive only applies on pages where there is no explicit expiry information set by the origin server.

Limit the amount of time spending in personal stuff during work hours

noreply@blogger.com (Unknown) — Fri, 15 Sep 2006 08:25:00 +0000

Several people in her works hours are doing personal stuff such as reading gmail, yahoo mail, reading blogs, etc. In order to avoid these kind of behaviour you can do some acctions to remember them they are working and it's not good wasting work hours with not related things.

Help to accomplish this task:

session helper from squid-2.6.
deny_info redirecting to a web page with a reminder.

Recommended compile options

noreply@blogger.com (Unknown) — Mon, 11 Sep 2006 10:53:00 +0000

--prefix= if you want to install Squid in another location
than /usr/local/squid

--enable-storeio="ufs aufs null" on most platforms to allow choice of
more performing disk I/O.

--enable-auth to enable all authentication schemes to have them
available the day you want to use them.

--enable-delay-pools to enable the delay pools shaping capability.

--enable-snmp to enable monitoring & statistics collection using SNMP

--with-large-files if you are on a 32-bit platform to enable support
for large log files bigger than 2GB. Warning: Do not use this option on
64-bit platforms.

Golden Rule

noreply@blogger.com (Unknown) — Tue, 05 Sep 2006 08:32:00 +0000

Start with the first error, ignore the rest.

Some notes regarding NTLM

noreply@blogger.com (Unknown) — Tue, 05 Sep 2006 08:25:00 +0000

Squid 2.6Stable 3 now support the NTLM passthrough.

An alternative which is recommended and works for all proxies is to have the web site using https on authenticated content. https is tunneled via the proxy, not proxied, and therefore works fine even with non-HTTP-compliant authentication such as NTLM.

From now this blog will be in english language

noreply@blogger.com (Unknown) — Tue, 05 Sep 2006 08:24:00 +0000

Aumentar numero de Filedescriptores

noreply@blogger.com (Unknown) — Tue, 06 Jun 2006 07:13:00 +0000

Cuando compilamos una versión de squid, tenemos que tener en cuenta el número de filedescriptors que va a tener disponible. Cuando realizamos el configure, aparecerá una linea;

checking Maximum number of filedescriptors we can open... 1024

que es los que actualmente tenemos en nuestro sistema operativo disponibles.
Para aumentarlos;

#ulimit -HSn 2048

Volvemos a compilar, y comprobamos que efectivamente ha cogido bien el soporte de filedescriptors que ahora tendrá disponible.

checking Maximum number of filedescriptors we can open... 2048

1024 Filedescriptors

2048 Filedescriptors

SNMP y Squid

noreply@blogger.com (Unknown) — Tue, 30 May 2006 09:50:00 +0000

Para ver los diferentes parametros que podemos monitorizar con squid en mrtg.

snmpwalk host:3401 -v 1 -c public .1.3.6.1.4.1.3495 -m /usr/local/squid/share/mib.txt

Para lo que previamente tendremos que haber configurado el acceso snmp en squid.conf
para que permita consultas a host.

Ocultar la versión de squid

noreply@blogger.com (Unknown) — Fri, 19 May 2006 15:20:00 +0000

Para ocultar la versión de squid que se muestra en una página de error;

Edita src/errorpage.c
Linea:69 :)

>De
"Generated %T by %h (%s)\n"
a
"Generated %T by %h \n"

Y vuelve a compilar squid.

Update on 6th September
From 2.6Stable1 it's available the httpd_supress_version_string directive (default off)

Bloquear Malware con Squid

noreply@blogger.com (Unknown) — Fri, 19 May 2006 08:35:00 +0000

Hoy en día existen URLS que ocultan todo tipo de efectos dañiños tales como Virus, troyanos gusanos y todo lo denominado malware.
Un usuario desea saber como se puede conseguir con squid filtrar archivos peligrosos por su extensión, pero las típicas reglas que bloquean
\.com$
\.scr$
\.bat$
estas extensiones no sirven cuando hoy en día existen multitud de técnicas para ocultar archivos en urls tales como
http://www.mikes.educv.ro/albums/cartao.scr?4d325356ae47122a6e7b8f1f07cae26d
La solución para esto pasa por integrar squid con listas de URLs disponibles para él.
Aquí se explica la configuración de squid.