Squid Proxy Server 3.1: Beginner's Guide: Reduce bandwidth use and deliver your most frequently requested web pages more quickly with Squid Proxy Server. This guide will introduce you to the fundamentals of the caching system and help you get the most from Squid.

Before we explore a configuration file in detail, let's have a look at the minimal configuration that you will need to get started. Get ready with the configuration file located at /opt/squid/etc/squid.conf, as we are going to make the changes and additions necessary to quickly set up a minimal proxy server.

cache_dir ufs /opt/squid/var/cache/ 500 16 256
acl my_machine src 192.0.2.21 # Replace with your IP address
http_access allow my_machine

We should add the previous lines at the top of our current configuration file (ensuring that we change the IP address accordingly). Now, we need to create the cache directories. We can do that by using the following command:

$ /opt/squid/sbin/squid -z

We are now ready to run our proxy server, and this can be done by running the following command:

$ /opt/squid/sbin/squid

Squid will start listening on port 3128 (default) on all network interfaces on our machine. Now we can configure our browser to use Squid as an HTTP proxy server with the host...

Squid's configuration file can normally be found at /etc/squid/squid.conf, /usr/local/squid/etc/squid.conf, or ${prefix}/etc/squid.conf where ${prefix} is the value passed to the --prefix option, which is passed to the configure command before compiling Squid.

In the newer versions of Squid, a documented version of squid.conf, known as squid.conf.documented, can be found along side squid.conf. In this chapter, we'll cover some of the import directives available in the configuration file. For a detailed description of all the directives used in the configuration file, please check http://www.squid-cache.org/Doc/config/.

The syntax for Squid's documented configuration file is similar to many other programs for Linux/Unix. Generally, there are a few lines of comments containing useful related documentation before every directive used in the configuration file. This makes it easier to understand and configure directives, even for people who are not familiar with...

This directive is used to specify the port where Squid will listen for client connections. The default behavior is to listen on port 3128 on all the available interfaces on a machine.

Now, we'll see the various ways to set the HTTP port in the squid.conf file:

Access Control Lists (ACLs) are the base elements for access control and are normally used in combination with other directives such as http_access, icp_access, and so on, to control access to various Squid components and web resources. ACLs identify a web transaction and then directives such as http_access, cache, and then decides whether the transaction should be allowed or not. Also, we should note that the directives related to accessing resources generally end with _access.

Every access control list definition must have a name and type, followed by the values for that particular ACL type:

acl ACL_NAME ACL_TYPE value
acl ACL_NAME ACL_TYPE "/path/to/filename"

The values for any ACL name can either be specified directly after ACL_TYPE or Squid can read them from a separate file. Here we should note that the values in the file should be written as one value per line.

Let's construct an access control list for the domain name example.com:

acl example_site dstdomain example.com

In this code, example_site is the name of the ACL with type dstdomain, which reflects that the value, example.com, is the domain name.

Now if we want to construct an access control list which can cover a lot of example websites, we have the following three possible ways of doing it:

While Squid is running on our server, it can be accessed in several ways for example, via normal web browsing by end users or as a parent or sibling proxy server by neighboring proxy servers. Squid provides various directives to control access to different resources. Next, we'll learn about granting or revoking access to different resources.

http_access allow|deny [!]ACL_NAME

Let's have a look at a few cases for controlling HTTP access using example ACLs. When we have multiple access rules, Squid matches a particular request against them from top to bottom and keeps doing so until a definite action (allow or deny) is determined. Please note that if we have multiple ACLs within a single access rule, then a request is matched against all the ACLs from left to right, and Squid stops processing the rule as soon as it encounters an ACL that can't identify the request. An access rule with multiple ACLs results in a definite action, only if the request is identified by all the ACLs used in the rule.

acl my_home_machine src 192.0.2.21
acl my_lab_machine src 198.51.100.86
http_access allow my_home_machine
http_access allow my_lab_machine

The ACLs and access rules in the previous code will allow hosts 192.0.2.21 and 198.51.100.86 to access the proxy server. The aforementioned access rules may also be written as:

acl my_machines...

Cache peers or neighbors are the other proxy servers with which our Squid proxy server can:

We normally deploy more than one proxy server in the same network to share the load of a single server for better performance. The proxy servers can use each other's cache to retrieve the cached web documents locally to improve performance. Let's have a brief look at the directives provided by Squid for communication among different cache peers.

cache_peer HOSTNAME_OR_IP_ADDRESS TYPE PROXY_PORT ICP_PORT [OPTIONS]

Let's add a proxy server (parent.example.com) that will act as a parent proxy to our proxy server:

cache_peer parent.example.com parent 3128 3130 default proxy-only

3130 is the standard ICP port. If the other proxy server is not using the standard ICP port, we should change the code accordingly. This code will direct Squid to use parent.example.com as a proxy server to satisfy client requests in case it's not able to do so itself.

The option default specifies that this cache peer should be used as a last resort in the scenario where other peers can't be contacted. The option proxy-only specifies that the content fetched using this peer should not be cached locally. This is helpful when we don't want to replicate cached web documents, especially when the two peers are connected with a high bandwidth backbone.

All this time, we have been talking about the caching of web documents and how it helps in saving bandwidth and improving the end user experience, now it's time to learn how and where Squid actually keeps these cached documents so that they can be served on demand. Squid uses main memory (RAM) and hard disks for storing or caching the web documents.

Caching is a complex process but Squid handles it beautifully and exposes the directives using squid.conf, so that we can control how much should be cached and what should be given the highest priority while caching. Let's have a brief look at the caching-related directives provided by Squid.

We can use extra RAM space available on a running system after sparing a chunk of memory that can be utilized by the running process under heavy load. To find out the amount of free RAM available on our system, we can use either the top or free command. To find out the free RAM in Megabytes, we can use the free command as follows:

$ free -m

For more details, please check the top(1) and free(1) man pages.

Now, let's say we have 4 GB of total RAM on the server and all the processes are running comfortably in 1 GB of RAM space. After securing another 512 MB for emergency situations where running processes may take extra memory, we can safely allocate 2.5 GB of RAM for caching.

To specify the cache size in the main memory, we use the directive cache_mem. It has a very simple format. As we have learned before, we can specify the memory size in bytes, KB, MB, or GB. Let's specify the cache memory size for the previous example:

cache_mem 2500 MB...

The cache directory location may not be on the same disk or partition. We can mount another disk drive and specify that as the directory for caching. For example, let's say we have another drive connected as /dev/sdb and one of the partitions is /dev/sdb1, we can mount it to the /drive/ and use it right away.

$ mkdir /drive/
$ mount /dev/sdb1 /drive/squid_cache/
$ mkdir /drive/squid_cache
$ chown squid:squid /drive/squid_cache/

In the previous code, we created a directory /drive/ and mounted /dev/sdb1, the partition from the other disk drive, to it. Then, we created a directory squid_cache in the directory /drive/ and changed the ownership of the directory to Squid, so that Squid can have write access to this directory. Now we can use /drive/squid_cache/ as one of the directories with the cache_dir directive.

So far we have learned the meaning of different parameters used with the cache_dir directive. Let's see an example of the cache directory /squid_cache/ with 50GB of free space:

cache_dir aufs /squid_cache/ 51200 32 512

We have a cache directory /squid_cache/ with 50 GB of free space with the values of L1 and L2 as 32 and 512 respectively. So, if we assume the average size of a cached object to be 16 KB, there will be 51200x1024÷(32x512x16) = 200 cached objects in each of the directories at the second level, which is quite good.

Although Squid performs quite well with default caching options, we can tune it to perform even better, by not caching the unwanted web objects and caching a few non-cacheable web documents. This will achieve higher bandwidth savings and reduced latency. Let's have a look at a few techniques that can be helpful.

If we don't want to cache responses for certain requests or clients, we can deny it using this option. The default behavior is to allow all cacheable responses to be cached. As servers in our local area network are close enough that we may not want to waste cache space on our proxy server by caching responses from these servers, we can selectively deny caching for responses from local servers.

acl local_machines dst 192.0.2.0/24 198.51.100.0/24
cache deny local_machines

This code will prevent responses from the servers in the networks 192.0.2.0/24 and 198.51.100.0/24 from being cached by the proxy server.

Let's see an example of a refresh_pattern and try to calculate the freshness of an object:

refresh_patten -i ^http://example.com/test.jpg$ 0 60% 1440

Let's say a client requested the image at http://example.com/text.jpg an hour ago, and the image was last modified (created) on the web server six hours ago. Let's assume that the web server didn't specify the expiry time. So, we have the following values for the different variables:

Let's check whether the object is still fresh or not:

Now, let's calculate the time when the object will expire. The object age is 5 hours and percent value is 60 percent...

As all the requests and responses pass through Squid, it can add, modify, or delete the HTTP headers accompanied with requests and responses. These actions are usually performed to achieve anonymity or to hide the client-specific information. Squid has three directives, namely, request_header_access, reply_header_access, and header_replace to modify the HTTP headers in requests and responses. Let's have a brief look at them.

request_header_access...

For every request received from a client, Squid needs to resolve the domain name before it can contact the target web server. For this purpose, Squid can either use the built-in internal DNS client or, external DNS program to resolve the hostnames.

The default behavior is to use the internal DNS client for resolving hostnames unless we have used the --disable-internal-dns option but it must be set with the configure program before compiling Squid, as shown:

$ ./configure --disable-internal-dns

Let's have a quick look at the DNS-related configuration directives provided by Squid.

cache_dns_program...

A list of IP addresses can be passed to this directive or several IP addresses can be written on different lines like the following:

dns_nameservers 192.0.2.25 198.51.100.25
dns_nameservers 203.0.113.25

The previous configuration lines will set the name servers to 192.0.2.25, 198.51.100.25, and 203.0.113.25.

hosts_file /etc/hosts

Squid logs all the client requests and events to files. Squid provides various directives to control the location of log files, format of log messages, and to choose which requests to log. Let's have a brief look at some of the directives. We'll learn about logging in detail in Chapter 5, Understanding Log Files and Log Formats.

URL rewriters and redirectors are third party, independent helper programs that we can use with Squid to modify or rewrite requests from clients. In most cases, we try to redirect a client to a different web page or resource from the one that was initially requested by the client.

The interesting part is that URL rewriters can be coded in any programming language. URL rewriters are run as independent processes and communicate with Squid using standard I/O.

URL rewriters provide a totally new area of opportunity as we can redirect clients to custom error pages for different scenarios, redirect users to local mirrors of websites or software repositories, block advertisements with small blank images, and so on.

Squid doesn't have any URL rewriters by default as we are supposed to write our own URL rewriters because the possibilities are enormous. It is also possible to download URL rewriters written by others and use them right away. We'll learn about how to use or...

Squid has hundreds of configuration directives to control it in various ways. It's not possible to discuss all of them here, we'll try to cover the important ones.

cache_effective_user squid

We have learned a lot in this chapter about configuring Squid. After this chapter, we should feel more comfortable in dealing with the Squid configuration file. We should be able to apply the things we learnt in this chapter to fine tune Squid to achieve better performance.

Although we learned about a lot of configuration directives, we specifically covered:

We also discussed restricting access to the Squid server, modifying HTTP headers...